Benchmark Finds AI Agents Read SaaS Sites Better Than They Can Sign Up

Summary

Agent Conversion's D-001 benchmark tested one AI browsing agent on three commercial tasks across 30 B2B SaaS websites, with each task run twice. After excluding one company where the measurement system produced no valid observation, 174 runs across 29 measurable companies were analyzed. The agent completed 149 of 174 valid journeys, or 85.6%; when the site did not refuse access, completion was 149 of 160, or 93.1%. Reading official pricing succeeded in 49 of 58 runs, reaching sales in 53 of 58, and starting self-service signup in 47 of 58. Of 25 valid failures, 14, or 56%, were caused by bot challenges or verification pages returned by the site, while 10 came from agent behavior such as timeouts, navigation errors, or incorrect answers; one was caused by a site button that reported a click without navigating. On Airtable, Cloudflare, DigitalOcean, and Webflow, the agent reached sales in both runs but failed self-service signup in both runs. The benchmark recorded 72 of 87 company-task pairs completing twice, for an 82.8% repeatable completion rate, and found only five unstable pairs. It did not support the expected claim that second attempts were markedly less reliable than first attempts. The production score agreed with independent adjudication on 166 of 174 valid runs, but all eight disagreements were overly generous, including failed journeys scored above 90. The authors emphasize that the results describe one agent, one model, one cloud network, one date, and a non-random company sample; they conclude that agent readiness should be judged by whether an agent can complete a commercial journey, not only by whether it can read a site or receive a high score.