AI shopping agents clear just 57% of Visa's agentic commerce benchmark as wrong purchases complicate the trust case
AI shopping agents clear just 57% of Visa’s agentic commerce benchmark as wrong purchases complicate the trust case #
Visa has released what it calls the first industry-wide benchmark for agentic commerce. The opening results show that 22 AI models it tested failed nearly 43% of realistic shopping scenarios, and in more than half of those failures, the agent still went ahead and completed a purchase.
The Visa Agentic Commerce Benchmark, released November 5 in San Francisco, put 22 models from 12 research labs through 15 simulated shopping scenarios covering consumer electronics categories including televisions, laptops, and personal care appliances. Across all runs, the models achieved a 57.2% pass rate, with a 42.8% failure rate that Visa says points to a reliability gap in current agentic shopping technology.
The benchmark requires each model to first engage in dialogue to determine what a consumer wants before recommending or completing a purchase, rather than starting from a predefined shopping list. This design reflects how buyers frequently arrive with only a vague sense of their requirements. Under the benchmark’s rules, failing to surface those needs before transacting counts as a failed run.
Performance varied widely across the 22 models. Top performers cleared 82.2% of their runs, while the least capable managed only 28.9%. Fortune’s reporting on the results identified Google and Meta agents as sharing the leading position at around 82%, a Claude-based model in third place at approximately 76%, and an OpenAI model in sixth at roughly 64%, though Visa’s official release did not name individual labs.
The finding with the clearest commercial implications is that 54% of failed runs still ended in a completed transaction. In those cases, the agent either recommended a product that did not meet the customer’s identified requirements, or it never fully determined what the customer needed. That works out to roughly one in four of all test runs ending in a wrong purchase rather than the agent pausing to collect more information.
Rubail Birwadker, Visa’s Global Head of Growth, said in the announcement that while AI agents have the potential to drive the next era of commerce, trust is the essential prerequisite for that future. Visa is positioning the benchmark as an ongoing industry resource rather than a one-time evaluation: it is publicly available online so developers and researchers can run models through the same tasks, and Visa has said it will issue updated results as model capabilities progress.
Visa framed the initiative as a guide for payment providers and developers, aimed at clarifying what improvements are needed before autonomous shopping agents can be trusted to act reliably on a consumer’s behalf.
Copy link