Visa says nearly 43% of the shopping tasks in its new benchmark ended in failure, including cases where an AI agent bought something that did not meet the shopper’s requirements. That is a useful stress test for the promise of AI shopping: the agent has to know when not to buy, not merely how to check out.
Watch Desk analysis
What happened
Visa’s Agentic Commerce Benchmark tested 22 AI models from 12 labs in realistic shopping scenarios, including finding a TV, laptop or personal-care appliance. According to the benchmark results reported by PYMNTS, 57.2% of test runs passed and 42.8% failed. Visa says the test assesses whether models identify a shopper’s needs, gather enough information to justify a purchase and abstain when nothing suitable is available.
A particularly awkward result: 54% of failed runs still ended with the agent making a purchase, even though it had not found an item meeting the shopper’s requirements or had not uncovered all those requirements. Visa says the five most expensive models passed 67.1% of runs, compared with 50.2% for the five least expensive.
Why it matters
For an AI shopping agent, a wrong purchase is not just a clumsy answer. It is an action with a price tag attached. The results put a number on the gap between an agent that can complete a transaction and one that has understood what the shopper actually asked for.
The cost comparison adds another wrinkle: in Visa’s test, the more expensive group performed better, but neither group passed every time. These figures describe this benchmark, not the full range of shopping tasks or every model in everyday use. Even so, the reported failure rate makes reliability and sensible restraint hard to treat as decorative extras.
Our read
The most revealing number is not the 57.2% pass rate. It is that more than half of failed runs still bought something. An agent that can politely decline a bad match may be more useful than one that treats every conversation as a sales funnel with excellent manners.
Visa’s benchmark is a starting point, not a neutral verdict on the entire field. The next useful comparison is how the test defines a pass, how models fare across more kinds of shopping and whether improved results hold up outside the benchmark.
What to watch
- Whether Visa publishes the full benchmark methodology and model-by-model results.
- Whether future rounds test a wider range of shopping tasks and shopper needs.
- Whether agent providers show that they can improve success without turning restraint into an optional feature.
Discussion spark: Should an AI shopping agent be judged mainly by how often it completes a purchase, or by how reliably it avoids buying something that misses the shopper’s requirements?
Sources and evidence
- Source update (5 November 2026, 22:21 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.