Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Visa shopping benchmark: AI agents failed 42.8% of runs

AI summary

Visa said its Agentic Commerce Benchmark tested 22 AI models from 12 labs on 15 shopping tasks, including selecting televisions, laptops and personal care appliances. The models failed 42.8% of their runs, while the best passed 82.2% and the worst 28.9%. Each model had three attempts at each scenario, and all three attempts passed 34.2% of the time. Visa did not name the tested models; Fortune separately reported that agents from Google and Meta shared the top score, followed by a Claude agent in third place and an OpenAI agent in sixth. The supplied report does not independently verify Visa’s results.

In-depth insight

The benchmark tests whether shopping agents can uncover omitted requirements, make the correct decision to buy or not buy, justify their choices and perform consistently across repeated scenarios. Visa said the results show that consistency and reliability remain problems for AI agents in commerce.

Visa tested 22 AI models as shoppers, and they failed 42.8% of the time
Credit: Visa
Image Credits
Credit: Visa

Visa has launched an AI shopping benchmark, and in its first round the AI models failed 42.8% of their test runs. The Visa Agentic Commerce Benchmark put 22 AI models from 12 labs through 15 shopping tasks, the company said in a statement on Thursday.

The tasks included buying a TV, a laptop and a personal care appliance. The customer did not give every detail up front. Each AI had to work out what they needed by talking to them. If nothing on offer fitted, the AI had to decide not to buy. To rule out lucky guesses, it also had to find enough information to justify its choice.

#Visa#Visa Agentic Commerce Benchmark#Google#Meta#Claude#OpenAI

Original · 3 min readRead the original
Group chat

No comments yet. Start the conversation.