Can you trust AI chatbots with your holiday shopping?
Product.ai found Gemini, ChatGPT, Claude, and Perplexity disagree on prices and specs most of the time
Eighty-six per cent of AI shopping questions produced a repeatable factual conflict, according to a study published this week by Product.ai, a startup that checks product claims against evidence.
The findings shows that AI chatbots are not ready to be used for holiday shopping without any supervision by a human being. The experiment included both paid and free variants of ChatGPT, Claude, Gemini, and Perplexity based on 220 shopping queries, including laptops, TVs, mattresses, sunscreen, and robotic vacuum cleaners.
Each question was asked five times for each service, yielding a total of 8,794 responses. Conflict was considered any contradicting detail that could be easily checked.
Questions asking models to compare two products directly produced conflicts 97% of the time, compared to a 75% conflict rate for straightforward factual questions. Pricing was a particular weak spot: of 913 verifiable answers, only 85% matched the current listed price, and when answers were wrong, the price was off by a median of $300.
Gemini posted the highest rate of costly errors in the study, at 56% on its free tier and 54% on paid. Claude's accuracy improved substantially on its paid version, with costly errors dropping to 21% from 44% on the free tier.
Perplexity had the lowest costly-error rate at 14% on paid, followed by ChatGPT's paid tier at 17%. Gemini's free tier also contradicted itself most often, giving inconsistent answers to the same question 29% of the time.
"In short, it says that these LLMs aren't there yet when it comes to this end-to-end experience," said Dakota Nunley, Product.ai's head of search product.
The spokesman for Perplexity dismissed this comment because their firm remains focused on accuracy, whereas Perplexity leads the market in. The spokespeople from ChatGPT, Claude, and Gemini were not responsive to the inquiry.
According to Nunley, one should employ the AI as an instrument of exploration and not rely on it solely, confirming the information via multiple engines or by visiting the manufacturing companies’ websites while shopping.
