General-purpose artificial intelligence chatbots are being used for financial guidance despite error rates that rise sharply when questions become more complex, according to a new study by financial technology firm Saturn. The research found that ChatGPT, Claude, CoPilot, Grok and Gemini gave incorrect answers to financial inquiries 57% of the time on average. Accurate answers accounted for 43% of responses.
Saturn tested 18 popular AI models using 121 financial questions. Each question was put to the models 5 times, producing more than 10,000 queries. The company said the results pointed to particular risks for consumers who use chatbots to make decisions on pensions, borrowing and other personal-finance matters without checking the response.
Mistakes appeared in 88% of answers to harder questions, according to the study. Some models answered 99% of more complex questions incorrectly. Free models made mistakes in 63% of responses, compared with 49% for paid-for models. On the hardest questions, free models made mistakes in 93% of answers.
The findings are published in Saturn's report, titled 'Artificial Authority: Should you trust AI to deliver financial advice?'. Saturn is calling on the Financial Conduct Authority, or FCA, to bring AI financial advice inside the regulatory perimeter.
The study found errors ranging from incorrect calculations to omissions of vital risk warnings. Some responses failed to account for upcoming tax changes, while others presented rules that did not exist as though they were established financial guidance.
Claude Haiku 4.5 was the weakest-performing model in Saturn's overall results, making mistakes in 82% of answers. Google's Gemini 3.1 Pro was second-worst, failing 73% of tests. Claude Opus 5 (reasoning) performed best overall, though it still made mistakes in 39% of answers.
Performance deteriorated further on complex queries. Google's Gemini 3.5 Flash and Claude Haiku 4.5 each gave wrong answers 99% of the time when asked complex financial questions, Saturn said.
Several examples in the report illustrate how errors could translate into costly decisions. Saturn said an incorrect pension-tax answer from Claude Haiku 4.5 could leave a pension saver facing a £17,500 charge from HMRC. Claude also invented a student-loan rule, telling a graduate that repayments could stop if they were moving abroad.
A Gemini model gave a borrower incorrect reassurance that a mortgage payment holiday would leave their credit score unaffected. Such answers matter because users may treat a fluent response as reliable guidance even where the chatbot has failed to identify a financial risk or has fabricated a rule.
Consumer use of the tools is already significant. The FCA's The Mills Review found that 26% of consumers trust general-purpose AI tools, including ChatGPT and Claude, for financial advice.
A PensionBee survey of 1,000 US adults who use AI chatbots for personal finance found that 57% would act on a chatbot's money advice without checking it first. Nearly a quarter, or 23%, said a chatbot had already provided incorrect information about their finances. Five percent said they learned the information was wrong only after acting on it.
Saturn's results do not establish how the models compare with human financial advisers, and the study does not specify how the 121 questions were selected or weighted across financial topics. The findings nevertheless identify a gap between the confidence some consumers place in general-purpose AI and the reliability shown in the company's tests, particularly where a response involves tax, credit or pension consequences.
Comments (0)