An AI agent read a user’s emails about a $680K 401K and a vested stock grant, then suggested a $601 business-class ticket despite the availability of a $91 economy fare, according to an example in a new paper.
The paper asks whether an assistant with access to personal information may use it in the way a seller might. “We asked a simple question: if we hand all of that to our assistant and ask it to shop for us, will it use that knowledge against us, the way a seller might?” said Aman Priyanshu, a Cisco Foundation AI researcher and co-author.
Researchers at Cisco Foundation AI and Carnegie Mellon University found that popular chatbots, including Claude and ChatGPT, recommended more expensive products when users’ personal data suggested they had greater wealth. The paper is a preprint and has yet to undergo peer review.
The researchers built 32 synthetic personas using five binary attributes: financial status, employment, health, life events and neighbourhood demographics. Each persona made the same request without reference to income, choosing from a fixed catalogue of 200 options with fixed prices. The tests covered flights between Denver and Chicago, individual health insurance plans in a Colorado zip code, and computer science PhD programmes.
They then ran 325,000 experiments across 13 chatbot models from four families: OpenAI, Anthropic, Google and Qwen. The models included OpenAI GPT-5 variants, Anthropic Claude variants, Google Gemini Flash variants and open-weight Qwen3.5 models ranging from 2B to 35B parameters.
Eight of the 13 models suggested pricier choices to personas inferred to be wealthy, despite identical shopping requests. Claude Opus 4.8 had the widest differences, recommending flights that averaged $198 more for wealthy users and insurance plans that averaged $284 more per month.
Gemini 2.5 Flash showed a $177 difference for flights and a $217 monthly difference for insurance. GPT-5 recommended flights averaging $107 more to wealthy profiles, among the smaller gaps found in capable models.
The pattern persisted when the instruction was explicit. When a wealthy profile asked Gemini 2.5 Flash for the cheapest flight, it received a recommendation averaging $208 more than the option suggested to a low-income profile making the same request.
The researchers also tested what happened when they withheld parts of the personas’ structured data. Removing that financial information and allowing models to read only email inboxes still left a considerable share of the pricing gap. For Gemini 2.5 Flash, access to only two emails resulted in a $175 gap, larger than the $91 gap seen with full inbox access.
Blocking employment information increased GPT-5’s insurance gap by 40%. Directly blocking financial attributes largely eliminated the disparity.
Hard numerical price ceilings brought the gap close to zero for most capable models. Gemini 2.5 Flash still showed a $108 difference even when given an explicit ceiling.
OpenAI said the version of ChatGPT assessed in the research differs from the one used for its consumer shopping experience. Anthropic and Google did not respond to Bloomberg’s requests for comment.
Comments (0)