AI developers who rely on web-scale image datasets could inherit malicious content for about $60, researchers found, creating a route for an attacker to influence the results a model produces.
The weakness stems from the way LAION-400M keeps links to images rather than the images themselves. That leaves a gap between the time a curator records a web address and the point when a later user downloads its content. In 2023, a team led by Nicholas Carlini of Google, with Florian Tramèr of ETH Zurich among its members, demonstrated two cheap ways to poison large web datasets.
LAION-400M contains 400 million image and text pairs gathered from Common Crawl. Its links can point to images hosted on domains that later expire, allowing somebody else to register those domains and change what later downloads retrieve.
The researchers called the technique split-view poisoning. They showed that an attacker could register expired domains still listed in LAION-400M or the similar COYO-700M dataset and substitute malicious images for approximately $60. They estimated that 0.01% of either dataset could be poisoned this way. In LAION-400M, 0.01% is about 40,000 image and caption pairs.
The research described a possible attack rather than a confirmed case of poisoning in either dataset. Its concern rests on the scale of the change that a small number of substitutions could cause. Earlier work by Carlini and Andreas Terzis found that a few hundred carefully built examples could plant a hidden trigger in a model.
Dataset problems have also appeared outside the researchers' demonstration. In December 2023, Stanford researchers reported 1,008 links in LAION-5B leading to known or likely child sexual abuse material. LAION withdrew LAION-5B from circulation, removed 2,236 links and released a cleaned version, Re-LAION-5B, on 30 August 2024.
The risks extend beyond images. In January 2024, Anthropic researchers trained models that wrote secure code when told the year was 2023 and inserted exploitable code when told it was 2024. Supervised fine-tuning, reinforcement learning and adversarial training failed to remove that backdoor behaviour.
A 2024 study in Nature by Ilia Shumailov and colleagues found that models repeatedly trained on material generated by earlier models undergo “model collapse”. An October 2025 study by Anthropic, the UK AI Security Institute and the Alan Turing Institute found that as few as 250 documents could install a backdoor that made models produce gibberish in response to a trigger phrase. The researchers said the number of poisoned documents required did not rise with model or dataset size in their experiments.
Data poisoning corrupts information that an AI system learns from or retrieves, with the aim of making the system deliver an attacker's chosen results. An attacker does not need access to a model or its weights to do it. Once a backdoor is embedded in a model's weights, it travels with every deployment of that checkpoint, including copies that have been fine-tuned, quantized or sent into a customer VPC.
The National Institute of Standards and Technology included data poisoning and unclear data origins among the risks companies should track in its 2024 profile for generative AI. The European Union's AI Act transparency rules took effect on 2 August 2026, requiring providers of AI systems that create images, audio, video or text to mark that material as AI-made in a machine-readable form.
Comments (0)