The Year in AI Papers
The year/Anthropic

Paper 2510.07192

Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples

Alexandra Souly, Javier Rando, Ed Chapman, Xander Davies, Burak Hasircioglu, Ezzeldin Shereen, Carlos Mougan, Vasilios Mavroudis, Erik Jones, Chris Hicks, Nicholas Carlini, Yarin Gal, et al.

Published
Oct 2025
Research lab
Anthropic
Citations
76
GitHub
Not linked

01 In brief

Summary

This paper investigates whether poisoning attacks on large language models (LLMs) require a constant number of poisoned samples regardless of dataset size, rather than a fixed percentage.

The authors conducted the largest pretraining poisoning experiments to date, training models from 600M to 13B parameters on Chinchilla-optimal datasets (6B to 260B tokens).

They found that 250 poisoned documents successfully backdoored models across all scales, despite larger models training on over 20 times more clean data.

The attacks, including denial-of-service and language-switching backdoors, were effective when the absolute number of poisoned samples was fixed, while the poisoning percentage varied.

Ablations showed that per-batch poison density and batch frequency had minimal impact, and continued clean training degraded attack success somewhat.

The same dynamics were observed during fine-tuning of Llama-3.1-8B-Instruct and GPT-3.5-turbo, where the number of poisoned samples, not the poisoning rate, determined attack success.

The results suggest that poisoning attacks become easier as models scale, highlighting the need for more research on defenses.

02 From the paper

Abstract

Poisoning attacks can compromise the safety of large language models (LLMs) by injecting malicious documents into their training data. Existing work has studied pretraining poisoning assuming adversaries control a percentage of the training corpus. However, for large models, even small percentages translate to impractically large amounts of data. This work demonstrates for the first time that poisoning attacks instead require a near-constant number of documents regardless of dataset size. We conduct the largest pretraining poisoning experiments to date, pretraining models from 600M to 13B parameters on chinchilla-optimal datasets (6B to 260B tokens). We find that 250 poisoned documents similarly compromise models across all model and dataset sizes, despite the largest models training on more than 20 times more clean data. We also run smaller-scale experiments to ablate factors that could influence attack success, including broader ratios of poisoned to clean data and non-random distributions of poisoned samples. Finally, we demonstrate the same dynamics for poisoning during fine-tuning. Altogether, our results suggest that injecting backdoors through data poisoning may be easier for large models than previously believed as the number of poisons required does not scale up with model size, highlighting the need for more research on defences to mitigate this risk in future models.