Filtering viral proteins from pretraining data weakens AI models on viral tasks
A study of ESM-2 protein language models finds data filtering selectively degrades viral-protein performance, but fine-tuning on modest compute restores it.
In plain words
Protein language models are AI systems trained on protein sequences. Researchers removed viral proteins from the training data of the ESM-2 model family and found the resulting models performed worse on viral-protein tasks. However, retraining briefly on the removed data brought performance back for little cost.
What was announced
The authors, writing on arXiv, examined whether pretraining data filtering, a common safeguard for open-weight models, actually works. Using the ESM-2 family, they report that excluding viral proteins from training hurt masked language modelling, mutation effect prediction and long-range contact prediction on viral proteins, while non-viral performance was largely kept. The degradation was strongest for proteins from human-infecting viruses and grew as models scaled from 8M to 3B parameters, with stricter filtering becoming more effective at larger sizes.
Durability proved limited. Fine-tuning on sequences from the removed viral families recovered the lost capability, and although recovery costs rose with model size, the authors state it stayed around 80 USD of H100 GPU time even for the largest models. They conclude that such filtering is moderately effective at degrading viral-protein performance but not durable against fine-tuning with modest resources and access to relevant data.
What it does not show
- The source is an abstract, so the full paper's methods and results are not covered here.
- Findings are reported for the ESM-2 protein model family and may not extend to other model types.
- The cost and recovery figures come from the authors' own study and have not been independently confirmed.
For security and IT teams
- Status
- The paper is a v2 arXiv listing with a replace-cross announcement type, indicating a revised posting.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.