AISI: Data Filtering Is Not Durable for Open-Weight Protein Models
The UK AI Security Institute pretrained ESM-2 protein language models after excluding viral protein sequences, finding that filtering reduced performance on viral proteins while largely preserving performance on non-viral proteins. However, fine-tuning on the withheld public sequences fully restored viral-protein performance, requiring about $80 of H100 compute even for the largest 3B-parameter model. AISI says the approach therefore needs additional measures, such as data access controls, to form a robust safeguard.
The results show that pretraining-data filtering can selectively suppress a targeted biological capability, but it does not by itself reliably prevent that capability from being recovered.
Specialised biological AI models (or tools) are driving rapid advances in biotechnology. They are typically released as open-weight models and are smaller than frontier large language models (LLMs), as they are trained predominantly on biological data only. This makes them easier to train, share and modify.
However, robustly safeguarding open-weight models is difficult. Once the weights and inference code are openly available, the effectiveness of safeguards is harder to guarantee. Post-training protections can often be cheaply undone through fine-tuning, and safety classifiers can often be removed, though how much effort this takes varies and is not always clear.
To be effective, safeguards for open-weight models must be tamper-resistant. Of the safeguards developed to date, one of the most promising is stringent filtering of the pre-training data, since this often requires significantly more fine-tuning compute to undo, compared to post-training interventions. It has sometimes been pointed to by developers as a justification for openly releasing model weights.
Prior work on LLMs (see here and here) suggests filtering can selectively degrade performance in a targeted domain, while leaving performance in other domains largely intact. This suggests it could, in principle, be used to remove information relevant to a dual-use capability from specialised scientific models, such as protein models. However, the training data for these models is much less varied than for LLMs, which makes a target domain harder to isolate: the data left behind after filtering may closely resemble the data removed, which could in turn affect how well filtering works in these contexts.