← News

Models & toolsSources: arXiv · 19 Nov 2026

Withheld training data can be replaced by general model progress, study finds

An arXiv abstract reports that language models can regain deliberately withheld skills from general training progress, with the needed data shrinking as models improve.

In plain words

AI developers sometimes leave certain skills out of training data to keep models from learning them. The researchers found that models still pick up those skills later, because learning everything else acts as a substitute for the missing data. The more advanced the model becomes, the less withheld data is needed to recover the skill.

What was announced

Researchers behind a new arXiv paper examined how long data filtering delays a capability and how much data is needed to restore it. They withheld the training signal for individual skills in models of chess, Othello, molecular strings, bracket languages and web text, then measured what happened. According to the abstract, the general progress a model makes on the rest of its training data substitutes for the data that was withheld, and the delay seen in larger models could be predicted from smaller ones.

The results suggest filtering is a delay rather than a block. A chess model never explicitly trained to respond to check still ranked legal responses almost as well as a trained model, the authors report. Once models were trained on the withheld data, they acquired the skill with a small fraction of the data needed from initialization, and that fraction fell as models progressed, while a control rule of equal difficulty was learned much more slowly. The authors also withheld a programming language from a 7-billion-parameter model and found it was regained the same way. They conclude that data filtering delays a capability, and the data required to regain it decreases as models improve.

What it does not show

  • The evidence is an abstract only; full methods, benchmarks and caveats in the paper are not covered here.
  • The abstract does not quantify how long filtering delays capabilities or how much data is needed in absolute terms.
  • No independent replication of the findings is reported in the supplied material.

For security and IT teams

Scope
The finding covers withheld skills in chess, Othello, molecular strings, bracket languages, web text and one programming language in a 7-billion-parameter model, as described in the abstract.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.