Study finds bias in on-policy self-distillation
A research update argues that on-policy self-distillation (OPSD) can bias a model toward a teacher’s preferred continuations, including stylistic tokens such as hedging words, without ensuring improvement in its ability to solve tasks. Token-level analysis covering 800 rollouts across 200 level 4–5 MATH-500 problems found systematic disagreements over wording rather than mathematical correctness. The authors conclude that OPSD is not a silver bullet for continual learning and question whether further research investment is worthwhile, while acknowledging that it may remain useful in certain contexts.
The token-level findings challenge the assumption that dense teacher signals from OPSD reliably indicate better reasoning, because much of the observed signal concerns stylistic differences rather than mathematical correctness.
Foundations
The bias of on-policy self-distillation
Why on-policy self-distillation is not the silver bullet for continual learning.
Authors
Date
November 6, 2026