On-policy distillation can fail to preserve generalization
A preprint by Wang and six other authors extends on-policy distillation with a multi-teacher approach for consolidating capabilities from several teachers. Using Qwen3 models across mathematical reasoning, coding, agent, and long-context tasks, the authors report that strong in-distribution results can mask failures outside the training distribution: extrapolative scores were 2.45%–16.25% below the base model, while compositional scores were 18.19%–24.34% below the best teachers. The authors propose geometry-based patching and base-preserving replay to improve out-of-distribution performance while largely preserving in-distribution performance.
The paper identifies a blind spot in capability consolidation: strong in-distribution distillation does not necessarily mean a model preserves or transfers teacher capabilities outside the supervised distribution.
Computer Science > Machine Learning
arXiv:2611.05609 (cs)
[Submitted on 3 Nov 2026]
Title:On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation
Authors:Yichen Wang, Yufa Zhou, Siqi Zhu, Haozheng Luo, Jingzhi Bao, Zhichao Liu, Zijun Wei
View a PDF of the paper titled On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation, by Yichen Wang and 6 other authors
#Multi-teacher on-policy distillation#On-policy distillation#Qwen3#Geometry-based patching#Base-preserving replay