Multi-teacher on-policy distillation

Headlines · 1

1. Nov 2
   On-policy distillation can fail to preserve generalization

   A preprint by Wang and six other authors extends on-policy distillation with a multi-teacher approach for consolidating capabilities from several teachers. Using Qwen3 models across mathematical reasoning, coding, agent, and long-context tasks, the authors report that strong in-distribution results can mask failures outside the training distribution: extrapolative scores were 2.45%–16.25% below the base model, while compositional scores were 18.19%–24.34% below the best teachers. The authors propose geometry-based patching and base-preserving replay to improve out-of-distribution performance while largely preserving in-distribution performance.

   Hugging Face · Papers · 🔥 0

Experience and discussion from the community

Share my Multi-teacher on-policy distillation experienceAsk about Multi-teacher on-policy distillation