On-policy distillation

Headlines · 3

1. Nov 2
   On-policy distillation can fail to preserve generalization

   A preprint by Wang and six other authors extends on-policy distillation with a multi-teacher approach for consolidating capabilities from several teachers. Using Qwen3 models across mathematical reasoning, coding, agent, and long-context tasks, the authors report that strong in-distribution results can mask failures outside the training distribution: extrapolative scores were 2.45%–16.25% below the base model, while compositional scores were 18.19%–24.34% below the best teachers. The authors propose geometry-based patching and base-preserving replay to improve out-of-distribution performance while largely preserving in-distribution performance.

   Hugging Face · Papers · 🔥 0
2. Oct 19
   Ask-to-Distill optimizes synthetic questions for on-policy distillation

   A new arXiv preprint (2610.27450, submitted 20 Oct 2026) by Jiashuo Sun and 11 co-authors introduces Ask-to-Distill, a framework that treats synthetic questions as "distribution elicitors" and optimizes the teacher supervision they expose along student-generated trajectories. A student-initialized question synthesizer is trained using a distributional surrogate for question value computed from a frozen student–teacher pair, without reference answers, correctness rewards or per-candidate student updates; the reward combines teacher–student disagreement with teacher probability mass on the student's candidate support, plus length and diversity controls. The authors report that across mathematical reasoning, code generation and agentic tasks, the method improves average performance and teacher-gap recovery over unoptimized student-generated prompts and KL-filtered real questions, and matches or exceeds real-bank on-policy distillation in the evaluated settings, with code, data and models stated as publicly available.

   Hugging Face · Papers · 🔥 0
3. Oct 13
   CriticOPD uses teacher critiques to fix on-policy distillation errors

   An arXiv paper submitted on 14 October 2026, "Recovering Reliable Supervision via Critique-Guided On-Policy Distillation" by Kaixiang Zhao and six co-authors, examines on-policy distillation (OPD), which trains a student on its own rollouts with token-level teacher supervision and assumes the teacher can reach a correct answer from any prefix the student writes. The authors call this property the teacher's recoverability and say it fails on incorrect rollouts: the teacher follows student errors even though it recognizes them. Their method, CriticOPD, has the teacher critique an incorrect rollout so the student can rewrite its solution, with the teacher scoring the revision including the critique as privileged information while the student trains on the revision without the critique. The authors report gains over OPD on two Qwen3 students across six mathematical reasoning benchmarks, largest on competition problems.

   Hugging Face · Papers · 🔥 0

Experience and discussion from the community

Share my On-policy distillation experienceAsk about On-policy distillation