CriticOPD adds teacher critiques to on-policy distillation
A new arXiv paper proposes CriticOPD, a method for improving on-policy distillation (OPD), which trains a student on its own rollouts using token-level teacher supervision. The authors argue that OPD assumes the teacher is "recoverable" — able to reach a correct answer from any prefix the student writes — and show this fails on incorrect rollouts, where the teacher recognizes the error but follows it anyway. CriticOPD has the teacher critique an incorrect rollout so the student can rewrite its solution, with the teacher scoring the output that includes the critique as privileged information while the student trains on the revision without it. Across two Qwen3 student models and six mathematical reasoning benchmarks, CriticOPD improved over OPD, with the largest gains on competition problems; code is linked from the paper page.
It offers a concrete way to turn a teacher model's error recognition into reliable training signal, relevant to researchers working on distillation and weak-to-strong supervision.