Waypoint-OPD gates policy-shift distillation with a mid-training teacher
An arXiv paper introduces Waypoint-OPD, a new take on policy-shift distillation, which reuses the difference between a teacher model's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories. The authors argue that this endpoint shift alone cannot show whether local action preferences evolved consistently during teacher training, since the same endpoint difference can come from early and late shifts that reinforce or partly cancel each other. Waypoint-OPD therefore uses an intermediate teacher checkpoint from the same RL run to compare early and late shifts in relative action preferences at student-visited prefixes, keeping the endpoint shift only when the two point in the same direction and suppressing it when they oppose; this adds just one extra frozen-teacher scoring pass. Using RL-trained Qwen3-4B teachers with Qwen3-8B, Qwen3-14B and Qwen3-30B-A3B students on mathematical and code reasoning, the method posts the highest domain-average score among compared distillation methods in all six domain-student settings, and ablations with the Qwen3-8B student show that removing or shuffling the gates degrades math reasoning.
It offers a low-overhead idea for making RL-derived capability transfer to smaller models more reliable, though the evidence so far comes from a single non-peer-reviewed paper limited to the Qwen3 family and math/code tasks.