An arXiv preprint introduces ActionOPD, an action-level on-policy distillation method for language-model agents. At states the student actually encounters, it samples actions from both the student and a black-box teacher and rewards student actions by their semantic agreement with the sampled teacher actions, aligning behavior without needing teacher token probabilities or a shared tokenizer. Training Qwen3-4B and Qwen3-8B with two teachers on curated tool-use tasks yields average gains of 3.3–6.9 percentage points across four out-of-distribution benchmarks, and on τ²-bench it is competitive with offline distillation and token-level OPD; the authors also explore a Self-ActionOPD variant using a same-model teacher with privileged context.

ActionOPD·Qwen3-4B·Qwen3-8B·+2