Reality check: self-distillation methods in agentic RL show limited gains over GRPO
A new arXiv paper offers a reality check on self-distillation in agentic reinforcement learning. The authors compared ten representative methods against a GRPO baseline on ALFWorld, ScienceWorld, Search-QA and WebShop across three model sizes, running 132 training runs that totaled 12,124.4 recorded H800 GPU-hours, with repeated evaluations over eight rollout seeds for the pretrained backbones and all eleven trained methods. They report three findings: gains over GRPO remain limited in scope, with no method consistently outperforming it; strong final performance can mask unstable training; and additional training cost does not consistently translate into better performance.
The benchmark gives agent-training research a shared basis for comparing methods that accounts for cost and training stability, not just final scores.