KLPO

Headlines · 1

1. 3d
   KLPO: a critic-free, KL-regularized policy optimization method for LLM agents

   A new arXiv paper, "On KL-Regularized Policy Optimization" by Yifan Zhang (submitted 6 Oct 2026), proposes KLPO, which anchors the KL regularizer at the sampler so asynchronous RL for LLM agents can train on trajectories from stale checkpoints and mismatched inference probabilities without importance weights. The paper says the regularized improvement step has a closed-form Gibbs solution, and that for token-level policy mirror descent targets the gradient can be computed from terminal returns without a critic, using one rollout per prompt. It further proves independent Monte Carlo estimates of the KL term keep gradients unbiased and shows SPPO, GPO, REBEL and BPO arise as special cases of KLPO.

   Hugging Face · Papers · 🔥 0

Experience and discussion from the community

Share my KLPO experienceAsk about KLPO