Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

A practical guide to DPO and GRPO for preference alignment

AI summary

A Juejin AI tutorial explains how DPO simplifies RLHF by removing the reward model and online reinforcement-learning loop, while GRPO retains reinforcement learning but replaces the value model with group-relative rewards. It also demonstrates DPO training with Hugging Face TRL and DPOTrainer, then connects GRPO with verifiable rewards to the training of DeepSeek-R1-style reasoning models. The article contrasts the engineering trade-offs of the two approaches and surveys tools including TRL, veRL, OpenRLHF, vLLM and SGLang.

In-depth insight

The guide gives practitioners a concrete comparison and implementation path for two widely used preference-alignment approaches, including configuration examples and the task boundaries of GRPO with RLVR.

上一篇我们学习了 RLHF 的完整流程:先训一个奖励模型(Reward Model) 来模拟人类的偏好打分,再用 PPO(Proximal Policy Optimization,近端策略优化) 算法让模型在奖励信号的引导下更新参数。这套流程由 OpenAI 在 InstructGPT 上跑通,是大模型从会续写变成会对话的关键一步。

不过上一篇结尾也提到,PPO 这条路走起来并不轻松。今天我们要看的,是偏好对齐在 PPO 之后演化出的两条新路线:一条走向简化,把奖励模型和强化学习整个绕过去,这就是 DPO;另一条走向强化,把 PPO 里最贵的部件砍掉,再配上不需要奖励模型的任务奖励,最终训出了 DeepSeek-R1 这类推理模型,这就是 GRPO 和 RLVR。

PPO 痛点回顾

上一篇的结尾我们总结了一些 PPO 的工程痛点,这里回顾一下:

  • 显存翻倍再翻倍:训练时要同时在显存里放四个模型,正在训练的策略模型和价值模型,加上冻结的参考模型和奖励模型,每个模型都要占一份显存
  • 超参极其敏感:学习率、KL 惩罚系数、clip 范围、batch 大小,每一项调不好都会明显影响效果
  • 训练不稳定:强化学习本身就以难调著称,PPO 叠加在大模型上之后,loss 曲线抖动、训练中途崩掉都是家常便饭
  • 链路长、环节多:SFT、奖励模型、PPO 是三次完整的训练,任何一步没训好都会把误差传导下去

#Direct Preference Optimization#Group Relative Policy Optimization#Reinforcement Learning with Verifiable Rewards#Proximal Policy Optimization#DeepSeek-R1#InstructGPT#Hugging Face TRL#DeepSeekMath#Qwen3-0.6B#veRL#OpenRLHF#Tulu 3

Original · 15 min readRead the original
Group chat

No comments yet. Start the conversation.