Explainer: How RLHF Trains Models from Human Preferences
A Juejin AI tutorial walks through the full RLHF pipeline: SFT cold start with roughly 13,000 prompts, training a reward model from ranked human preferences on about 33,000 prompts, and PPO reinforcement learning that scores complete generations with the reward model. It explains the Bradley-Terry loss used to fit the reward model, the score head swapped in for the vocabulary projection, and the four models loaded during PPO (Actor, Reward, Reference, Critic), including the KL penalty that limits reward hacking. It cites OpenAI's 2022 InstructGPT paper, in which a 1.3B InstructGPT reportedly outperformed the 175B GPT-3 on human preference evaluations, and closes by previewing DPO and DeepSeek's GRPO as simpler alternatives.
It consolidates the technical mechanics behind preference alignment — reward-model training, PPO's four-model setup, and KL-based reward-hacking mitigation — into one Chinese-language explainer with code sketches for practitioners building aligned models.
在上一篇文章中,我们学习了如何用标注好的问答数据对预训练模型做监督微调,还学习了 LoRA 这种只训练一小部分参数的高效微调方式。经过 SFT,模型学会了模仿标准答案的格式和口吻,从一个只会续写文本的底座模型,变成了一个能按指令回答问题的助手。
不过 SFT 有一个绕不开的局限:它教模型模仿的标准答案只有一个。但同一个问题往往有很多种回答方式,有的准确有的含糊,有的有用有的无聊,甚至有的在胡说八道。好坏之间的差别很微妙,很难写成一条条标准答案让模型去背。还有一点更棘手:对于那些不该回答的有害请求,SFT 最多靠几条例拒样本让模型学着拒绝,模型并不知道拒绝的边界在哪里。对于这些问题,2022 年 OpenAI 发表的 InstructGPT 论文给出了一条新路:不再让模型模仿标准答案,而是让人类告诉模型哪个回答更好,用这个偏好信号来训练模型。这就是 RLHF(Reinforcement Learning from Human Feedback,基于人类反馈的强化学习),也就是用人类偏好作为奖励信号、再用强化学习去优化模型的方法。ChatGPT 之所以在 2022 年底一鸣惊人,背后的关键一步正是它。今天我们就把 RLHF 的完整流程讲清楚。
#OpenAI#InstructGPT#DeepSeek#DPO#PPO#Bradley-Terry model#TRL