DPO loss fell, but final-answer accuracy did not improve
The project author reports that DPO reduced validation loss from 0.628564 to 0.155683 and raised the validation preference score, but Qwen3-0.6B answered 29 of 32 controlled cases correctly versus 30 for the original model. The evaluation workflow included rule-based scoring, pairwise LLM judges, position-bias checks, human calibration and paired bootstrap; all six model comparisons were ultimately marked inconclusive. The 32-case study covered controlled chain-of-thought samples involving arithmetic and multistep numerical updates, and its human labels came from one annotator.
The report shows that training metrics are proxy measures rather than capability conclusions. Behavioral tests, evaluator-reliability checks and statistical uncertainty analysis are needed before claiming that post-training improved capability.
最近,我在给自己的手搓 LLM 项目补后训练。
做到 DPO 时,训练指标看起来非常舒服:
val loss: 0.628564 → 0.155683
val score: 1.3776 → 22.8115
loss 降了约 75%,偏好 score 也大幅上涨。
如果只看这两条曲线,我完全可以写一句:DPO 偏好优化取得了明显效果。
但把训练后的模型放到同一套行为测试里,结果却是:
#Qwen3-0.6B#DeepSeek#Qwen4 Max#EleutherAI lm-evaluation-harness#Inspect AI#OpenCompass#Promptfoo