Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

DPO loss fell, but final-answer accuracy did not improve

AI summary

The project author reports that DPO reduced validation loss from 0.628564 to 0.155683 and raised the validation preference score, but Qwen3-0.6B answered 29 of 32 controlled cases correctly versus 30 for the original model. The evaluation workflow included rule-based scoring, pairwise LLM judges, position-bias checks, human calibration and paired bootstrap; all six model comparisons were ultimately marked inconclusive. The 32-case study covered controlled chain-of-thought samples involving arithmetic and multistep numerical updates, and its human labels came from one annotator.

In-depth insight

The report shows that training metrics are proxy measures rather than capability conclusions. Behavioral tests, evaluator-reliability checks and statistical uncertainty analysis are needed before claiming that post-training improved capability.

最近,我在给自己的手搓 LLM 项目补后训练。

做到 DPO 时,训练指标看起来非常舒服:

val loss:  0.628564 → 0.155683
val score: 1.3776   → 22.8115

loss 降了约 75%,偏好 score 也大幅上涨。

如果只看这两条曲线,我完全可以写一句:DPO 偏好优化取得了明显效果。

但把训练后的模型放到同一套行为测试里,结果却是:

#Qwen3-0.6B#DeepSeek#Qwen4 Max#EleutherAI lm-evaluation-harness#Inspect AI#OpenCompass#Promptfoo

Original · 11 min readRead the original
Group chat

No comments yet. Start the conversation.