LLM裁判的写作评估能力与通用能力关联较弱
一项被NeurIPS 2026“Can We Trust AI Evaluation?”研讨会接收的研究,在三项长文本写作任务中将LLM裁判与人类偏好进行了比较。研究使用四个LLM生成文本,结果显示裁判质量会随任务和评估标准变化,与MMLU表现仅呈弱相关;成对比较表现更稳定,而能力最强的LLM也未能始终复现人类判断。
研究结果提示,在其测试环境中,不应仅凭模型通用能力选择LLM裁判,并支持采用更一致的成对比较方式。
content type paperpublished November 2026
Evaluating the Evaluators: Investigating LLM Judges for Personalized Writing Style Assessment
AuthorsNatalie Mackraz‡, Andrew Silva, Aini Putkonen, Brihi Joshi†**, Joelle Alcaidinho, Arnav Arora§**, Barry-John Theobald, Katherine Metcalf‡
This paper was accepted at the Can We Trust AI Evaluation? Workshop at NeurIPS 2026.
We evaluate how good large language model (LLM) judges are for personalized writing tasks. This is important when a user’s preferred writing style is known (e.g., “upbeat”) and an LLM judge is used to evaluate whether generated text adheres to this preference. We examine how the writing task, the combination of generator and judge LLM, the evaluation objective, and general commonsense and reasoning ability impact LLM judge performance. To this end, we generate text with four LLMs for three long-form writing tasks, collect human labels evaluating which of two responses is better, collect judge-LLM, pairwise-ranking annotations, and then compare the human and judge LLM annotations. We find that judge quality correlates weakly with general LLM ability measured using MMLU (Pearson’s ρ ≤ 0.2) and varies by both writing task and evaluation criteria (i.e., general helpfulness versus style personalization). We also find that LLM evaluators are more consistent and performant using paired comparisons rather than scoring responses independently. Finally, we find that the “strongest” LLMs do not most reliably reproduce the human evaluations, which we hypothesize relates to exaggerated sensitivity to subtle details that human annotators and weaker LLMs ignore.