LLM评判者的可信度判断易受来源归属影响
一项预印本实验测试:当论点内容保持不变、仅改变其署名来源时,大语言模型是否会作出不同评价。实验涵盖10个话题、20个论点和四种来源条件,包括相关领域专家、无关领域专家、非专家和不标明来源。GPT-5.6 Sol、Claude Opus 5.5、Gemini 4 Argon与DeepSeek Flash均从逻辑、可信度、说服力和认同度四个维度接受评估。作者报告称,可信度是受来源归属影响最明显的维度。若干模型中,将论点归于无关领域专家会降低其可信度,甚至低于不标明来源的情况,而认同度相对不易受来源影响。关于来源效应是否取决于论点强度的检验,经校正后不够稳健,且人类对强、弱论点的验证并不完善,因此作者将该部分视为探索性结果。该预印本尚未经过同行评审。
实验结果表明,即使被署名专家的领域与论点无关,来源标签仍可能影响LLM的可信度判断,因而引发对自动化论点评估偏差的担忧。
I’m the author of a new preprint looking at a fairly simple question: if the argument itself stays the same, does changing the claimed source change how an LLM evaluates it?
I tested 20 arguments across 10 topics under four source conditions:
- relevant expert
- irrelevant expert
- non-expert
- no source
The arguments were evaluated by GPT-5.6 Sol, Claude Opus 5.5, Gemini 4 Argon, and DeepSeek Flash across four dimensions: logic, credibility, persuasiveness, and agreement.
A few findings stood out:
Credibility was by far the most source-sensitive dimension. Source attribution affected credibility much more consistently than logic or agreement.