Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

LLM Evaluators’ Credibility Judgments Are Sensitive to Source Attribution

AI summary

A preprint author tested whether LLMs evaluate identical arguments differently when only the attributed source changes. The experiment covered 20 arguments across 10 topics and four source conditions: relevant expert, irrelevant expert, non-expert and no source. GPT-5.6 Sol, Claude Opus 5.5, Gemini 4 Argon and DeepSeek Flash were evaluated on logic, credibility, persuasiveness and agreement. The author reports that credibility was by far the most source-sensitive dimension. In several models, attributing an argument to an expert in an unrelated domain reduced credibility relative to giving no source, while agreement was comparatively resistant to source attribution. Tests of whether source effects depended on argument strength were not statistically robust after correction, and human validation of strong versus weak arguments was imperfect, so the author treats that analysis as exploratory. The preprint has not been peer reviewed.

In-depth insight

The results suggest that source labels can influence LLM credibility judgments even when an attributed expert’s domain is irrelevant, raising a concern about bias in automated argument assessment.

I’m the author of a new preprint looking at a fairly simple question: if the argument itself stays the same, does changing the claimed source change how an LLM evaluates it?

I tested 20 arguments across 10 topics under four source conditions:

  • relevant expert
  • irrelevant expert
  • non-expert
  • no source

The arguments were evaluated by GPT-5.6 Sol, Claude Opus 5.5, Gemini 4 Argon, and DeepSeek Flash across four dimensions: logic, credibility, persuasiveness, and agreement.

A few findings stood out:

Credibility was by far the most source-sensitive dimension. Source attribution affected credibility much more consistently than logic or agreement.

#GPT-5.6 Sol#Claude Opus 5.5#Gemini 4 Argon#DeepSeek Flash

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.