Ai2's FairStress benchmark tests how readily LLMs abandon correct decisions
Ai2 (the Allen Institute for AI) introduced FairStress, a benchmark that measures demographic fairness in LLMs beyond the final choice a model makes, by testing how readily a model abandons a well-supported decision when pressed with persuasive arguments it calls adversarial prefixes. Ai2 says it evaluated 15 open-weight and frontier models, including Olmo, Llama, Qwen, Gemma, Claude and the GPT family, and reports an average gap in abandoning correct answers of 6.9 percentage points when demographic identity was implied and 9.3 points when it was explicitly labeled; on FairStress-Core, models including Claude Opus 5 and GPT-5.5 picked the minority-coded candidate about 60–65% of the time when neither candidate was objectively stronger. FairStress is available for download and is being built into Ai2's Olmo-eval evaluation workbench.
Ai2 argues a model's fairness cannot be judged solely by whether it reaches the same final choice across otherwise equivalent scenarios, so the open benchmark gives researchers and model builders a tool to diagnose hidden asymmetries and test proposed fixes.
FairStress: A deeper evaluation of fairness in LLM decision-making
October 20, 2026
Ai2
Today we’re introducing FairStress, a benchmark for measuring demographic fairness in LLMs beyond the choice a model ultimately makes in decisions impacting people. FairStress is available for download, and we’re also building it into Olmo-eval, our open workbench for running and comparing model evaluations.
#Ai2#FairStress#Olmo-eval#Olmo#Llama#Claude Opus 5#GPT-5.5#Qwen2.5-7B-Instruct#Gemma-3-12B-it#Tülu 3.1 8B