Tülu 3.1 8B

Headlines · 1

1. Oct 20
   Ai2's FairStress benchmark tests how readily LLMs abandon correct decisions

   Ai2 (the Allen Institute for AI) introduced FairStress, a benchmark that measures demographic fairness in LLMs beyond the final choice a model makes, by testing how readily a model abandons a well-supported decision when pressed with persuasive arguments it calls adversarial prefixes. Ai2 says it evaluated 15 open-weight and frontier models, including Olmo, Llama, Qwen, Gemma, Claude and the GPT family, and reports an average gap in abandoning correct answers of 6.9 percentage points when demographic identity was implied and 9.3 points when it was explicitly labeled; on FairStress-Core, models including Claude Opus 5 and GPT-5.5 picked the minority-coded candidate about 60–65% of the time when neither candidate was objectively stronger. FairStress is available for download and is being built into Ai2's Olmo-eval evaluation workbench.

   Ai2 Blog · 🔥 0

Experience and discussion from the community

Share my Tülu 3.1 8B experienceAsk about Tülu 3.1 8B