Ai2 launches StateBench for long-context state tracking
Ai2 introduced StateBench, a benchmark with data and code for evaluating how language models track changing entity information across long documents. Its four difficulty levels test increasingly complex state updates; Ai2 reports that directly trained recurrent and hybrid models generalized farther than transformers, while Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna still failed on harder tasks as inputs grew longer.
StateBench provides a finer-grained way to diagnose where long-context models lose track of evolving information and to compare recurrent, hybrid and transformer architectures.
StateBench: Testing how language models track state changes over long documents
November 3, 2026
Ai2
We’re introducing StateBench, a benchmark for long-context state tracking—how well language models keep track of information about entities in long documents as they evolve.
#StateBench#Ai2#Gated DeltaNet#Olmo 3#Olmo Hybrid#Gemini 3.8 Flash#Claude Opus 5.5#GPT-6 Sol#GPT-6 Luna