Grok 4.20 reportedly loops for hundreds of turns in Pokémon Red
PokeBench author /u/VibeCodyH says Grok 4.20 in a non-reasoning configuration left Red's house at turn 36 but did not obtain a starter until turn 755. The model's notes reportedly described more than 90 turns of oscillation, repeated wall collisions and a persistent loop, while it continued walking into walls. The author says Grok 4.6 beat Brock in 316 turns for about $10, compared with about $30 for Grok 4.20, which barely obtained its starter.
The reported logs provide a concrete agent-failure example: a model identified its own repeated loop without escaping it, while showing slower progress and higher reported cost than Grok 4.6.
I've been reading the turn logs from PokeBench. My benchmark where vision models play Pokemon Red. Grok 4.20 non-reasoning left the house at turn 36 and didn't get a starter until turn 755. The whole time its notes say things like "We've been oscillating uselessly for 90+ turns" and later "The persistent loop of walking into walls and Mom's dialogue has continued for hundreds of turns." Then it walks into the wall again.
So it knows it's stuck and still can't get out. Grok 4.6 beat Brock in 316 turns for about $10, while 4.20 burned $30 and barely got its starter.
Why do you think a model can describe its own loop correctly and still not break it? Vision problem or planning problem?
submitted by /u/VibeCodyH [link] [comments]