DeepSeek V4.1 Flash's Terminal-Bench score differs across model card, AA and Vals
A post on r/LocalLLaMA compares three Terminal-Bench 4.0 scores for DeepSeek-V4.1-Flash-1017: 42.3% reported on the model card, 34% from Artificial Analysis and 31.82% from the Vals index. The poster notes that the model card says the Claude Code harness was used and guesses that AA and Vals instead use mini-swe-agent, which he suggests could explain the lower results; he also points to the benchmark's eight-hour task limit and provider token throughput as possible sources of skew. Taking the 42.3% figure, he says the model would be on par with Fable 5, released June 9, 2026.
The post shows the same model's agentic benchmark score can differ substantially depending on the evaluation harness, though the poster's explanation is only a guess.
| DeepSeek-V4.1-Flash-1017's Terminal bench 4.0 score: - Model card reported 42.3% - AA reports 34% - Vals index reports 31.82% AA Vals Index Model card Model card quotes: We use the Claude Code harness for Terminal-Bench 4.0 I am guessing AA & Vals index uses mini-swe-agent thus causing a lower score, also each Terminal bench 4.0 task has a constraint of 8 hours, so box powering the agent & provider's tokens/sec during benchmark can skew results. Fable 5 score on Terminal bench 4.0: - AA reports 42% - Vals reports 41.41% If we take the score of 42.3 from model card, that puts DSv4.1 Flash-1017 on par with Fable 5 (released on June 9, 2026) submitted by /u/DigiDecode_ [link] [comments] |
#DeepSeek#DeepSeek-V4.1-Flash#Terminal-Bench 4.0#Artificial Analysis#Vals Index#Claude Code#Fable 5