GPT-6.1 Sol Tested on Silent Save Failures
A Reddit tester used 20 airline customer-service tasks from τ-bench to compare GPT-6.1 Sol, Claude Sonnet 5.5, and DeepSeek V4.1 Flash on detecting saves reported as successful despite making no changes. Raising GPT-6.1 Sol's reasoning effort increased usage from 22 to 939 tokens per conversation, but detected only one more failure, at 4 of 20 versus 3 of 20, while customer wait time rose from 33 to 58 seconds. The test also reported costs of about 3–4 cents per GPT-6.1 Sol conversation, compared with about 23 cents for Sonnet and under half a cent for DeepSeek.
The test highlights the trade-off between additional reasoning, detection performance, latency, and cost for agents handling operations that may fail silently.
| I tested whether AI agents notice when a save says “success” but nothing actually changed, and whether turning up reasoning effort helps. I used 20 airline customer service tasks from τ-bench and compared GPT-6.1 Sol with Claude Sonnet 5.5 and DeepSeek V4.1 Flash. Findings for GPT: If anyone's interested: Write-up: https://outsideengineer.substack.com/p/thinking-harder-doesnt-help-ai-catch Code and transcripts: https://github.com/brianna-rose05/thinking-harder-silent-failures submitted by /u/TruthDeep8659 [link] [comments] |