Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

GPT-6.1 Sol Tested on Silent Save Failures

AI summary

A Reddit tester used 20 airline customer-service tasks from τ-bench to compare GPT-6.1 Sol, Claude Sonnet 5.5, and DeepSeek V4.1 Flash on detecting saves reported as successful despite making no changes. Raising GPT-6.1 Sol's reasoning effort increased usage from 22 to 939 tokens per conversation, but detected only one more failure, at 4 of 20 versus 3 of 20, while customer wait time rose from 33 to 58 seconds. The test also reported costs of about 3–4 cents per GPT-6.1 Sol conversation, compared with about 23 cents for Sonnet and under half a cent for DeepSeek.

In-depth insight

The test highlights the trade-off between additional reasoning, detection performance, latency, and cost for agents handling operations that may fail silently.

I tested whether AI agents notice when a save says “success” but nothing actually changed, and whether turning up reasoning effort helps. I used 20 airline customer service tasks from τ-bench and compared GPT-6.1 Sol with Claude Sonnet 5.5 and DeepSeek V4.1 Flash. Findings for GPT: If anyone's interested: Write-up: https://outsideengineer.substack.com/p/thinking-harder-doesnt-help-ai-catch Code and transcripts: https://github.com/brianna-rose05/thinking-harder-silent-failures submitted by /u/TruthDeep8659 [link] [comments]

#GPT-6.1 Sol#Claude Sonnet 5.5#DeepSeek V4.1 Flash

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.