GPT-6.1 Sol trails rivals in personal Salesforce agent test
AI summary
A Salesforce developer tested GPT-6.1 Sol, Claude Opus, Claude Sonnet, Claude Haiku and DeepSeek on 50 agent tasks, grading each attempt by actual database changes rather than model responses. GPT-6.1 Sol scored 72%, behind Opus at 87%, Sonnet at 85%, DeepSeek at 82% and Haiku at 80%. It caught every probability reset after a stage change and was the best at reporting automatically made changes in long multi-step jobs, succeeding in 21 of 36. The tester attributed much of the gap to failures to restore original close dates or disclose outdated dates when reopening deals.
| I've been working on a project testing AI agents in a Salesforce dev org and wanted to share how GPT-6.1 Sol did, since the results were mixed. Five models (GPT-6.1 Sol, Claude Opus, Sonnet, Haiku and DeepSeek) worked through 50 tasks, and each attempt was graded on what actually changed in the database, not on the model's reply. Some of the findings for GPT-6.1 Sol: If anyone's interested: Write-up: https://outsideengineer.substack.com/p/evaluating-ai-agents-on-salesforce Code and transcripts: https://github.com/brianna-rose05/crm-agent-bench submitted by /u/TruthDeep8659 [link] [comments] |
#GPT-6.1 Sol#Claude Opus#Claude Sonnet#Claude Haiku#DeepSeek#Salesforce