Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Surge AI launches DAYJOB: Tax benchmark

AI summary

Surge AI introduced DAYJOB: Tax, a benchmark with 50 expert-developed environments covering eight areas of professional tax work, including preparation, compliance, advisory, review, and disputes. In vendor-reported results using a 95% rubric threshold, Claude Sonnet 5.5 led with a 10.8% pass rate, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%. A condo-sale case showed Astra missing a consequential eligibility deadline in all five attempts, while Opus identified it in all five.

In-depth insight

The benchmark tests whether AI agents can independently identify consequential issues and complete end-to-end tax workflows, rather than merely answer isolated questions or perform calculations.

Anna was planning to sell her condo. She had moved out in August 2024 and wanted to know whether closing in October 2026 or January 2027 would be better for taxes.

An AI agent reviewed both options and correctly concluded that either date would satisfy the residence requirement for a tax break worth up to $250,000 in excluded gains.

But it left out an important deadline.

Under the usual rules, homeowners must have lived in a property for at least two of the five years before selling it to qualify for the exclusion. Because Anna moved out in August 2024, she would stop satisfying that requirement around the end of August 2027. If her sale was delayed beyond August 2027, she could lose eligibility.

Across five attempts, Astra never identified the deadline. Opus identified it in all five attempts.

The agent evaluated the proposed dates correctly, but its advice left Anna unaware of a deadline that could affect a major financial decision. A tax adviser should recognize consequential issues even when the client doesn't know to ask about them.

#Surge AI#DAYJOB: Tax#Claude Sonnet 5.5#Claude Fable 5.1#Claude Opus 5.5#Astra

Original · 6 min readRead the original
Group chat

No comments yet. Start the conversation.