Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

SubQ Snap 1.2 gains five points on Decision Index v2.1

AI summary

Artificial Analysis reports that SubQ Snap 1.2 scored 73 on Decision Index v2.1, up from 68 for SubQ Snap 1.1. Drex Max v1.1 and GLiDE v1.2 are also listed at 73, though the report says the three models are within 0.1 points, so an exact tie is not established. The report states that pricing is unchanged at $0.04 per million input tokens and that the model supports a 384K-token context window.

In-depth insight

The update produced the largest gains on AA-Paperclip and AA-Symmetry, both at eight points. Paperclip's all-pass rate rose from 46% to 62%, while gains on AA-Omniscience-Decision and HLE-MultiChoice came from higher abstention rather than higher accuracy.

Subquadratic’s SubQ Snap 1.2 gains 5 points over SubQ Snap 1.1 on the Artificial Analysis Decision Index v2.1, and is tied with Drex Max v1.1 and GLiDE v1.2 as the leading decision model

SubQ Snap 1.2 is the latest decision model from @subquadratic. Pricing is unchanged at $0.04 per 1M input tokens, and it supports a 384K token context window.

Decision Index v2.1 updates AA-Symmetry to v1.1. AA-Symmetry measures whether a model keeps the same decision and confidence when a question changes in ways that should not matter. v1.1 moves the answer options into the state and rewords them with the same meaning.

Key takeaways:

➤ SubQ Snap 1.2 scores 73 on the Artificial Analysis Decision Index v2.1, up from 68 for SubQ Snap 1.1: Drex Max v1.1 (73) and GLiDE v1.2 (73) are within 0.1 points. The largest gains are on AA-Paperclip (+8 points) and AA-Symmetry (+8 points).

➤ The AA-Paperclip gain comes from more documents answered fully correctly: the all-pass rate rises from 46% to 62%, level with Drex Max v1.1 (63%) and GLiDE v1.2 (62%).

➤ The gains on AA-Omniscience-Decision and HLE-MultiChoice come from abstention, not accuracy: non-hallucination rates rise from 85% to 89% and from 46% to 55%. Accuracy is flat at 21% and 9%. On AA-Omniscience-Decision, SubQ Snap 1.2 attempts 29% of questions, down from 33%.

#Subquadratic#SubQ Snap 1.2#SubQ Snap 1.1#Drex Max v1.1#GLiDE v1.2

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.