Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Tool Use Nearly Doubles VIALS-AA Scores in Artificial Anlys Report

AI summary

Artificial Anlys reports that tool use nearly doubled average VIALS-AA scores across models, from about 25% to 47%. Claude Fable 5.2 with tools led at 67% and $4.20 per task, but scored 44% without tools at $0.73 per task. Claude Haiku 5.5 with tools scored 49% at $0.07 per task. VIALS-AA is Artificial Anlys' implementation of JoinHandshake AI's benchmark for visual reasoning in life-sciences research.

In-depth insight

The report says lower-cost models in an agent harness can outperform single-shot large models on both cost and performance for visual reasoning and scientific-artifact interpretation. In its test, Haiku 5.5 with tools scored higher than Fable 5.2 without tools while costing about one-tenth as much per task.

Take away its tools, and Claude Fable 5.2 scores below Claude Haiku 5.5 with tools on VIALS-AA; across models tool use nearly doubles scores, from ~25% to ~47% on average

VIALS-AA is our implementation of @joinHandshakeAI's VIALS benchmark for visual reasoning in life sciences research. By default, models work as agents: they think, run code and view images in a sandbox. We reran VIALS-AA in a one-shot format: one request with the question and images, and no tools.

Lower cost models in an agent harness can beat single-shot large models on both cost and performance in visual reasoning and interpretation of scientific artifacts.

Claude Fable 5.2 (max) leads VIALS-AA at 67% with tools, at a cost of $4.20 per task; without tools it scores 44% at $0.73 per task. Claude Haiku 5.5 (max) scores 49% with tools, at $0.07 per task: higher than Fable without tools, at about a tenth of the cost.

#Claude Fable 5.2#Claude Haiku 5.5#Anthropic#VIALS-AA#VIALS#JoinHandshake AI

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.