Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Google DeepMind and Artificial Analysis launch MKA benchmark

AI summary

Google DeepMind and Artificial Analysis launched the Multimodal Knowledge Agent (MKA) evaluation, which tests agents on 200 private visual tasks drawn from 10 industries, including architectural drawings, engineering schematics, and clinical scans. Tasks are scored against expert rubrics averaging about 70 weighted criteria, with Climb WSR as the headline metric; Artificial Analysis runs them in its Stirrup harness with code execution, web access, and image-viewing tools. Artificial Analysis reports Claude Fable 5.2 leading with a 57% Climb WSR, followed by Gemini 4 Carbon at 54% and Claude Opus 5.5 at 53%, alongside substantial differences in cost per task.

In-depth insight

MKA provides a new source of benchmark evidence for agents that must interpret dense visual files and produce professional deliverables, while its task set and cost comparisons may help practitioners compare model capability and efficiency.

We’ve partnered with Google DeepMind to launch their Multimodal Knowledge Agent (MKA) evaluation, a new benchmark for agentic knowledge work over complex visual files

@GoogleDeepMind’s MKA is a new benchmark requiring agents to review visually complex project files such as schematics and scans, and produce deliverables like slide decks, spreadsheets, reports, and marked-up images

Multimodal Knowledge Agent (MKA) overview:

➤ 200 private real-world visual tasks across 10 industries: Agents work from dense visual project files including architectural drawings, engineering schematics, and clinical scans to produce deliverables including slide decks, spreadsheets, PDFs, Word documents, and marked-up images. Unlike many agentic evaluations, these inputs emphasize realistic visual complexity and domain-specificity, and are genuinely required to complete the task rather than just providing additional context

➤ Climb vs. Regression rubric scoring: Each task is graded against expert rubrics, averaging ~70 weighted criteria per task. Criteria are split into ‘Regression’ (baseline checks models should pass) and ‘Climb’ criteria (harder checks designed to separate strong models). The headline score is Climb WSR (Weighted Satisfaction Rate) - the weighted proportion Climb rubric criteria successfully completed

#Google DeepMind#Artificial Analysis#Multimodal Knowledge Agent (MKA)#Stirrup#Claude Fable 5.2#Gemini 4 Carbon#Claude Opus 5.5#GPT-6.1 Sol#GPT-6 Luna

Original · 2 min readRead the original
Group chat

No comments yet. Start the conversation.