MKA多模态知识智能体评测发布
Google DeepMind与Artificial Analysis推出Multimodal Knowledge Agent(MKA)评测,通过来自10个行业的200项私有视觉任务测试智能体,输入包括建筑图纸、工程原理图和临床扫描等。任务按平均约70项加权标准的专家量规评分,主要指标为Climb WSR,并由Artificial Analysis在具备代码执行、网页访问和图像查看工具的Stirrup环境中运行。Artificial Analysis称Claude Fable 5.2以57%的Climb WSR领先,Gemini 4 Carbon和Claude Opus 5.5分别为54%和53%,各模型的任务成本差异也很大。
MKA为需要理解复杂视觉文件并生成专业交付物的智能体提供了新的评测依据,其任务设计和成本对比可帮助从业者比较模型能力与效率。
We’ve partnered with Google DeepMind to launch their Multimodal Knowledge Agent (MKA) evaluation, a new benchmark for agentic knowledge work over complex visual files
@GoogleDeepMind’s MKA is a new benchmark requiring agents to review visually complex project files such as schematics and scans, and produce deliverables like slide decks, spreadsheets, reports, and marked-up images
Multimodal Knowledge Agent (MKA) overview:
➤ 200 private real-world visual tasks across 10 industries: Agents work from dense visual project files including architectural drawings, engineering schematics, and clinical scans to produce deliverables including slide decks, spreadsheets, PDFs, Word documents, and marked-up images. Unlike many agentic evaluations, these inputs emphasize realistic visual complexity and domain-specificity, and are genuinely required to complete the task rather than just providing additional context
➤ Climb vs. Regression rubric scoring: Each task is graded against expert rubrics, averaging ~70 weighted criteria per task. Criteria are split into ‘Regression’ (baseline checks models should pass) and ‘Climb’ criteria (harder checks designed to separate strong models). The headline score is Climb WSR (Weighted Satisfaction Rate) - the weighted proportion Climb rubric criteria successfully completed
#Google DeepMind#Artificial Analysis#Multimodal Knowledge Agent (MKA)#Stirrup#Claude Fable 5.2#Gemini 4 Carbon#Claude Opus 5.5#GPT-6.1 Sol#GPT-6 Luna