创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

MKA多模态知识智能体评测发布

AI 总结

Google DeepMind与Artificial Analysis推出Multimodal Knowledge Agent(MKA)评测,通过来自10个行业的200项私有视觉任务测试智能体,输入包括建筑图纸、工程原理图和临床扫描等。任务按平均约70项加权标准的专家量规评分,主要指标为Climb WSR,并由Artificial Analysis在具备代码执行、网页访问和图像查看工具的Stirrup环境中运行。Artificial Analysis称Claude Fable 5.2以57%的Climb WSR领先,Gemini 4 Carbon和Claude Opus 5.5分别为54%和53%,各模型的任务成本差异也很大。

深度见解

MKA为需要理解复杂视觉文件并生成专业交付物的智能体提供了新的评测依据,其任务设计和成本对比可帮助从业者比较模型能力与效率。

We’ve partnered with Google DeepMind to launch their Multimodal Knowledge Agent (MKA) evaluation, a new benchmark for agentic knowledge work over complex visual files

@GoogleDeepMind’s MKA is a new benchmark requiring agents to review visually complex project files such as schematics and scans, and produce deliverables like slide decks, spreadsheets, reports, and marked-up images

Multimodal Knowledge Agent (MKA) overview:

➤ 200 private real-world visual tasks across 10 industries: Agents work from dense visual project files including architectural drawings, engineering schematics, and clinical scans to produce deliverables including slide decks, spreadsheets, PDFs, Word documents, and marked-up images. Unlike many agentic evaluations, these inputs emphasize realistic visual complexity and domain-specificity, and are genuinely required to complete the task rather than just providing additional context

➤ Climb vs. Regression rubric scoring: Each task is graded against expert rubrics, averaging ~70 weighted criteria per task. Criteria are split into ‘Regression’ (baseline checks models should pass) and ‘Climb’ criteria (harder checks designed to separate strong models). The headline score is Climb WSR (Weighted Satisfaction Rate) - the weighted proportion Climb rubric criteria successfully completed

#Google DeepMind#Artificial Analysis#Multimodal Knowledge Agent (MKA)#Stirrup#Claude Fable 5.2#Gemini 4 Carbon#Claude Opus 5.5#GPT-6.1 Sol#GPT-6 Luna

原文约 2 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。