创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

Artificial Analysis发布决策指数v2

AI 总结

Artificial Analysis发布Decision Index v2,新版Decision Score同时考量答案正确性与置信度校准。私有数据集占比从v1的25%升至45%。新版加入包含1500份文档的私有基准AA-Paperclip和一致性测试AA-Symmetry,增加EvalSafe-ONET-AA,因饱和度较高移除WildGuardTest,并将When2Call权重从25%降至10%。Claude Haiku 5.5(Max)与GPT-6 Luna(Max)分别获得77分和76分,专业决策模型Drex Max v1.1与GLiDE v1.2均以73分并列。GPT-6 Luna每百万项任务成本为1530美元,单项耗时11.34秒;Drex Max v1.1分别为102美元和275毫秒。GLiDE v1.2每百万项任务成本为780美元,接近Drex Max v1.1的八倍,原因是其每道题都会重新计算文档输入费用。

Introducing the Artificial Analysis Decision Index v2: a new proprietary evaluation, updated benchmarks, and a headline metric that rewards model calibration

We’ve released the second version of the Artificial Analysis Decision Index to better reflect real-world utility and guard against benchmark gaming. 45% of the Index is now built on private datasets, up from 25% in v1. Drex Max v1.1 from @NaceAI narrowly leads among decision models at 73, within 1 point of GLiDE v1.2 (@fastinoAI) and Mammoth 27B (@baselabs)

Updates in Decision Index v2:

➤ Decision Score: We adopt a new headline metric that rewards a model for both correctness and calibrated confidence. Each question scores from 0 to 100: right answers score above 50, wrong answers below 50, and confidence in its top answer moves the score within each half

➤ AA-Paperclip (30%): We added a new private benchmark of 1,500 documents including resumes, receipts, invoices, and AI and machine learning research papers. Models make decisions and extract facts from each document and are scored jointly as a single decision, correct only if all 15 answers are right

#Artificial Analysis#Artificial Analysis Decision Index v2#AA-Paperclip#AA-Symmetry#Claude Haiku 5.5#GPT-6 Luna#Drex Max v1.1#GLiDE v1.2#Mammoth 27B#Nace AI#Fastino Labs#Baseten

原文约 2 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。