创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

FrontierHarness Eval v1.1比较五种模型的智能体框架

AI 总结

FrontierHarness与Baseten发布FrontierHarness Eval v1.1,评测覆盖22种框架配置、5个模型、62项终端任务及每项4次尝试。报告称,该基准使用830亿个token完成27,280次智能体运行,对比通过率与按官方价格计算的推理成本。按照评测采用的“成本与通过率”规则,不同模型对应不同的最佳性价比框架:GLM-5.3搭配Codex CLI,GLM-5.3-Flash搭配Pi,DeepSeek V4.1 Flash 1017搭配ZCode,Muse搭配Oh My Pi,Kimi-K3搭配Alma。OpenCode v2的平均单任务成本最低,为0.561美元;ZCode的通过率最高,为61.9%。报告估算,失败和超时的尝试占按官方价格计算推理成本的64.5%。

TL;DR

We partnered with Baseten to build the largest-scale harness leaderboard using 83B tokens across 27,280 agent runs. We wanted to answer two questions: when you switch your model, should you consider switching the harness too? What is the best harness for each model?

Previously, our v1.0 FrontierHarness leaderboard ranked harnesses on a single model, Kimi K3.

V1.1 expands the benchmark to 22 harnesses × 5 models × 62 tasks × 4 attempts, 76× the size of v1.0.

Through our experiments, we found that you should consider switching harnesses when switching models to maximize your experience.

Each model had a different best-value (pass rate vs. cost) harness: Codex CLI with GLM-5.3, Pi with GLM-5.3 Flash, ZCode with DeepSeek V4.1 Flash 1017, Oh My Pi with Muse and Alma with Kimi-K3. Cost ranks moved far between models.

部分来源内容未展示,可前往原文继续阅读。

#FrontierHarness Eval v1.1#Baseten#GLM-5.3#GLM-5.3-Flash#DeepSeek-V4.1-Flash-1017#Muse#Kimi-K3#OpenCode v2#ZCode#Codex CLI#Pi

原文约 21 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。