FrontierHarness Eval v1.1比较五种模型的智能体框架
FrontierHarness与Baseten发布FrontierHarness Eval v1.1,评测覆盖22种框架配置、5个模型、62项终端任务及每项4次尝试。报告称,该基准使用830亿个token完成27,280次智能体运行,对比通过率与按官方价格计算的推理成本。按照评测采用的“成本与通过率”规则,不同模型对应不同的最佳性价比框架:GLM-5.3搭配Codex CLI,GLM-5.3-Flash搭配Pi,DeepSeek V4.1 Flash 1017搭配ZCode,Muse搭配Oh My Pi,Kimi-K3搭配Alma。OpenCode v2的平均单任务成本最低,为0.561美元;ZCode的通过率最高,为61.9%。报告估算,失败和超时的尝试占按官方价格计算推理成本的64.5%。
TL;DR
We partnered with Baseten to build the largest-scale harness leaderboard using 83B tokens across 27,280 agent runs. We wanted to answer two questions: when you switch your model, should you consider switching the harness too? What is the best harness for each model?
Previously, our v1.0 FrontierHarness leaderboard ranked harnesses on a single model, Kimi K3.
V1.1 expands the benchmark to 22 harnesses × 5 models × 62 tasks × 4 attempts, 76× the size of v1.0.
Through our experiments, we found that you should consider switching harnesses when switching models to maximize your experience.
Each model had a different best-value (pass rate vs. cost) harness: Codex CLI with GLM-5.3, Pi with GLM-5.3 Flash, ZCode with DeepSeek V4.1 Flash 1017, Oh My Pi with Muse and Alma with Kimi-K3. Cost ranks moved far between models.
部分来源内容未展示,可前往原文继续阅读。
#FrontierHarness Eval v1.1#Baseten#GLM-5.3#GLM-5.3-Flash#DeepSeek-V4.1-Flash-1017#Muse#Kimi-K3#OpenCode v2#ZCode#Codex CLI#Pi