Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

FrontierHarness Eval v1.1 compares agent harnesses across five models

AI summary

FrontierHarness and Baseten published FrontierHarness Eval v1.1, a benchmark covering 22 harness configurations, five models, 62 terminal tasks and four attempts per task. The report says it used 83 billion tokens across 27,280 agent runs, comparing pass rates and list-price inference costs. It found that each model had a different best-value harness under the benchmark’s cost-versus-pass-rate rule: Codex CLI with GLM-5.3, Pi with GLM-5.3-Flash, ZCode with DeepSeek V4.1 Flash 1017, Oh My Pi with Muse and Alma with Kimi-K3. OpenCode v2 had the lowest average cost per task at $0.561, while ZCode had the highest pass rate at 61.9%. The report estimated that failed and timed-out trials accounted for 64.5% of list-price inference costs.

TL;DR

We partnered with Baseten to build the largest-scale harness leaderboard using 83B tokens across 27,280 agent runs. We wanted to answer two questions: when you switch your model, should you consider switching the harness too? What is the best harness for each model?

Previously, our v1.0 FrontierHarness leaderboard ranked harnesses on a single model, Kimi K3.

V1.1 expands the benchmark to 22 harnesses × 5 models × 62 tasks × 4 attempts, 76× the size of v1.0.

Through our experiments, we found that you should consider switching harnesses when switching models to maximize your experience.

Each model had a different best-value (pass rate vs. cost) harness: Codex CLI with GLM-5.3, Pi with GLM-5.3 Flash, ZCode with DeepSeek V4.1 Flash 1017, Oh My Pi with Muse and Alma with Kimi-K3. Cost ranks moved far between models.

Some source content is omitted. Continue reading on the original site.

#FrontierHarness Eval v1.1#Baseten#GLM-5.3#GLM-5.3-Flash#DeepSeek-V4.1-Flash-1017#Muse#Kimi-K3#OpenCode v2#ZCode#Codex CLI#Pi

Original · 21 min readRead the original
Group chat

No comments yet. Start the conversation.