10 coding agents tested on one local Qwen 3.8 27B
A Reddit user compared 10 coding agents using the same Qwen 3.8 27B Q4 model through llama.cpp on a MacBook Pro M5 Max with 64 GB of memory. Across five trap-containing tasks, every agent passed four or five tasks, while token consumption, completion time, cache reuse, and network behavior varied substantially; Pi used the fewest median tokens, while Qwen Code used the most. Codex CLI, Cline, OpenCode, and goose attempted an internet connection on every run, although the sandbox blocked those requests.
The setup highlights that local coding agents can have similar task-success rates while differing sharply in efficiency and offline behavior, which the report identifies as practical considerations for users selecting an agent.
Part 2 of my agent test (part 1 was a 4B model)
Hardware I used: MacBook Pro M5 Max, 64 GB. Qwen 3.8 27B Q4 on llama.cpp, thinking off.
Setup
- same model and server for everyone
- 5 small coding tasks, each with a little trap
- each agent locked in a sandbox: it can only touch its own folder and only talk to the model
- auto-approve on, telemetry off, 5 min limit
Results
| # | Agent | Passed | Median time | Median tokens | Cache reuse | Tried to go online |
|---|---|---|---|---|---|---|
| 1 | Pi | 5/5 | 38 s | 9.2k | 72% | no |
| 2 | DeepSeek Harness | 5/5 | 60 s | 35.3k | 83% | no |
| 3 | Codex CLI | 5/5 | 94 s | 36.3k | 79% | every run |
| 4 | OpenClaw | 5/5 | 106 s | 39.0k | 72% | unclear |
| 5 | Cline | 5/5 | 152 s | 29.6k | 79% | every run |
| 6 | OpenCode | 4/5 | 50 s | 32.4k | 75% | every run |
| 7 | Hermes | 4/5 | 80 s | 64.9k | 93% | no |
| 8 | Atomic Agent | 5/5 | 56 s | 30.4k | 86% | no |
| 9 | goose | 4/5 | 98 s | 20.3k | 76% | every run |
| 10 | Qwen Code | 4/5 | 183 s | 107.9k | 74% | unclear |
#Qwen 3.8 27B#llama.cpp#Pi#DeepSeek Harness#Codex CLI#OpenClaw#Cline#OpenCode#Hermes#Atomic Agent#goose#Qwen Code