Blind test compares five self-hosted AI agents on Qwen 3.8 27B
Reddit user /u/GapNew4766 reported a scripted blind test of OpenCode, goose, Atomic Agent, Hermes and OpenClaw, all running locally on an M5 Max MacBook Pro with 64GB of memory. Each used an uncensored Qwen 3.8 27B model in llama.cpp and completed the same five tasks, including merging contact lists, renaming a function across three files, fixing a bug, counting log errors and converting a receipt PDF into a spreadsheet. All five passed three tasks, while four passed the bug-fix and receipt-conversion tasks; the agent-to-label mapping remained hidden at publication.
The report provides task-level results and timings under a shared local-model setup, offering a practical comparison of five self-hosted AI agents, although it comes from one author and covers only five tasks.
everything here ran on my own hardware, an M5 Max MacBook Pro with 64GB. blind test of self-hosted AI agents, every one on Qwen 3.8 27B uncensored Q4_K_M in llama.cpp, same 5 everyday tasks like counting log errors and renaming a function. names hidden, results below, take a guess
The tasks
- Merge two contact lists, messy emails, two different Alex Kims. All 5 got it right (46 people).
- Rename a function across 3 files with messy spacing. All 5 passed.
- Fix a bug, with hidden tests. 4 passed. C turned "Q3/Q4" into "q3q4" instead of "q3-q4".
- Count the errors in 3 log files. All 5 got 43.
- Turn a receipt PDF into a spreadsheet. 4 passed. A and B installed a missing Python library without asking. C stopped and asked "can I install this?", nobody answered, so it failed. In a test with nobody watching, that counts as a fail. On my actual hardware, that's the behavior I'd want.
| Agent | 1. Contacts | 2. Rename | 3. Bug fix | 4. Errors | 5. Receipt | Total |
|---|---|---|---|---|---|---|
| A | ✅ 4:54 | ✅ 0:28 | ✅ 0:43 | ✅ 0:26 | ✅ 3:58 | 10:29 |
| B | ✅ 6:03 | ✅ 1:11 | ✅ 0:55 | ✅ 0:54 | ✅ 15:04 | 24:07 |
| C | ✅ 13:32 | ✅ 0:24 | ❌ 0:22 | ✅ 0:57 | ❌ 0:49 | 16:04 |
| D | ✅ 6:16 | ✅ 1:12 | ✅ 0:56 | ✅ 0:56 | ✅ 2:50 | 12:10 |
| E | ✅ 5:50 | ✅ 1:01 | ✅ 0:21 | ✅ 0:26 | ✅ 3:56 | 11:34 |
Setup
One agent at a time, auto-approve on, a script decides pass or fail. Agents in alphabetical order, NOT the results order: OpenCode, goose, Atomic Agent, Hermes, OpenClaw
Which agent is A, B, C, D and E? Would you rather run one that installs stuff on its own and finishes, or one that asks first? Drop your guesses below, I'll share the full answer key in a day or two.
submitted by /u/GapNew4766 [link] [comments]
#OpenCode#goose#Atomic Agent#Hermes#OpenClaw#Qwen 3.8 27B#llama.cpp#M5 Max MacBook Pro