五款自托管智能体盲测结果公布
Reddit 用户 /u/GapNew4766 报告了对 OpenCode、goose、Atomic Agent、Hermes 和 OpenClaw 的脚本化盲测，五款智能体均在配备 64GB 内存的 M5 Max MacBook Pro 上本地运行。每款智能体都使用 llama.cpp 和未审查的 Qwen 3.8 27B 模型，完成相同的五项任务，包括合并联系人列表、跨三个文件重命名函数、修复错误、统计日志错误及将票据 PDF 转为电子表格。五款智能体均通过其中三项，修复错误和票据转换各有四款通过，发布时智能体与字母标签的对应关系仍未公开。
该报告在同一本地模型环境下给出了逐项结果和耗时，可供实际比较五款自托管智能体，但目前仅有单一作者来源，且测试仅覆盖五项任务。
everything here ran on my own hardware, an M5 Max MacBook Pro with 64GB. blind test of self-hosted AI agents, every one on Qwen 3.8 27B uncensored Q4_K_M in llama.cpp, same 5 everyday tasks like counting log errors and renaming a function. names hidden, results below, take a guess
The tasks
- Merge two contact lists, messy emails, two different Alex Kims. All 5 got it right (46 people).
- Rename a function across 3 files with messy spacing. All 5 passed.
- Fix a bug, with hidden tests. 4 passed. C turned "Q3/Q4" into "q3q4" instead of "q3-q4".
- Count the errors in 3 log files. All 5 got 43.
- Turn a receipt PDF into a spreadsheet. 4 passed. A and B installed a missing Python library without asking. C stopped and asked "can I install this?", nobody answered, so it failed. In a test with nobody watching, that counts as a fail. On my actual hardware, that's the behavior I'd want.
| Agent | 1. Contacts | 2. Rename | 3. Bug fix | 4. Errors | 5. Receipt | Total |
|---|---|---|---|---|---|---|
| A | ✅ 4:54 | ✅ 0:28 | ✅ 0:43 | ✅ 0:26 | ✅ 3:58 | 10:29 |
| B | ✅ 6:03 | ✅ 1:11 | ✅ 0:55 | ✅ 0:54 | ✅ 15:04 | 24:07 |
| C | ✅ 13:32 | ✅ 0:24 | ❌ 0:22 | ✅ 0:57 | ❌ 0:49 | 16:04 |
| D | ✅ 6:16 | ✅ 1:12 | ✅ 0:56 | ✅ 0:56 | ✅ 2:50 | 12:10 |
| E | ✅ 5:50 | ✅ 1:01 | ✅ 0:21 | ✅ 0:26 | ✅ 3:56 | 11:34 |
Setup
One agent at a time, auto-approve on, a script decides pass or fail. Agents in alphabetical order, NOT the results order: OpenCode, goose, Atomic Agent, Hermes, OpenClaw
Which agent is A, B, C, D and E? Would you rather run one that installs stuff on its own and finishes, or one that asks first? Drop your guesses below, I'll share the full answer key in a day or two.
submitted by /u/GapNew4766 [link] [comments]
#OpenCode#goose#Atomic Agent#Hermes#OpenClaw#Qwen 3.8 27B#llama.cpp#M5 Max MacBook Pro