RTX 3060 上的本地大模型引擎基准测试
AI 总结
一名 Reddit 用户发布单机测试，在配备 RTX 3060 12 GB 的设备上比较 RustingStrata、PolyStrata 和 llama.cpp b12119 运行 Qwen3.6-35B-A3B 的表现。使用 5,290 个 token 的提示词时，RustingStrata 的预填充速度为 2,330 tok/s，PolyStrata 为 1,458–1,549 tok/s，llama.cpp 为 1,692 tok/s；无 MTP 解码速度分别为 90.9 tok/s、78.5 tok/s 和 53.5 tok/s。该测试使用了不同的提示词和配置，报告还指出 PolyStrata 的无 MTP 运行仍接受了部分草稿。
深度见解
这项测试提供了在消费级硬件上比较本地大模型推理引擎的详细数据，但结果来自单一机器、模型和量化版本的 Reddit 自报测试，尚未得到广泛独立验证。
RTX 3060 12 GB, Ryzen 5 7600X, 32 GB RAM, one GGUF file: Qwen3.6-35B-A3B-UD-Q3_K_M.gguf (Unsloth, MTP layer included, 15.9 GiB, does not fit in VRAM). Prompt of 5290 tokens, 128 generated tokens.
| RustingStrata | PolyStrata | llama.cpp b12119 | |
|---|---|---|---|
| Prefill, 5290-token prompt | 2330 tok/s | 1458-1549 tok/s | 1692 tok/s (-ncmoe 22, -ub 2048) |
| Decode, no MTP | 90.9 tok/s | 78.5 tok/s (--no-draft, see below) | 53.5 tok/s |
| Decode with MTP drafts | 128.4 tok/s | 97.5-100.4 tok/s | 77.7 tok/s (--spec-type draft-mtp, 92 of 102 drafts accepted) |
How each was run:
- RustingStrata:
rustingstrata bench --gpu --mtp 0|1 --pp 5290 --tg 128 --reps 3, mean of 3. - PolyStrata: its own
polystrata.py benchmark --model qwen3.6 --prompt-tokens 4000 --max-tokens 128. That prompt came to 5290 tokens, so I used that length for the other two. I ran it three times; the first run after setup was cold (decode 24.6 tok/s) and is left out, the numbers above are runs 2 and 3. MTP drafts are on by default. For the no-MTP row I added the engine's--no-draftflag to its config, it still offered a few drafts afterwards (6 of 16 and 15 of 36 accepted in the log), so that row is not a clean no-MTP number. Prefill in those runs: 1602-1643 tok/s. - llama.cpp:
llama-bench -ngl 99 -fa 1 -ub 2048 -b 2048 -p 5290 -n 128 -r 3, with-ncmoeswept over 18 to 26. 22 was the fastest that fits in 12 GB (no MTP). The MTP row isllama-server --spec-type draft-mtpwith-ncmoe 23(22 does not fit with the MTP context and drops to 20 tok/s), a 5381-token prompt of repo docs, 3 runs, prefill about 1565 tok/s.
#RustingStrata#PolyStrata#llama.cpp#Qwen3.6-35B-A3B#RTX 3060