tandem.cpp 加速消费级 GPU 上的 CPU 辅助 MoE 解码
独立项目 tandem.cpp 发布了面向 llama.cpp 的补丁,通过让常用 MoE 专家留在 GPU、其余专家由 CPU 处理,在模型无法装入显存时加速单用户解码。项目报告称,在配备 i5-12400F、64 GB DDR4 内存和 RTX 4090 的系统上,采用贪心采样的 Qwen3.8-Flash-Next 达到 97.3 个解码 token/s,而 llama.cpp 为 29.8 token/s。项目还报告了 gpt-oss-120b 等其他 MoE 模型的性能提升,但该实现专门针对 Qwen3.8 调优。
通过联合使用 GPU 与 CPU 资源,tandem.cpp 可能让显存容量不足的 MoE 模型更适合进行单用户解码。报告中的结果也显示,这种 CPU 辅助方式在消费级硬件上可能带来明显性能收益。
tandem.cpp is a patch set for llama.cpp that makes MoE models that don't fit in VRAM run faster. It keeps the most-used experts on the GPU and lets the CPU help with the rest, for any MoE model llama.cpp supports. Single-user decode is the target.
Consumer PC (i5-12400F, 64 GB DDR4, RTX 4090), GSM8K, decode tok/s:
| Model | llama.cpp | tandem.cpp | Strata |
|---|---|---|---|
| Qwen3.8-Flash-Next IQ3_S + MTP, greedy | 29.8 | 97.3 (3.3×) | 103.6 |
| same, Qwen's default sampling | 29.4 | 96.1 (3.3×) | 98.1 |
| gpt-oss-120b, greedy | 27.8 | 51.5 (1.85×) | — |
| gpt-oss-120b, recommended sampling | 27.7 | 48.0 (1.73×) | — |
Tuned for Qwen3.8 only, but other MoE models that don't fit in VRAM gain too: typically 1.4–2.1× (gpt-oss-120b, GLM-4.5-Air, Qwen3.6-35B-A3B, DeepSeek-V4.1). Prompt processing is 1.4–1.8× faster on a 64k prompt.
Strata is shown in its exact mode (every routed expert computed, like llama.cpp). Its default skips low-ranked experts, which is faster but approximate (127.1 here).
Independent project, not affiliated with ggml-org. Build, run commands, and full results on 4 machines: https://github.com/ewhacc/tandem.cpp
submitted by /u/smflx [link] [comments]
#tandem.cpp#llama.cpp#Qwen3.8-Flash-Next#gpt-oss-120b#GLM-4.5-Air#Qwen3.6-35B-A3B#DeepSeek-V4.1#Strata