tandem.cpp accelerates CPU-assisted MoE decoding on consumer GPUs
The independent tandem.cpp project has released patches for llama.cpp that keep frequently used MoE experts on the GPU while the CPU handles the rest, targeting single-user decoding when a model does not fit in VRAM. The project reports that Qwen3.8-Flash-Next reached 97.3 decoding tokens per second with greedy sampling on an i5-12400F system with 64 GB DDR4 memory and an RTX 4090, compared with 29.8 tokens per second for llama.cpp. It also reports gains for gpt-oss-120b and other MoE models, although the implementation is tuned specifically for Qwen3.8.
By combining GPU and CPU resources, tandem.cpp could make single-user decoding more practical for MoE models that exceed available VRAM. The reported results also indicate whether this CPU-assisted approach offers meaningful gains on consumer hardware.
tandem.cpp is a patch set for llama.cpp that makes MoE models that don't fit in VRAM run faster. It keeps the most-used experts on the GPU and lets the CPU help with the rest, for any MoE model llama.cpp supports. Single-user decode is the target.
Consumer PC (i5-12400F, 64 GB DDR4, RTX 4090), GSM8K, decode tok/s:
| Model | llama.cpp | tandem.cpp | Strata |
|---|---|---|---|
| Qwen3.8-Flash-Next IQ3_S + MTP, greedy | 29.8 | 97.3 (3.3×) | 103.6 |
| same, Qwen's default sampling | 29.4 | 96.1 (3.3×) | 98.1 |
| gpt-oss-120b, greedy | 27.8 | 51.5 (1.85×) | — |
| gpt-oss-120b, recommended sampling | 27.7 | 48.0 (1.73×) | — |
Tuned for Qwen3.8 only, but other MoE models that don't fit in VRAM gain too: typically 1.4–2.1× (gpt-oss-120b, GLM-4.5-Air, Qwen3.6-35B-A3B, DeepSeek-V4.1). Prompt processing is 1.4–1.8× faster on a 64k prompt.
Strata is shown in its exact mode (every routed expert computed, like llama.cpp). Its default skips low-ranked experts, which is faster but approximate (127.1 here).
Independent project, not affiliated with ggml-org. Build, run commands, and full results on 4 machines: https://github.com/ewhacc/tandem.cpp
submitted by /u/smflx [link] [comments]
#tandem.cpp#llama.cpp#Qwen3.8-Flash-Next#gpt-oss-120b#GLM-4.5-Air#Qwen3.6-35B-A3B#DeepSeek-V4.1#Strata