Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

tandem.cpp accelerates CPU-assisted MoE decoding on consumer GPUs

AI summary

The independent tandem.cpp project has released patches for llama.cpp that keep frequently used MoE experts on the GPU while the CPU handles the rest, targeting single-user decoding when a model does not fit in VRAM. The project reports that Qwen3.8-Flash-Next reached 97.3 decoding tokens per second with greedy sampling on an i5-12400F system with 64 GB DDR4 memory and an RTX 4090, compared with 29.8 tokens per second for llama.cpp. It also reports gains for gpt-oss-120b and other MoE models, although the implementation is tuned specifically for Qwen3.8.

In-depth insight

By combining GPU and CPU resources, tandem.cpp could make single-user decoding more practical for MoE models that exceed available VRAM. The reported results also indicate whether this CPU-assisted approach offers meaningful gains on consumer hardware.

tandem.cpp is a patch set for llama.cpp that makes MoE models that don't fit in VRAM run faster. It keeps the most-used experts on the GPU and lets the CPU help with the rest, for any MoE model llama.cpp supports. Single-user decode is the target.

Consumer PC (i5-12400F, 64 GB DDR4, RTX 4090), GSM8K, decode tok/s:

Modelllama.cpptandem.cppStrata
Qwen3.8-Flash-Next IQ3_S + MTP, greedy29.897.3 (3.3×)103.6
same, Qwen's default sampling29.496.1 (3.3×)98.1
gpt-oss-120b, greedy27.851.5 (1.85×)—
gpt-oss-120b, recommended sampling27.748.0 (1.73×)—

Tuned for Qwen3.8 only, but other MoE models that don't fit in VRAM gain too: typically 1.4–2.1× (gpt-oss-120b, GLM-4.5-Air, Qwen3.6-35B-A3B, DeepSeek-V4.1). Prompt processing is 1.4–1.8× faster on a 64k prompt.

Strata is shown in its exact mode (every routed expert computed, like llama.cpp). Its default skips low-ranked experts, which is faster but approximate (127.1 here).

Independent project, not affiliated with ggml-org. Build, run commands, and full results on 4 machines: https://github.com/ewhacc/tandem.cpp

submitted by /u/smflx [link] [comments]

#tandem.cpp#llama.cpp#Qwen3.8-Flash-Next#gpt-oss-120b#GLM-4.5-Air#Qwen3.6-35B-A3B#DeepSeek-V4.1#Strata

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.