tandem.cpp adds AMD ROCm support and large-model CPU/GPU offloading
The developer of tandem.cpp announced that its llama.cpp patch set can now build with HIP for AMD ROCm, while keeping frequently used MoE experts on the GPU and offloading remaining work to the CPU. In developer-run tests, DeepSeek-V4.1-Flash IQ2_XS with 139 GiB of experts reached 14.2 decode tokens per second on a system with an RTX 5070 Ti and 64 GB RAM, compared with 7.4 for llama.cpp. The AMD RX 7900 XTX test also included Qwen3.8 IQ3_S with MTP, but the reported performance figures have not been independently verified and Vulkan support is not yet available.
The update expands local MoE inference to AMD hardware and offers a practical CPU/GPU offloading approach for models whose expert weights exceed available VRAM, although the performance claims are self-reported.
Quick update on tandem.cpp, my patch set for llama.cpp that runs MoE models too big for your VRAM faster. It keeps the hot experts on the GPU and lets the CPU help with the rest. Repo: https://github.com/ewhacc/tandem.cpp
DeepSeek-V4.1-Flash IQ2_XS (ISTA GSQ-RCO, 139 GiB of experts) on normal PCs
Decode tok/s, one user, GSM8K questions, greedy, exact (no expert skipping on any engine), same quant on every engine:
| PC | llama.cpp | tandem.cpp | Strata |
|---|---|---|---|
| RTX 5070 Ti 16 GB, 64 GB RAM, i5-14600KF | 7.4 | 14.2 | 10.4 |
| RTX 5070 Ti 16 GB, 64 GB RAM, i5-12400F, budget DRAM-less SSD | - | 10.4 | 5.7 |
| RTX 5090 32 GB, 128 GB RAM, i5-14600KF | 11.5 | 33.1 | 28.0 |
At 64 GB the model is about twice your RAM, so it streams from the SSD.
Less RAM than the model: Qwen3.8-Flash-Next IQ3_S (ISTA GSQ-RCO, 78 GiB) + MTP on a 32 GB PC
#tandem.cpp#llama.cpp#AMD ROCm#HIP#DeepSeek-V4.1-Flash#DeepSeek-V4.1#Qwen3.8-Flash-Next#Qwen3.8#gpt-oss-120b#GLM-4.5-Air#Qwen3.6#Strata