Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

tandem.cpp adds AMD ROCm support and large-model CPU/GPU offloading

AI summary

The developer of tandem.cpp announced that its llama.cpp patch set can now build with HIP for AMD ROCm, while keeping frequently used MoE experts on the GPU and offloading remaining work to the CPU. In developer-run tests, DeepSeek-V4.1-Flash IQ2_XS with 139 GiB of experts reached 14.2 decode tokens per second on a system with an RTX 5070 Ti and 64 GB RAM, compared with 7.4 for llama.cpp. The AMD RX 7900 XTX test also included Qwen3.8 IQ3_S with MTP, but the reported performance figures have not been independently verified and Vulkan support is not yet available.

In-depth insight

The update expands local MoE inference to AMD hardware and offers a practical CPU/GPU offloading approach for models whose expert weights exceed available VRAM, although the performance claims are self-reported.

Quick update on tandem.cpp, my patch set for llama.cpp that runs MoE models too big for your VRAM faster. It keeps the hot experts on the GPU and lets the CPU help with the rest. Repo: https://github.com/ewhacc/tandem.cpp

DeepSeek-V4.1-Flash IQ2_XS (ISTA GSQ-RCO, 139 GiB of experts) on normal PCs

Decode tok/s, one user, GSM8K questions, greedy, exact (no expert skipping on any engine), same quant on every engine:

PCllama.cpptandem.cppStrata
RTX 5070 Ti 16 GB, 64 GB RAM, i5-14600KF7.414.210.4
RTX 5070 Ti 16 GB, 64 GB RAM, i5-12400F, budget DRAM-less SSD-10.45.7
RTX 5090 32 GB, 128 GB RAM, i5-14600KF11.533.128.0

At 64 GB the model is about twice your RAM, so it streams from the SSD.

Less RAM than the model: Qwen3.8-Flash-Next IQ3_S (ISTA GSQ-RCO, 78 GiB) + MTP on a 32 GB PC

#tandem.cpp#llama.cpp#AMD ROCm#HIP#DeepSeek-V4.1-Flash#DeepSeek-V4.1#Qwen3.8-Flash-Next#Qwen3.8#gpt-oss-120b#GLM-4.5-Air#Qwen3.6#Strata

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.