tandem.cpp新增AMD ROCm支持
tandem.cpp开发者宣布,这一llama.cpp补丁集现已可通过HIP为AMD ROCm构建,并将常用MoE专家保留在GPU上,其余工作交由CPU处理。在开发者测试中,采用IQ2_XS量化、包含139 GiB专家的DeepSeek-V4.1-Flash在RTX 5070 Ti和64 GB内存配置下达到14.2个解码令牌每秒,而llama.cpp为7.4。AMD RX 7900 XTX测试还覆盖了Qwen3.8 IQ3_S与MTP,但相关性能数据尚未经过独立验证,目前也不支持Vulkan。
此次更新将本地MoE推理扩展到AMD硬件,并为专家权重超出显存的模型提供CPU与GPU协同卸载方案,但性能数据仍来自开发者自测。
Quick update on tandem.cpp, my patch set for llama.cpp that runs MoE models too big for your VRAM faster. It keeps the hot experts on the GPU and lets the CPU help with the rest. Repo: https://github.com/ewhacc/tandem.cpp
DeepSeek-V4.1-Flash IQ2_XS (ISTA GSQ-RCO, 139 GiB of experts) on normal PCs
Decode tok/s, one user, GSM8K questions, greedy, exact (no expert skipping on any engine), same quant on every engine:
| PC | llama.cpp | tandem.cpp | Strata |
|---|---|---|---|
| RTX 5070 Ti 16 GB, 64 GB RAM, i5-14600KF | 7.4 | 14.2 | 10.4 |
| RTX 5070 Ti 16 GB, 64 GB RAM, i5-12400F, budget DRAM-less SSD | - | 10.4 | 5.7 |
| RTX 5090 32 GB, 128 GB RAM, i5-14600KF | 11.5 | 33.1 | 28.0 |
At 64 GB the model is about twice your RAM, so it streams from the SSD.
Less RAM than the model: Qwen3.8-Flash-Next IQ3_S (ISTA GSQ-RCO, 78 GiB) + MTP on a 32 GB PC
#tandem.cpp#llama.cpp#AMD ROCm#HIP#DeepSeek-V4.1-Flash#DeepSeek-V4.1#Qwen3.8-Flash-Next#Qwen3.8#gpt-oss-120b#GLM-4.5-Air#Qwen3.6#Strata