Developer reports cudaMemcpy tuning boosts MoE prefill on RTX 3090
AI summary
Developer Maknee released a tool that tunes CUDA memory copies for CPU offloading in MoE inference. In a self-reported llama-bench test using an RTX 3090, Ryzen 7 7800X3D and Qwen3.6-35B-A3B-Q8_0, prefill throughput increased from 345 to 466 tokens per second. The developer published the code and benchmark setup.
In-depth insight
The tool directly targets CPU offloading in MoE inference. Publishing the code and benchmark setup allows users to inspect and reproduce the developer’s reported test.
https://i.redd.it/5dlnj6wict1i1.gif I made a tool that tunes copies that speeds up CPU offloading. 1 3090 + 7800X3D llama-bench -m Qwen3.6-35B-A3B-Q8_0.gguf -ngl 99 -fa 1 -ot exps=CPU -p 2048 -n 0 -r 3 cmt tune -- llama-bench -m Qwen3.6-35B-A3B-Q8_0.gguf -ngl 99 -fa 1 -ot exps=CPU -p 2048 -n 0 -r 3 before: prefill 345 t/s after: prefill 466 t/s blog: https://maknee.github.io/blog/2026/CudaMemcpy/ code: https://github.com/Maknee/cudamemcpy-tuner submitted by /u/makneeee
[link] [comments] |
#cudaMemcpy tuner#llama.cpp#Qwen3.6-35B-A3B#RTX 3090#AMD Ryzen 7 7800X3D