Qwen 3.6 MoE offloading hits 2,606 t/s prefill on RTX 9070 XT
Reddit user /u/willeyh reported running Qwen 3.6 35B-A3B in Q4 on an RTX 9070 XT 16GB, with offloaded experts placed in DDR4-3200 RAM. Increasing the chunk size to 5120 reportedly pushed prefill above 2,000 t/s at all tested context lengths and peaked at 2,606 t/s at 32K, while larger chunks produced no further gains. ExLlamaV3's cpu_threads 6 setting slightly improved decoding, but several other flags regressed and cpu_moe_pinned_arena caused core dumps.
The self-reported measurements provide a practical data point for optimizing MoE expert offloading on a 16GB consumer GPU, although the 5120 chunk size exceeds TabbyAPI's cited recommended range of 512–4096.
| This is a follow up on https://www.reddit.com/r/LocalLLaMA/comments/1xun15v/moe_and_expert_offloading_on_llama_exllamav3/ While we wait for Qwen 4, here is Qwen 3.6 35B A3B at Q4 on 9070XT 16GB with offloaded experts on DDR4 3200Mhz ram. I've been tinkering still with the config for experts offloading and found substantial gains from my previous post. While decode stayed roughly in line fro last time, increasing chunk size to 5120 saw prefill over 2000 t/s at all context lengths, topping at 2606 t/s @ 32K. From ~1300t/s GPU utilization increases with batch size as it allows for more parallel processing. I saw no gains over 5120. TabbyAPI's documented ideal range is 512–4096, so your miles might vary. Exllama flags like cpu_threads 6 also made a small improvement in decode. Swizzle, gdn_sub, bg_gdn and wide flags were all regressions. (only tested isolated 1by1). cpu_moe_pinned_arena caused some core dumps, and the one run I managed to complete had a small gain. Vision have been turned off and offloaded. I still have not found an .exl3 quant for the Coder version of Qwen 3.8 Flash Next, and the qfn version I can run leaves me with 8K context with almost no vram or system ram left to spare. Results and more graphs and tables are at https://github.com/willeyh-git/moe-offload-bench All engines were compiled for ROCm on my system. submitted by /u/willeyh [link] [comments] |
#Qwen 3.6 35B-A3B#ExLlamaV3#TabbyAPI#AMD RTX 9070 XT#ROCm#Qwen 3.8 Flash Next