Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Qwen 3.6 MoE offloading hits 2,606 t/s prefill on RTX 9070 XT

AI summary

Reddit user /u/willeyh reported running Qwen 3.6 35B-A3B in Q4 on an RTX 9070 XT 16GB, with offloaded experts placed in DDR4-3200 RAM. Increasing the chunk size to 5120 reportedly pushed prefill above 2,000 t/s at all tested context lengths and peaked at 2,606 t/s at 32K, while larger chunks produced no further gains. ExLlamaV3's cpu_threads 6 setting slightly improved decoding, but several other flags regressed and cpu_moe_pinned_arena caused core dumps.

In-depth insight

The self-reported measurements provide a practical data point for optimizing MoE expert offloading on a 16GB consumer GPU, although the 5120 chunk size exceeds TabbyAPI's cited recommended range of 512–4096.

This is a follow up on https://www.reddit.com/r/LocalLLaMA/comments/1xun15v/moe_and_expert_offloading_on_llama_exllamav3/ While we wait for Qwen 4, here is Qwen 3.6 35B A3B at Q4 on 9070XT 16GB with offloaded experts on DDR4 3200Mhz ram. I've been tinkering still with the config for experts offloading and found substantial gains from my previous post. While decode stayed roughly in line fro last time, increasing chunk size to 5120 saw prefill over 2000 t/s at all context lengths, topping at 2606 t/s @ 32K. From ~1300t/s GPU utilization increases with batch size as it allows for more parallel processing. I saw no gains over 5120. TabbyAPI's documented ideal range is 512–4096, so your miles might vary. Exllama flags like cpu_threads 6 also made a small improvement in decode. Swizzle, gdn_sub, bg_gdn and wide flags were all regressions. (only tested isolated 1by1). cpu_moe_pinned_arena caused some core dumps, and the one run I managed to complete had a small gain. Vision have been turned off and offloaded. I still have not found an .exl3 quant for the Coder version of Qwen 3.8 Flash Next, and the qfn version I can run leaves me with 8K context with almost no vram or system ram left to spare. Results and more graphs and tables are at https://github.com/willeyh-git/moe-offload-bench All engines were compiled for ROCm on my system. submitted by /u/willeyh [link] [comments]

#Qwen 3.6 35B-A3B#ExLlamaV3#TabbyAPI#AMD RTX 9070 XT#ROCm#Qwen 3.8 Flash Next

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.