Reddit user benchmarks MoE offloading on a 16GB RDNA4 9070 XT
A Reddit user compared MoE expert offloading across ExLlamaV3, Strata and llama.cpp on a 16GB RDNA4 9070 XT with 32GB DDR4, targeting 128K context with Q8 KV caching. The reported winners were ExLlamaV3 with DFlash2 for Ornith 1.5 35B-A3B and Qwen3.6-35B-A3B, and Strata with speculative decoding for Qwen3.8-Flash-Next Coder, with 44.0–71.6 tokens per second at 128K. The author said 16GB of VRAM constrains context length and quantization, and cautioned that the benchmark may be flawed.
The self-reported results provide a detailed comparison of MoE expert-offloading engines, context lengths and quantization tradeoffs on a memory-constrained local AI system.
First off. I am by all means no expert, and excellent work has been done in the past few years that we all currently enjoy.
Came over this post https://www.reddit.com/r/LocalLLM/comments/1x5mxcj/qwen3827b_on_a_single_rx_9070_xt_150_toks_on_code/ and wanted to try the engine on MoEs.
I've been benchmarking MoE experts offloading for the past two days in different engines on ROCm for my RDNA4 9070XT 16GB vram card and 32 GB DDR4.
I'd set a goal at 128K context, which is the base minimum for agentic work, without compaction all the time. I've kept the KV at Q8. (Q8_0/Q5_1 decreased performance on exllamav3). Drafter cache, if added, is at Q4.
Some limitations need apply when limited on vram.
Lets face it If you want any kind of context, and a decent Q4 quant, 16GB ain't gonna cut it. Hot experts cache like Strata, Exllmv3 and now recently llama.cpp have made this more feasible on the slower system memory bandwidth.
#AMD#RDNA4 9070 XT#ExLlamaV3#Strata#llama.cpp#Ornith 1.5 35B-A3B#Qwen3.6-35B-A3B#Qwen3.8-Flash-Next Coder#DFlash2#ROCm