Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Reddit user benchmarks MoE offloading on a 16GB RDNA4 9070 XT

AI summary

A Reddit user compared MoE expert offloading across ExLlamaV3, Strata and llama.cpp on a 16GB RDNA4 9070 XT with 32GB DDR4, targeting 128K context with Q8 KV caching. The reported winners were ExLlamaV3 with DFlash2 for Ornith 1.5 35B-A3B and Qwen3.6-35B-A3B, and Strata with speculative decoding for Qwen3.8-Flash-Next Coder, with 44.0–71.6 tokens per second at 128K. The author said 16GB of VRAM constrains context length and quantization, and cautioned that the benchmark may be flawed.

In-depth insight

The self-reported results provide a detailed comparison of MoE expert-offloading engines, context lengths and quantization tradeoffs on a memory-constrained local AI system.

First off. I am by all means no expert, and excellent work has been done in the past few years that we all currently enjoy.

Came over this post https://www.reddit.com/r/LocalLLM/comments/1x5mxcj/qwen3827b_on_a_single_rx_9070_xt_150_toks_on_code/ and wanted to try the engine on MoEs.

I've been benchmarking MoE experts offloading for the past two days in different engines on ROCm for my RDNA4 9070XT 16GB vram card and 32 GB DDR4.

I'd set a goal at 128K context, which is the base minimum for agentic work, without compaction all the time. I've kept the KV at Q8. (Q8_0/Q5_1 decreased performance on exllamav3). Drafter cache, if added, is at Q4.

Some limitations need apply when limited on vram.

Lets face it If you want any kind of context, and a decent Q4 quant, 16GB ain't gonna cut it. Hot experts cache like Strata, Exllmv3 and now recently llama.cpp have made this more feasible on the slower system memory bandwidth.

#AMD#RDNA4 9070 XT#ExLlamaV3#Strata#llama.cpp#Ornith 1.5 35B-A3B#Qwen3.6-35B-A3B#Qwen3.8-Flash-Next Coder#DFlash2#ROCm

Original · 2 min readRead the original
Group chat

No comments yet. Start the conversation.