Reddit 用户实测 16GB RDNA4 9070 XT 的 MoE 卸载
一名 Reddit 用户在配备 16GB RDNA4 9070 XT 和 32GB DDR4 的设备上,以 Q8 KV 缓存和 128K 上下文为目标,对比了 ExLlamaV3、Strata 与 llama.cpp 的 MoE 专家卸载表现。报告显示,Ornith 1.5 35B-A3B 和 Qwen3.6-35B-A3B 在 ExLlamaV3 配合 DFlash2 时表现最佳,Qwen3.8-Flash-Next Coder 则由 Strata 配合推测解码领先,128K 下的解码速度为每秒 44.0 至 71.6 个 token。作者指出,16GB 显存会限制上下文长度和量化规格,并提醒该基准测试可能存在错误。
这组自测结果详细呈现了显存受限的本地 AI 系统中,MoE 专家卸载引擎、上下文长度和量化方案之间的取舍。
First off. I am by all means no expert, and excellent work has been done in the past few years that we all currently enjoy.
Came over this post https://www.reddit.com/r/LocalLLM/comments/1x5mxcj/qwen3827b_on_a_single_rx_9070_xt_150_toks_on_code/ and wanted to try the engine on MoEs.
I've been benchmarking MoE experts offloading for the past two days in different engines on ROCm for my RDNA4 9070XT 16GB vram card and 32 GB DDR4.
I'd set a goal at 128K context, which is the base minimum for agentic work, without compaction all the time. I've kept the KV at Q8. (Q8_0/Q5_1 decreased performance on exllamav3). Drafter cache, if added, is at Q4.
Some limitations need apply when limited on vram.
Lets face it If you want any kind of context, and a decent Q4 quant, 16GB ain't gonna cut it. Hot experts cache like Strata, Exllmv3 and now recently llama.cpp have made this more feasible on the slower system memory bandwidth.
#AMD#RDNA4 9070 XT#ExLlamaV3#Strata#llama.cpp#Ornith 1.5 35B-A3B#Qwen3.6-35B-A3B#Qwen3.8-Flash-Next Coder#DFlash2#ROCm