创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

Reddit 用户实测 16GB RDNA4 9070 XT 的 MoE 卸载

AI 总结

一名 Reddit 用户在配备 16GB RDNA4 9070 XT 和 32GB DDR4 的设备上,以 Q8 KV 缓存和 128K 上下文为目标,对比了 ExLlamaV3、Strata 与 llama.cpp 的 MoE 专家卸载表现。报告显示,Ornith 1.5 35B-A3B 和 Qwen3.6-35B-A3B 在 ExLlamaV3 配合 DFlash2 时表现最佳,Qwen3.8-Flash-Next Coder 则由 Strata 配合推测解码领先,128K 下的解码速度为每秒 44.0 至 71.6 个 token。作者指出,16GB 显存会限制上下文长度和量化规格,并提醒该基准测试可能存在错误。

深度见解

这组自测结果详细呈现了显存受限的本地 AI 系统中,MoE 专家卸载引擎、上下文长度和量化方案之间的取舍。

First off. I am by all means no expert, and excellent work has been done in the past few years that we all currently enjoy.

Came over this post https://www.reddit.com/r/LocalLLM/comments/1x5mxcj/qwen3827b_on_a_single_rx_9070_xt_150_toks_on_code/ and wanted to try the engine on MoEs.

I've been benchmarking MoE experts offloading for the past two days in different engines on ROCm for my RDNA4 9070XT 16GB vram card and 32 GB DDR4.

I'd set a goal at 128K context, which is the base minimum for agentic work, without compaction all the time. I've kept the KV at Q8. (Q8_0/Q5_1 decreased performance on exllamav3). Drafter cache, if added, is at Q4.

Some limitations need apply when limited on vram.

Lets face it If you want any kind of context, and a decent Q4 quant, 16GB ain't gonna cut it. Hot experts cache like Strata, Exllmv3 and now recently llama.cpp have made this more feasible on the slower system memory bandwidth.

#AMD#RDNA4 9070 XT#ExLlamaV3#Strata#llama.cpp#Ornith 1.5 35B-A3B#Qwen3.6-35B-A3B#Qwen3.8-Flash-Next Coder#DFlash2#ROCm

原文约 2 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。