创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

Qwen 3.6专家卸载预填充达2606 t/s

AI 总结

Reddit 用户 /u/willeyh 报告称,在配备 16GB 显存的 RTX 9070 XT 上以 Q4 运行 Qwen 3.6 35B-A3B,并将卸载的专家权重置于 DDR4-3200 内存。将分块大小提高到 5120 后,预填充速度在所有测试上下文长度下均超过 2000 t/s,并在 32K 时达到 2606 t/s,而继续增大分块未带来收益。ExLlamaV3 的 cpu_threads 6 参数小幅提升了解码速度,但其他多个参数出现回退,cpu_moe_pinned_arena 则导致核心转储。

深度见解

这组自报数据为在 16GB 消费级显卡上优化 MoE 专家卸载提供了实用参考,但 5120 的分块大小高于 TabbyAPI 所述 512–4096 的建议范围。

This is a follow up on https://www.reddit.com/r/LocalLLaMA/comments/1xun15v/moe_and_expert_offloading_on_llama_exllamav3/ While we wait for Qwen 4, here is Qwen 3.6 35B A3B at Q4 on 9070XT 16GB with offloaded experts on DDR4 3200Mhz ram. I've been tinkering still with the config for experts offloading and found substantial gains from my previous post. While decode stayed roughly in line fro last time, increasing chunk size to 5120 saw prefill over 2000 t/s at all context lengths, topping at 2606 t/s @ 32K. From ~1300t/s GPU utilization increases with batch size as it allows for more parallel processing. I saw no gains over 5120. TabbyAPI's documented ideal range is 512–4096, so your miles might vary. Exllama flags like cpu_threads 6 also made a small improvement in decode. Swizzle, gdn_sub, bg_gdn and wide flags were all regressions. (only tested isolated 1by1). cpu_moe_pinned_arena caused some core dumps, and the one run I managed to complete had a small gain. Vision have been turned off and offloaded. I still have not found an .exl3 quant for the Coder version of Qwen 3.8 Flash Next, and the qfn version I can run leaves me with 8K context with almost no vram or system ram left to spare. Results and more graphs and tables are at https://github.com/willeyh-git/moe-offload-bench All engines were compiled for ROCm on my system. submitted by /u/willeyh [link] [comments]

#Qwen 3.6 35B-A3B#ExLlamaV3#TabbyAPI#AMD RTX 9070 XT#ROCm#Qwen 3.8 Flash Next

原文约 1 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。