Qwen3.8 Flash-Next 本地推理提速与量化争议
一篇社区指南称，在配备 RTX 5080 的 OpenCode 会话中，Basalt、Unsloth UD-IQ3_XXS 量化与 Lamina drafter 可让 Qwen3.8 Flash-Next 平均解码达到 195.6 tokens/s，提示词读取约为 5,700 tokens/s。作者称 Basalt 的草稿解码速度约为 llama.cpp 的五倍、提示词读取约为七倍，Unsloth 量化版的 WikiText 困惑度为 3.247，接近 BF16 的 3.273。另一篇 Reddit 帖子称，Unsloth 对 ISTA-DASLab 的 GSQ-RCO 基准可复现性提出质疑，并在测试中发现同尺寸的 Unsloth 量化表现更好。
该指南提供了详细的本地推理配置与复现步骤，但其量化对比涉及一套被指难以复现、且未公布方法论的基准。
TL;DR
- Basalt is the engine and it's doing the heavy lifting. Same model file, same box: about 3.5x llama.cpp's decode without drafting and ~5x with it, and ~7x faster prompt reads.
- Unsloth's new UD-IQ3_XXS (Dynamic v3.1, uploaded Nov 4) is the same size as the GSQ-RCO IQ3_S everyone uses on 16GB cards, but measurably more accurate: wikitext perplexity 3.382 -> 3.247, which basically closes the gap to BF16 (3.273). Costs 1 to 3.5% decode speed.
- Lamina (a further-trained MTP drafter by eddoursul) gets more drafts accepted than the stock Basalt drafter. Small but real speedup, zero quality impact since the main model verifies every token.
- Real agent coding session (opencode, 151 requests, up to 202K context): 195.6 tok/s average decode, 263 tok/s best request, 156 tok/s worst, ~5,700 tok/s prompt reads.
Hardware:
- RTX 5080 16GB at its stock 360W limit (no overclock, no undervolt)
- Core Ultra 9 285K (8P + 16E), 2x32GB DDR5-7200
- PCIe 5.0 x16, Fedora
LAMINA Drafting: Basalt drafts with the model's own MTP layer. Basalt's upstream already ships a self-distilled version of that layer, and Lamina trains it further. It's a drop-in: same tensor layout, so the target model is byte-identical and only the drafter changes. What I saw:
#Qwen3.8 Flash-Next#Basalt#Unsloth UD-IQ3_XXS#Lamina#GSQ-RCO#llama.cpp#ISTA-DASLab#OpenCode