Qwen3.8 Flash-Next guide posts high local inference speeds amid quantization dispute
A community guide reports that Basalt, Unsloth's UD-IQ3_XXS quantization, and the Lamina drafter delivered 195.6 tokens/s average decode and about 5,700 tokens/s prompt reads for Qwen3.8 Flash-Next in an OpenCode session on an RTX 5080. The author says Basalt reached roughly five times llama.cpp's drafted decode speed and seven times its prompt-read speed, while Unsloth's quantization achieved 3.247 WikiText perplexity versus BF16's 3.273. A separate Reddit post says Unsloth disputed the reproducibility of ISTA-DASLab's GSQ-RCO benchmark numbers and found similarly sized Unsloth quantizations performing better in its testing.
The guide offers a detailed local-inference setup and reproducible configuration, but its quantization comparison overlaps with a disputed benchmark whose methodology was reportedly not published.
TL;DR
- Basalt is the engine and it's doing the heavy lifting. Same model file, same box: about 3.5x llama.cpp's decode without drafting and ~5x with it, and ~7x faster prompt reads.
- Unsloth's new UD-IQ3_XXS (Dynamic v3.1, uploaded Nov 4) is the same size as the GSQ-RCO IQ3_S everyone uses on 16GB cards, but measurably more accurate: wikitext perplexity 3.382 -> 3.247, which basically closes the gap to BF16 (3.273). Costs 1 to 3.5% decode speed.
- Lamina (a further-trained MTP drafter by eddoursul) gets more drafts accepted than the stock Basalt drafter. Small but real speedup, zero quality impact since the main model verifies every token.
- Real agent coding session (opencode, 151 requests, up to 202K context): 195.6 tok/s average decode, 263 tok/s best request, 156 tok/s worst, ~5,700 tok/s prompt reads.
Hardware:
- RTX 5080 16GB at its stock 360W limit (no overclock, no undervolt)
- Core Ultra 9 285K (8P + 16E), 2x32GB DDR5-7200
- PCIe 5.0 x16, Fedora
LAMINA Drafting: Basalt drafts with the model's own MTP layer. Basalt's upstream already ships a self-distilled version of that layer, and Lamina trains it further. It's a drop-in: same tensor layout, so the target model is byte-identical and only the drafter changes. What I saw:
#Qwen3.8 Flash-Next#Basalt#Unsloth UD-IQ3_XXS#Lamina#GSQ-RCO#llama.cpp#ISTA-DASLab#OpenCode