Reddit user: 1.66bpw exl3 quantization of Qwen3.8-Flash-Next runs under 36GB on one RTX 5090
A poster on r/LocalLLaMA (u/Unstable_Llama) reports that a 1.66 bit-per-weight exl3 quantization of Qwen3.8-Flash-Next fits entirely in a 36GB footprint on a single RTX 5090 with no offloaded experts. The poster describes inference as stable with no reasoning loops at roughly 200–300 tokens per second, but notes heavy compaction is needed because a 32GB GPU leaves little room for context even with cache quantization. The weights are said to be hosted on Hugging Face under turboderp/Qwen3.8-Flash-Next-exl3; the performance claims are the user's own and have not been independently verified.
If it holds up, the report indicates a Qwen Flash-class model can be run locally in sub-2-bit form on a single consumer GPU, though the poster says context capacity remains the binding constraint.
| Fits entirely in a 5090, no offloaded experts. Super-solid sub-2bpw! (Liquid cat demo video generated by the quantized model) From the Cat's mouth: Stable inference throughout, no reasoning loops, just smooth sailing at 200-300 t/s (and a lot of compacting since one 32 GB GPU still doesn't leave a lot of room for context, even with cache quantization. https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3 submitted by /u/Unstable_Llama [link] [comments] |
#Qwen3.8-Flash-Next#Qwen#Hugging Face#RTX 5090#exl3#turboderp