Lean-QwenFlash runs Qwen3.8-Flash-Next with low-memory NVMe offloading
A developer reported that Lean-QwenFlash ran the 125B-parameter Qwen3.8-Flash-Next on an RTX 4090 with 16 GB of VRAM and 8 GB of allocated system RAM by directly offloading to an NVMe SSD. Using a 3.05 bpw configuration and a Q4 KV cache, the reported RTX 4090 results were 28–33 tokens per second for prose and 19–22 tokens per second for code at 8K context, with decode speeds of about 20.8 and 20.0 tokens per second at 100K and 262K context. An RTX 3080 Laptop test, with 16 GB of VRAM and 32 GB of RAM, reached about 18–21 tokens per second at 100K context. The developer also published setup instructions, model downloaders and packing scripts. The performance figures are self-reported in the original post.
The report provides a concrete low-memory NVMe-offloading configuration, long-context benchmark results and published tooling for running a 125B-parameter model locally.
| Hey everyone, Most current setups for Qwen3.8-Flash-Next (like Strata) keep all 24k experts pinned in system RAM (~48–64 GB) and compute cache misses on the CPU. That works great on high-spec workstations, but not on normal machines or laptops with limited RAM. A few weeks ago, I noticed that https://github.com/sybil-solutions/glm-flash-lite was running GLM-5.3 with surprisingly low memory footprints considering the model, way below anything I'd seen at those speeds, which made me realized that something about the design was likely really efficient and being slept on. So I adapted that to the much lighter Qwen3.8 Flash Next model, and the results were pretty impressive. Benchmarks (3.05bpw_h5_ng5, Q4 KV cache) Setup instructions, model downloaders, and packing scripts are up on GitHub: https://github.com/MaxDevv/lean-qwenflash Big thanks to turboderp for ExLlamaV3, sybil-solutions for the original GLM offload implementation, and Alibaba for Flash-Next. submitted by /u/MaxDev0 [link] [comments] |
#Lean-QwenFlash#Qwen3.8-Flash-Next#GLM-5.3#ExLlamaV3#NVIDIA RTX 4090#NVIDIA RTX 3080 Laptop