capy-flash v0.7.1 runs 125B Qwen3.8-Flash-Next in 64 GB of RAM
The developer of capy-flash, an open-source inference engine for running Qwen3.8-Flash-Next (125B MoE) and Qwen3.8-27B on consumer RTX 50 cards, released v0.7.1, which he says cuts Flash-Next's RAM requirement to 64 GB on one or two GPUs, down from the two 16 GB cards and 96 GB of RAM his setup needed before. The release uses ISTA-DASLab's ~3-bit GSQ-RCO i-quants for the routed experts (about 48 GiB in RAM instead of 65 for NVFP4), keeps hot experts in VRAM and computes the rest on the CPU, with peak RAM around 57 GB. It also fits Qwen3.8-27B with 3-bit weights on a single 16 GB card at 64k context with vision and MTP, adds AVX2 CPU support and guards against empty replies. The developer reports 234 tok/s for Flash-Next on two 16 GB cards and 174 tok/s for the 27B on one card, measured on an RTX 5070 Ti with a Ryzen 9 9900X, and notes 3-bit weights cost quality on the 27B: MMLU-Pro falls from 0.78 to about 0.72 while GSM8K is on par with NVFP4.
I've been building capy-flash, an open-source inference engine for running Qwen3.8-Flash-Next (125B MoE) and Qwen3.8-27B on consumer RTX 50 cards. Until now it needed my setup (two 16 GB cards and 96 GB of RAM) or better. v0.7 is about smaller machines: start with what you have, add a second card later, same engine and same model files.
What's new
- Flash-Next on 64 GB of RAM, one or two GPUs. The routed experts are ISTA-DASLab's GSQ-RCO i-quants (~3 bit, copied as they are, not re-quantized): 48 GiB in RAM instead of 65 for NVFP4. The hot experts sit in VRAM, the rest are computed on the CPU. Peak RAM ~57 GB.
- Qwen3.8-27B on one 16 GB card, 3-bit GSQ weights (ISTA-DASLab), 64k context, vision, MTP. 13.9 GB of VRAM.
- AVX2 CPUs (Zen 2/3, Intel client) now work, and Windows/WSL2's page-locked memory limit no longer stops the server from starting.
- Guards against empty replies (the model stopping inside its reasoning), on by default.
Speed (LocalMaxxing reasoning-v1, one request, MTP, verified runs; RTX 5070 Ti 16 GB, Ryzen 9 9900X)
#capy-flash#Qwen3.8-Flash-Next#Qwen3.8-27B#ISTA-DASLab#RTX 5070 Ti#Strata