capy-flash v0.7.1：64GB内存跑125B MoE的Qwen3.8-Flash-Next
开源推理引擎 capy-flash 发布 v0.7.1，开发者称这一版本让 Qwen3.8-Flash-Next（125B MoE）在单卡或双卡搭配 64 GB 内存的机器上即可运行，而此前需要他本人所用的两张 16 GB 显卡加 96 GB 内存。该版本对路由专家采用 ISTA-DASLab 的约 3 bit GSQ-RCO i-quants（内存占用约 48 GiB，NVFP4 为 65），热专家留在显存、其余在 CPU 上计算，峰值内存约 57 GB；同时 Qwen3.8-27B 的 3 bit 权重可装入单张 16 GB 显卡，支持 64k 上下文、视觉与 MTP，并新增 AVX2 CPU 支持与空回复防护。开发者自报在 RTX 5070 Ti 加 Ryzen 9 9900X 上，双卡跑 Flash-Next 为 234 tok/s，单卡跑 27B 为 174 tok/s，并指出 27B 的 3 bit 量化有质量损失：MMLU-Pro 从 0.78 降到约 0.72，GSM8K 与 NVFP4 持平。
I've been building capy-flash, an open-source inference engine for running Qwen3.8-Flash-Next (125B MoE) and Qwen3.8-27B on consumer RTX 50 cards. Until now it needed my setup (two 16 GB cards and 96 GB of RAM) or better. v0.7 is about smaller machines: start with what you have, add a second card later, same engine and same model files.
What's new
- Flash-Next on 64 GB of RAM, one or two GPUs. The routed experts are ISTA-DASLab's GSQ-RCO i-quants (~3 bit, copied as they are, not re-quantized): 48 GiB in RAM instead of 65 for NVFP4. The hot experts sit in VRAM, the rest are computed on the CPU. Peak RAM ~57 GB.
- Qwen3.8-27B on one 16 GB card, 3-bit GSQ weights (ISTA-DASLab), 64k context, vision, MTP. 13.9 GB of VRAM.
- AVX2 CPUs (Zen 2/3, Intel client) now work, and Windows/WSL2's page-locked memory limit no longer stops the server from starting.
- Guards against empty replies (the model stopping inside its reasoning), on by default.
Speed (LocalMaxxing reasoning-v1, one request, MTP, verified runs; RTX 5070 Ti 16 GB, Ryzen 9 9900X)
#capy-flash#Qwen3.8-Flash-Next#Qwen3.8-27B#ISTA-DASLab#RTX 5070 Ti#Strata