RTX 5070 Ti

Headlines · 1

1. Oct 22
   capy-flash v0.7.1 runs 125B Qwen3.8-Flash-Next in 64 GB of RAM

   The developer of capy-flash, an open-source inference engine for running Qwen3.8-Flash-Next (125B MoE) and Qwen3.8-27B on consumer RTX 50 cards, released v0.7.1, which he says cuts Flash-Next's RAM requirement to 64 GB on one or two GPUs, down from the two 16 GB cards and 96 GB of RAM his setup needed before. The release uses ISTA-DASLab's ~3-bit GSQ-RCO i-quants for the routed experts (about 48 GiB in RAM instead of 65 for NVFP4), keeps hot experts in VRAM and computes the rest on the CPU, with peak RAM around 57 GB. It also fits Qwen3.8-27B with 3-bit weights on a single 16 GB card at 64k context with vision and MTP, adds AVX2 CPU support and guards against empty replies. The developer reports 234 tok/s for Flash-Next on two 16 GB cards and 174 tok/s for the 27B on one card, measured on an RTX 5070 Ti with a Ryzen 9 9900X, and notes 3-bit weights cost quality on the 27B: MMLU-Pro falls from 0.78 to about 0.72 while GSM8K is on par with NVFP4.

   r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my RTX 5070 Ti experienceAsk about RTX 5070 Ti