capy-flash

Headlines · 4

1. Oct 23
   Seven Qwen 3.8 FN Inference Engines Benchmarked on One RTX 5090

   The developer of Basalt benchmarked seven Qwen 3.8 FN inference engines on an RTX 5090 with 128 GB of RAM. The report’s results placed Basalt and Infernix ahead overall: Basalt recorded 354 tokens per second for structured output and 190 tokens per second at 25K context, while both reached a 3.94-second time to first token at 25K. Each engine converted NVIDIA’s NVFP4 weights with its own tool and was configured according to its documentation. Every engine received the same seven greedy requests, with one cold run per request and no warm-up.

   r/LocalLLaMA · 🔥 0
2. Oct 22
   Reddit benchmark pits three NVFP4 engines for Qwen3.8-Flash-Next on one RTX 5090

   A Reddit user posting in r/LocalLLaMA benchmarked three NVFP4 inference engines — basalt, capy-flash and ninfer-ext — running Qwen3.8-Flash-Next on a single NVIDIA RTX 5090 with 32 GB, sweeping context in 16k steps up to the full 262,144 tokens with MTP speculation and int8 KV cache and reporting zero prefix-cache hits. The post says basalt was fastest at every context depth at 8.1k–9.2k tok/s prefill and 176–265 tok/s greedy decode, capy-flash second at 6.8k–8.0k tok/s and 138–198 tok/s, and ninfer-ext third at 4.2k–6.3k tok/s and 73–106 tok/s. The author also reports that sampled decoding at temperature 1.0 cost 7–10% of decode throughput for every engine and nothing in prefill, and that tuning changed results substantially; the numbers are one user's measurements and have not been independently verified.

   r/LocalLLaMA · 🔥 0
3. Oct 22
   capy-flash v0.7.1 runs 125B Qwen3.8-Flash-Next in 64 GB of RAM

   The developer of capy-flash, an open-source inference engine for running Qwen3.8-Flash-Next (125B MoE) and Qwen3.8-27B on consumer RTX 50 cards, released v0.7.1, which he says cuts Flash-Next's RAM requirement to 64 GB on one or two GPUs, down from the two 16 GB cards and 96 GB of RAM his setup needed before. The release uses ISTA-DASLab's ~3-bit GSQ-RCO i-quants for the routed experts (about 48 GiB in RAM instead of 65 for NVFP4), keeps hot experts in VRAM and computes the rest on the CPU, with peak RAM around 57 GB. It also fits Qwen3.8-27B with 3-bit weights on a single 16 GB card at 64k context with vision and MTP, adds AVX2 CPU support and guards against empty replies. The developer reports 234 tok/s for Flash-Next on two 16 GB cards and 174 tok/s for the 27B on one card, measured on an RTX 5070 Ti with a Ryzen 9 9900X, and notes 3-bit weights cost quality on the 27B: MMLU-Pro falls from 0.78 to about 0.72 while GSM8K is on par with NVFP4.

   r/LocalLLaMA · 🔥 0
4. Oct 19
   Qwen3.8-27B and Qwen3.8-Flash-Next local inference benchmark roundup

   An r/LocalLLaMA user, FeydRowan, compiled single-request decode speeds for Qwen3.8-27B and the 125B MoE Qwen3.8-Flash-Next on consumer hardware, marking rows as “verified” only when backed by a LocalMaxxing run or a paired run on the same machine. The author's own capy-flash rows were measured on 2× RTX 5070 Ti with 96 GB DDR5, while most other figures are as reported by each project; the post also compares tensor parallelism versus layer splitting, expert offload, quant formats and drafters, and calculates tokens per second per €1,000 using EU VAT-inclusive prices. It puts 2× R9700 on top for Flash-Next cost-efficiency and says 96 GB of DDR5 is about 40% of the bill for such builds, but the performance and price data are largely user-compiled and self-reported.

   r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my capy-flash experienceAsk about capy-flash