capy-flash v0.7.4 expands local Qwen3.8 inference
capy-flash v0.7.4 is a standalone local inference engine for Qwen3.8-27B and Qwen3.8-Flash-Next, designed around NVIDIA NVFP4 quantization and multi-GPU tensor parallelism. The author reports broader single-GPU support, persistent context caching, faster long-prompt processing, and compatibility with Claude Code, Codex CLI, and OpenCode; the post also includes local throughput benchmarks, but these results have not been independently verified.
The release gives Qwen3.8 users a locally run inference option for RTX 50-series hardware, including persistent long-context sessions and agent-client integrations.
It's been a few releases since my last post, so here is where capy-flash is now. It's my own engine for two models, Qwen3.8-27B and Qwen3.8-Flash-Next. It started as my ninfer-tp2 fork of NInfer, which added tensor parallel over PCIe, but at this point it's an engine of its own and very little of the original is left.
What it's for
Two RTX 50 cards on Linux. Tensor parallel across both, and for Flash-Next a VRAM expert cache shared by the two cards, with the misses computed on the CPU. Everything is tuned on one box: 2× RTX 5070 Ti 16 GB, Ryzen 9 9900X, 96 GB DDR5. It was born specifically for the NVFP4 Nvidia quantisation.
It also runs on smaller setups, so more people can try it: one 16 GB card (5060 Ti and up), Flash-Next with 64 GB of RAM, mixed pairs like a 5070 Ti with a 5060 Ti. People have posted numbers from 2× 5080, 2× RTX PRO 4000 Blackwell and 2× 5060 Ti in the issues. It runs on Windows too, through WSL2 or Docker. Two cards there are slower than on Linux, and a native Windows port is in progress.
#capy-flash v0.7.4#Qwen3.8-27B#Qwen3.8-Flash-Next#NVIDIA NVFP4#NVIDIA RTX 50 Series#Claude Code#Codex CLI#OpenCode#Strata#NInfer