ExLlamaV3

Headlines · 2

1. 7d
   Lean-QwenFlash runs Qwen3.8-Flash-Next with low-memory NVMe offloading

   A developer reported that Lean-QwenFlash ran the 125B-parameter Qwen3.8-Flash-Next on an RTX 4090 with 16 GB of VRAM and 8 GB of allocated system RAM by directly offloading to an NVMe SSD. Using a 3.05 bpw configuration and a Q4 KV cache, the reported RTX 4090 results were 28–33 tokens per second for prose and 19–22 tokens per second for code at 8K context, with decode speeds of about 20.8 and 20.0 tokens per second at 100K and 262K context. An RTX 3080 Laptop test, with 16 GB of VRAM and 32 GB of RAM, reached about 18–21 tokens per second at 100K context. The developer also published setup instructions, model downloaders and packing scripts. The performance figures are self-reported in the original post.

   r/LocalLLaMA · 🔥 0
2. Oct 19
   Qwen3.8-27B and Qwen3.8-Flash-Next local inference benchmark roundup

   An r/LocalLLaMA user, FeydRowan, compiled single-request decode speeds for Qwen3.8-27B and the 125B MoE Qwen3.8-Flash-Next on consumer hardware, marking rows as “verified” only when backed by a LocalMaxxing run or a paired run on the same machine. The author's own capy-flash rows were measured on 2× RTX 5070 Ti with 96 GB DDR5, while most other figures are as reported by each project; the post also compares tensor parallelism versus layer splitting, expert offload, quant formats and drafters, and calculates tokens per second per €1,000 using EU VAT-inclusive prices. It puts 2× R9700 on top for Flash-Next cost-efficiency and says 96 GB of DDR5 is about 40% of the bill for such builds, but the performance and price data are largely user-compiled and self-reported.

   r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my ExLlamaV3 experienceAsk about ExLlamaV3