AutoRound

Headlines · 1

1. Oct 19
   Community recipe runs Qwen 3.8 Flash Next on two CMP 170HX GPUs at 840 tok/s aggregate

   Reddit user /u/dead-fish on r/LocalLLaMA published an optimization recipe for running Qwen 3.8 Flash Next on two CMP 170HX mining cards, both connected over PCIe Gen2 x4 with no x16 mod and no TP2. The setup uses a W4A16 AutoRound checkpoint, a patched vLLM, pipeline parallelism and native MTP speculative decoding, with the n-gram table memory-mapped from NVMe and everything else fitting on the two GPUs. The author reports 840 tok/s aggregate decode for eight concurrent code requests, about 116 tok/s per request with 231 ms time to first token, 6,300–7,100 tok/s prefill on 4k–64k prompts and about 5,900 tok/s at 128k, while using the full 262,144-token window.

   r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my AutoRound experienceAsk about AutoRound