Community recipe runs Qwen 3.8 Flash Next on two CMP 170HX GPUs at 840 tok/s aggregate
Reddit user /u/dead-fish on r/LocalLLaMA published an optimization recipe for running Qwen 3.8 Flash Next on two CMP 170HX mining cards, both connected over PCIe Gen2 x4 with no x16 mod and no TP2. The setup uses a W4A16 AutoRound checkpoint, a patched vLLM, pipeline parallelism and native MTP speculative decoding, with the n-gram table memory-mapped from NVMe and everything else fitting on the two GPUs. The author reports 840 tok/s aggregate decode for eight concurrent code requests, about 116 tok/s per request with 231 ms time to first token, 6,300–7,100 tok/s prefill on 4k–64k prompts and about 5,900 tok/s at 128k, while using the full 262,144-token window.
The post ships a full recipe, pinned Docker image, patches and detailed results, and the author claims it is faster than any published two-card CMP 170HX recipe they found, giving a reproducible reference for running long-context models on cheap mining GPUs.
I've been working on optimizing Qwen 3.8 Flash Next on my local setup with two CMP 170HX cards and finally got to a place where I feel like the results are worth sharing. Both cards are running over PCIe Gen2 x4 - no x16 mod or TP2.
The setup uses the W4A16 AutoRound checkpoint, a patched vLLM, pipeline parallelism and native MTP speculative decoding. The n-gram table is memory-mapped from NVMe but everything else fits on the two GPUs.
Decode throughput by concurrency — aggregate tok/s, not per-user speed:
| Concurrent requests | Code | JSON | Prose |
|---|---|---|---|
| 1 | 226 | 198 | 157 |
| 2 | 363 | 320 | 212 |
| 4 | 560 | 537 | 385 |
| 8 | 840 | 873 | 567 |
At eight concurrent code requests, the measured per-request decode rate was 116 tok/s, with 231 ms to first token.
A few other results: - 6,300–7,100 tok/s prefill on 4k–64k prompts; about 5,900 tok/s at 128k. - Prose decode held up from 158 tok/s at 4k context to 142 tok/s at 128k. - HumanEval: 157/164 (95.7%) and GSM8K: 1,271/1,319 (96.4%), both in chat mode with thinking off. - Uses a full 262,144-token window
For throughput, I used the sparkDash protocol: greedy decoding, thinking off, fixed 400-token outputs, and medians of three runs, not the best run. The long-context decode checks are an addition to that protocol.
This is faster than any published 2x CMP 170hx recipe I've found so I wanted to share with the community. These cards are flying :)
Full recipe, pinned Docker image, patches, and detailed results: https://github.com/AlexBrandes/cmp170hx-qwen38-flash-next
submitted by /u/dead-fish [link] [comments]