双CMP 170HX本地跑Qwen 3.8 Flash Next
r/LocalLLaMA 用户 /u/dead-fish 发布了在两张 CMP 170HX 矿卡上运行 Qwen 3.8 Flash Next 的优化方案，两张卡均通过 PCIe Gen2 x4 连接，未做 x16 改造，也未使用 TP2。方案采用 W4A16 AutoRound 量化权重、打过补丁的 vLLM、流水线并行和原生 MTP 投机解码，n-gram 表从 NVMe 内存映射，其余部分全部放进两张显卡。作者给出的实测数据显示：8 路并发代码请求时聚合解码吞吐为 840 tok/s，单请求解码约 116 tok/s、首 token 延迟 231 ms；4k–64k 提示词预填充约 6,300–7,100 tok/s，128k 时约 5,900 tok/s，并可使用完整的 262,144 token 上下文窗口。
这是一份附带完整配方、Docker 镜像、补丁和详细数据的社区方案，作者称其速度超过他找到的任何已公开的双 CMP 170HX 配方，为廉价矿卡运行大上下文模型提供了可复现的参考。
I've been working on optimizing Qwen 3.8 Flash Next on my local setup with two CMP 170HX cards and finally got to a place where I feel like the results are worth sharing. Both cards are running over PCIe Gen2 x4 - no x16 mod or TP2.
The setup uses the W4A16 AutoRound checkpoint, a patched vLLM, pipeline parallelism and native MTP speculative decoding. The n-gram table is memory-mapped from NVMe but everything else fits on the two GPUs.
Decode throughput by concurrency — aggregate tok/s, not per-user speed:
| Concurrent requests | Code | JSON | Prose |
|---|---|---|---|
| 1 | 226 | 198 | 157 |
| 2 | 363 | 320 | 212 |
| 4 | 560 | 537 | 385 |
| 8 | 840 | 873 | 567 |
At eight concurrent code requests, the measured per-request decode rate was 116 tok/s, with 231 ms to first token.
A few other results: - 6,300–7,100 tok/s prefill on 4k–64k prompts; about 5,900 tok/s at 128k. - Prose decode held up from 158 tok/s at 4k context to 142 tok/s at 128k. - HumanEval: 157/164 (95.7%) and GSM8K: 1,271/1,319 (96.4%), both in chat mode with thinking off. - Uses a full 262,144-token window
For throughput, I used the sparkDash protocol: greedy decoding, thinking off, fixed 400-token outputs, and medians of three runs, not the best run. The long-context decode checks are an addition to that protocol.
This is faster than any published 2x CMP 170hx recipe I've found so I wanted to share with the community. These cards are flying :)
Full recipe, pinned Docker image, patches, and detailed results: https://github.com/AlexBrandes/cmp170hx-qwen38-flash-next
submitted by /u/dead-fish [link] [comments]