开发者称双3090运行Qwen3.8-27B达190 tok/s
一名开发者称，Geezuz双GPU CUDA内核在两块限功率280W的RTX 3090上运行Qwen3.8-27B时，单请求达到190 tok/s，八请求合计622 tok/s。该开源项目通过PCIe在两块GPU间切分计算，支持张量核心量化路径和MTP草稿，并附带兼容OpenAI的服务器，但速度及输出质量对比尚未获得独立验证。
该实现为消费级GPU上的Qwen高吞吐本地推理提供了新的开源选择，不过开发者的八请求上下文测试显示vLLM仍领先约15%。
Geezuz, its fast!
Qwen3.8-27B, two RTX 3090s power-limited to 280 W, plain PCIe, no NVLink, at 0 Context: (tests showing results with context are found on the next table)
| Requests | Q4_K_M | Q8_0 |
|---|---|---|
| 1 request | 190 tok/s | 143 tok/s |
| 8 requests at once (total) | 622 tok/s | 610 tok/s |
llama.cpp's fastest two-GPU setup did about 100 tok/s on the same prompts. This did 164 on average and 214 on code, at the same output quality (KL divergence against llama.cpp).
It's a fork of L-Forster's single-GPU megakernel from open-jet, where the whole forward pass runs as one persistent CUDA kernel. My fork splits it across two cards, which sync and exchange partial results directly over PCIe, and decodes up to 8 requests in one cycle with tensor-core paths for Q4_K, Q5_K, Q6_K and Q8_0.
The model drafts its next tokens with its own MTP head, and verifying 5 drafted tokens costs almost the same as generating 1, because the weights are only read once per pass.