Developer reports 190 tok/s for Qwen3.8-27B on two RTX 3090s
A developer reports that the Geezuz dual-GPU CUDA kernel ran Qwen3.8-27B on two power-limited RTX 3090s at 190 tok/s for one request and an aggregate 622 tok/s across eight requests. The open-source project splits computation across two GPUs over PCIe, supports tensor-core quantization paths and MTP drafting, and includes an OpenAI-compatible server, but its speed and output-quality comparisons have not been independently verified.
The reported implementation offers a new open-source option for high-throughput local Qwen inference on consumer GPUs, although vLLM reportedly retained a roughly 15% lead in the developer's eight-request context benchmark.
Geezuz, its fast!
Qwen3.8-27B, two RTX 3090s power-limited to 280 W, plain PCIe, no NVLink, at 0 Context: (tests showing results with context are found on the next table)
| Requests | Q4_K_M | Q8_0 |
|---|---|---|
| 1 request | 190 tok/s | 143 tok/s |
| 8 requests at once (total) | 622 tok/s | 610 tok/s |
llama.cpp's fastest two-GPU setup did about 100 tok/s on the same prompts. This did 164 on average and 214 on code, at the same output quality (KL divergence against llama.cpp).
It's a fork of L-Forster's single-GPU megakernel from open-jet, where the whole forward pass runs as one persistent CUDA kernel. My fork splits it across two cards, which sync and exchange partial results directly over PCIe, and decodes up to 8 requests in one cycle with tensor-core paths for Q4_K, Q5_K, Q6_K and Q8_0.
The model drafts its next tokens with its own MTP head, and verifying 5 drafted tokens costs almost the same as generating 1, because the weights are only read once per pass.