Developer runs DeepSeek V4.1 Flash IQ1_S on RTX 5090 at 53.6 tok/s
A developer posting on r/LocalLLaMA reports running DeepSeek V4.1 Flash 1017 at Unsloth's UD-IQ1_S quantization in Basalt, an inference engine they are developing, on an RTX 5090 plus an RTX 5060 Ti 16 GB and 128 GB of DDR5. With all 107 GiB of experts held in RAM and the 5090 used as an expert cache, the setup reached 53.6 tok/s decode and about 610 tok/s prefill, versus roughly 20 tok/s for a llama.cpp PR #30817 baseline on the same hardware and quantization. Letting the 5060 Ti compute some experts raised decode to 63.5 tok/s, but the author reports no perplexity or KL-divergence quality measurements yet, only greedy-match agreement with llama.cpp on test prompts.
The post offers an early, self-reported data point on running a large quantized model with expert offload across consumer GPUs, while explicitly leaving quality measurement for later.
Been working over the last few days getting DeepSeek V4.1 Flash 1017 (Unsloth UD-IQ1_S) running in Basalt (inference engine I'm developing).
Specs: RTX 5090 (400W) + RTX 5060 Ti 16 GB (180W), 128 GB DDR5.
Methodology:
- Single 243 GB file (weights + vision + drafter). Engram table on SSD.
- All experts (107 GiB) in RAM. 5090 as an expert cache (hottest experts) and later also tried hoding and computing some experts in the 5060 Ti for a performance boost.
- Prefill tested with a 6.2k-token prompt (my daily harness).
- Baseline was llama.cpp PR #30817 with --fit, same hardware and quant.
Results:
| Setup | Decode | Prefill warm / cold |
|---|---|---|
| llama.cpp --fit | ~20 tok/s | – |
| 5090, all experts in RAM | 53.6 tok/s | 610 / 595 tok/s |
| 5090 + 5060 Ti computing its own experts | 63.5 tok/s | 610 / 610 tok/s* |
#DeepSeek#DeepSeek V4.1 Flash#Unsloth#Basalt#llama.cpp#NVIDIA GeForce RTX 5090#NVIDIA GeForce RTX 5060 Ti