Developer Runs DeepSeek-V4.1-Flash-1017 Q2 at 41 Tokens per Second
A community developer reported running the DeepSeek-V4.1-Flash-1017 Q2 GGUF natively on Windows with one RTX 3090 and 192 GB of RAM, without WSL. On that system, the developer measured an average of 41 tokens per second and next-token agreement with DeepSeek's API of 90.2% on general text and 92.4% on code and mathematics. The developer also claimed the quantization fits the full 1M-token context and said the interface engine supports several published quantization sets.
The report offers a community-tested path for running a large-context DeepSeek model on a single high-end GPU, although its implementation and measurements remain limited to one developer's memory-constrained system.
No, this is not a lobotomized 1-bit squeeze, its full Q2 nazeshinjite's DeepSeek-V4.1-Flash-1017 Q2 GGUF on a single 3090, natively on Windows no WSL.
On an RTX 3090 with 192 GB of RAM I'm writing at 41 tok/s (average across multiple prompts) and the output agrees with DeepSeek's own API on the next token 90.2% of the time on general text and 92.4% on code/math, aka within a point of exact routing. Fits full 1M context.
My scuffed interface engine runs the three most-downloaded quants of 1017, their files as they are:
nazeshinjite's ds4 Q2 GGUF (the default)
ISTA-DASLab's GSQ-RCO (IQ2_XS / IQ2_XXS /IQ1_M)
Unsloth's (UD-IQ1_S to UD-Q8_K_XL).
Repo: https://github.com/musicurgy/delta-inference-engine
Hardware/model:
RTX 3090 24 GB, i5-13600K, 192 GB DDR5 (4x48 GB at 5600), a PCIe 4.0 NVMe. Windows 10/11, no WSL.
Setup:
Download the ZIP (or clone), double-click Start DeepSeek.bat. The first run installs Python if needed, downloads the prebuilt engine for your CPU and the model (every file SHA-256-checked, the space checked first), and starts the server; after that the same file just starts it. Choose Model.bat switches quants. If anything goes wrong, Start DeepSeek (debug).bat packs everything needed into one zip for an issue (no prompts, user name removed).
Privacy: localhost only by default, and nothing from your prompts is written to disk unless you turn a log on.
Needs:
Credits:
The engine is a Windows port of Xolotl, itself built on Strata; several of Xolotl's latest fixes are merged back in. DeepSeek for the model, nazeshinjite for the GGUF, ISTA-DASLab for GSQ-RCO, Unsloth for their quants. MIT licensed. Special thanks to Claude for doing most of the lifting.
If there is interest I'll add some support for muti GPU or other garbage. This project is very new and needs a lot of validation on other hardware, these numbers are from my machine, feel free to post yours. submitted by /u/musicurgy
[link] [comments] |
#DeepSeek-V4.1-Flash-1017 Q2 GGUF#DeepSeek#NVIDIA RTX 3090#delta-inference-engine#ISTA-DASLab#Unsloth#nazeshinjite