单张3090运行DeepSeek-V4.1量化模型
AI 总结
一名社区开发者称，可在单张 RTX 3090 和 192 GB 内存的 Windows 环境中原生运行 DeepSeek-V4.1-Flash-1017 Q2 GGUF，无需 WSL。其系统平均生成速度为每秒 41 个 token，与 DeepSeek API 的下一词元一致率在一般文本和代码数学任务中分别为 90.2% 和 92.4%。开发者还称该量化版本可容纳完整的 1M token 上下文，其界面引擎支持多种已发布量化文件。
深度见解
该报告展示了用单张高端 GPU 本地运行长上下文 DeepSeek 模型的社区方案，但实现方式和性能数据目前仅来自一套内存受限的系统。
No, this is not a lobotomized 1-bit squeeze, its full Q2 nazeshinjite's DeepSeek-V4.1-Flash-1017 Q2 GGUF on a single 3090, natively on Windows no WSL.
On an RTX 3090 with 192 GB of RAM I'm writing at 41 tok/s (average across multiple prompts) and the output agrees with DeepSeek's own API on the next token 90.2% of the time on general text and 92.4% on code/math, aka within a point of exact routing. Fits full 1M context.
My scuffed interface engine runs the three most-downloaded quants of 1017, their files as they are:
nazeshinjite's ds4 Q2 GGUF (the default)
ISTA-DASLab's GSQ-RCO (IQ2_XS / IQ2_XXS /IQ1_M)
Unsloth's (UD-IQ1_S to UD-Q8_K_XL).
Repo: https://github.com/musicurgy/delta-inference-engine
Hardware/model:
RTX 3090 24 GB, i5-13600K, 192 GB DDR5 (4x48 GB at 5600), a PCIe 4.0 NVMe. Windows 10/11, no WSL.
Setup:
Download the ZIP (or clone), double-click Start DeepSeek.bat. The first run installs Python if needed, downloads the prebuilt engine for your CPU and the model (every file SHA-256-checked, the space checked first), and starts the server; after that the same file just starts it. Choose Model.bat switches quants. If anything goes wrong, Start DeepSeek (debug).bat packs everything needed into one zip for an issue (no prompts, user name removed).
Privacy: localhost only by default, and nothing from your prompts is written to disk unless you turn a log on.
Needs:
Credits:
The engine is a Windows port of Xolotl, itself built on Strata; several of Xolotl's latest fixes are merged back in. DeepSeek for the model, nazeshinjite for the GGUF, ISTA-DASLab for GSQ-RCO, Unsloth for their quants. MIT licensed. Special thanks to Claude for doing most of the lifting.
If there is interest I'll add some support for muti GPU or other garbage. This project is very new and needs a lot of validation on other hardware, these numbers are from my machine, feel free to post yours. submitted by /u/musicurgy
[link] [comments] |
#DeepSeek-V4.1-Flash-1017 Q2 GGUF#DeepSeek#NVIDIA RTX 3090#delta-inference-engine#ISTA-DASLab#Unsloth#nazeshinjite