Reddit user runs 748B DeepSeek-V4.1-Flash on a used €2,000 PC at 7.2 tok/s
A Reddit user posting as /u/Vladowski on r/LocalLLaMA reported running DeepSeek-V4.1-Flash-1017, a 748B-parameter model, on a used PC bought for under €2,000 with an RTX 3090 24 GB, 128 GB DDR4 and an Intel i9-9900X. The run used an Unsloth UD-IQ1_S GGUF taking 233 GB on disk and a llama.cpp build from open PR #30817 with CUDA 12.8, with the model's 110 GB Engram table kept on NVMe and read row by row on demand, which the user says is how it fit. The user reported generation at 7.2 tok/s and said the answer contained a physics error, adding that they saw little practical use for the setup while Qwen models work well on the same hardware.
The post is an unverified anecdote, but it suggests a 748B-parameter model can technically be made to run on used consumer hardware through 1-bit quantization and on-demand loading of the Engram table.
I bought a used PC with an RTX 3090 24 GB, 128 GB DDR4 and an Intel i9-9900X last year for under €2,000.
With the release of DeepSeek-V4.1-Flash-1017 I decided to try running it on my machine. It is of course the smallest quant available (1-bit), but I was still happy to see it working at 7.2 tok/s.
The answer has a physics error in it (wondering if someone can spot it), and I don't see much practical use for it. Qwen models actually work pretty well on my setup. Still, it felt really nice to see a 748B parameter model running on such hardware.
Here are some technical details:
llama-server -m DeepSeek-V4.1-Flash-1017-UD-IQ1_S-00001-of-00005.gguf \ -ngl 99 --n-cpu-moe 36 -ot "per_layer_token_embd.weight=CPU" \ -lm mmap -lzm on -t 10 -c 16384 --jinja \ --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 \ --chat-template-kwargs '{"enable_thinking":false}'
Now I only need to decide whether to keep the model or free up the disk space :) submitted by /u/Vladowski
[link] [comments] |
#DeepSeek#DeepSeek-V4.1-Flash-1017#Unsloth#llama.cpp#NVIDIA#RTX 3090#Intel#Qwen#r/LocalLLaMA