二手PC跑通748B DeepSeek-V4.1-Flash
AI 总结
r/LocalLLaMA 用户 /u/Vladowski 发帖称，在一台去年以不到 2,000 欧元购入的二手 PC 上运行了 748B 参数的 DeepSeek-V4.1-Flash-1017，配置为 RTX 3090 24 GB、128 GB DDR4 内存和 Intel i9-9900X。该次运行使用占用 233 GB 磁盘空间的 Unsloth UD-IQ1_S GGUF，以及基于公开 PR #30817、CUDA 12.8 编译的 llama.cpp；模型中 110 GB 的 Engram 表不预先加载，而是留在 NVMe 上按需读取所需行，作者称这是它能跑起来的原因。该用户报告生成速度为 7.2 tok/s，并称模型回答中存在一处物理错误，表示看不到太多实际用途，同时认为 Qwen 模型在同一套硬件上表现不错。
深度见解
这是一则未经独立验证的个例，但表明通过 1-bit 量化和 Engram 表按需加载，748B 参数模型在技术上可以在二手消费级硬件上运行。
I bought a used PC with an RTX 3090 24 GB, 128 GB DDR4 and an Intel i9-9900X last year for under €2,000.
With the release of DeepSeek-V4.1-Flash-1017 I decided to try running it on my machine. It is of course the smallest quant available (1-bit), but I was still happy to see it working at 7.2 tok/s.
The answer has a physics error in it (wondering if someone can spot it), and I don't see much practical use for it. Qwen models actually work pretty well on my setup. Still, it felt really nice to see a 748B parameter model running on such hardware.
Here are some technical details:
llama-server -m DeepSeek-V4.1-Flash-1017-UD-IQ1_S-00001-of-00005.gguf \ -ngl 99 --n-cpu-moe 36 -ot "per_layer_token_embd.weight=CPU" \ -lm mmap -lzm on -t 10 -c 16384 --jinja \ --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 \ --chat-template-kwargs '{"enable_thinking":false}'
Now I only need to decide whether to keep the model or free up the disk space :) submitted by /u/Vladowski
[link] [comments] |
#DeepSeek#DeepSeek-V4.1-Flash-1017#Unsloth#llama.cpp#NVIDIA#RTX 3090#Intel#Qwen#r/LocalLLaMA