RustingStrata在单张RTX 3060上运行DeepSeek-V4.1-Flash
一则Reddit帖子称，纯Rust GGUF推理引擎RustingStrata可在单张RTX 3060 12 GB和30 GB内存上运行249 GB的DeepSeek-V4.1-Flash IQ2_XXS模型。该方案通过页缓存从SSD流式加载模型专家，并使用9 GB显存LRU缓存重复使用的专家，注意力机制和共享矩阵向量运算则在GPU上运行。在提交者的对比中，RustingStrata的解码速度为每秒2–3.6个token，高于llama.cpp b12116的每秒0.3–1.5个token，但当时尚不支持采样和工具调用，上下文上限为4096，结果也可能受页缓存状态影响。
据报道，这种从SSD向显存流式加载专家的方案可在有限GPU内存下运行超大MoE模型，但性能和正确性目前仍只是项目提交者未经独立验证的说法。
RustingStrata now runs DeepSeek-V4.1-Flash (249 GB) on a single RTX 3060 12 GB with 30 GB RAM
RustingStrata is a pure-Rust GGUF inference engine with CUDA and Vulkan backends. It now loads DeepSeek-V4.1-Flash at IQ2_XXS (249 GB) and runs it on a 12 GB card with 30 GB of RAM.
The experts stream from SSD through the page cache. A 9 GB LRU cache in VRAM holds the experts that get reused. Everything else (attention, shared matvecs) runs on the GPU.
| RustingStrata | llama.cpp b12116 (--cpu-moe) | |
|---|---|---|
| Decode | 2-3.6 tok/s | 0.3-1.5 tok/s |
| Prefill | ~2 tok/s short prompts, ~0.5 tok/s at 1400 tokens | 0.5-0.9 tok/s |
| Top-1 agreement with llama.cpp (teacher-forced) | 189/192 on short prompts, 21/23 on a 1425-token prompt | - |
| Sampling, tool calls | not yet | yes |
Same box for both: RTX 3060 12 GB, 30 GB RAM, 12 cores.
#RustingStrata#DeepSeek-V4.1-Flash#llama.cpp#RTX 3060#GGUF#CUDA#Vulkan