RustingStrata Runs DeepSeek-V4.1-Flash on a Single RTX 3060
A Reddit post reports that the pure-Rust GGUF inference engine RustingStrata can run the 249 GB DeepSeek-V4.1-Flash IQ2_XXS model on a single RTX 3060 12 GB with 30 GB RAM. It streams model experts from SSD through the page cache and uses a 9 GB LRU cache in VRAM, while attention and shared matrix-vector operations run on the GPU. In the submitter’s comparison, RustingStrata achieved 2–3.6 tok/s decoding versus 0.3–1.5 tok/s for llama.cpp b12116, but sampling and tool calls were not yet supported, the context cap was 4096, and results may vary with page-cache conditions.
The reported SSD-to-VRAM expert-streaming approach could enable local inference with a very large MoE model on limited GPU memory, although its performance and correctness remain unverified claims from the project submitter.
RustingStrata now runs DeepSeek-V4.1-Flash (249 GB) on a single RTX 3060 12 GB with 30 GB RAM
RustingStrata is a pure-Rust GGUF inference engine with CUDA and Vulkan backends. It now loads DeepSeek-V4.1-Flash at IQ2_XXS (249 GB) and runs it on a 12 GB card with 30 GB of RAM.
The experts stream from SSD through the page cache. A 9 GB LRU cache in VRAM holds the experts that get reused. Everything else (attention, shared matvecs) runs on the GPU.
| RustingStrata | llama.cpp b12116 (--cpu-moe) | |
|---|---|---|
| Decode | 2-3.6 tok/s | 0.3-1.5 tok/s |
| Prefill | ~2 tok/s short prompts, ~0.5 tok/s at 1400 tokens | 0.5-0.9 tok/s |
| Top-1 agreement with llama.cpp (teacher-forced) | 189/192 on short prompts, 21/23 on a 1425-token prompt | - |
| Sampling, tool calls | not yet | yes |
Same box for both: RTX 3060 12 GB, 30 GB RAM, 12 cores.
#RustingStrata#DeepSeek-V4.1-Flash#llama.cpp#RTX 3060#GGUF#CUDA#Vulkan