Betelgeuse developer reports faster out-of-core DeepSeek inference
AI summary
A Betelgeuse developer reports running DeepSeek V4.1 Flash Q8_XL (1017) on an RTX 3060 12GB with 16GB RAM, achieving 61.80 tok/s warm prefill and roughly 18–19 tok/s decode. The implementation combines resident cache partitioning, single-pass layer-major prefill and end-of-pass VRAM consolidation to reduce the bottleneck in out-of-core mixture-of-experts inference. These results are self-reported, and the comparison with Strata uses different hardware configurations.
In-depth insight
The reported approach may offer a practical way to improve first-token latency for large out-of-core MoE models on consumer hardware, but independent reproduction is still needed.