r/LocalLLaMA

Headlines · 3

1. Oct 21
   Reddit user runs 748B DeepSeek-V4.1-Flash on a used €2,000 PC at 7.2 tok/s

   A Reddit user posting as /u/Vladowski on r/LocalLLaMA reported running DeepSeek-V4.1-Flash-1017, a 748B-parameter model, on a used PC bought for under €2,000 with an RTX 3090 24 GB, 128 GB DDR4 and an Intel i9-9900X. The run used an Unsloth UD-IQ1_S GGUF taking 233 GB on disk and a llama.cpp build from open PR #30817 with CUDA 12.8, with the model's 110 GB Engram table kept on NVMe and read row by row on demand, which the user says is how it fit. The user reported generation at 7.2 tok/s and said the answer contained a physics error, adding that they saw little practical use for the setup while Qwen models work well on the same hardware.

   r/LocalLLaMA · 🔥 0
2. Oct 19
   Community recipe runs Qwen 3.8 Flash Next on two CMP 170HX GPUs at 840 tok/s aggregate

   Reddit user /u/dead-fish on r/LocalLLaMA published an optimization recipe for running Qwen 3.8 Flash Next on two CMP 170HX mining cards, both connected over PCIe Gen2 x4 with no x16 mod and no TP2. The setup uses a W4A16 AutoRound checkpoint, a patched vLLM, pipeline parallelism and native MTP speculative decoding, with the n-gram table memory-mapped from NVMe and everything else fitting on the two GPUs. The author reports 840 tok/s aggregate decode for eight concurrent code requests, about 116 tok/s per request with 231 ms time to first token, 6,300–7,100 tok/s prefill on 4k–64k prompts and about 5,900 tok/s at 128k, while using the full 262,144-token window.

   r/LocalLLaMA · 🔥 0
3. Oct 18
   Developer claims near-5x token speedup for a 27B model on RTX 3090

   Wccftech reports that a developer on Reddit's LocalLLaMA forum said Qwen3.8-27B on an NVIDIA RTX 3090 went from 82 tok/s to 382 tok/s after releasing and using a custom inference engine, a gain of nearly 5x. On a 25,000-token document task, speed rose from 260 to 382 tokens per second, while ordinary chat reached about 133 tok/s, still above the original 82 tok/s. The post also describes extending the context window to 138,696 tokens by storing working memory in a more compact format; the optional longer checking step uses half the request slots and 8,000 tokens of context, and Wccftech says it has not independently verified the figures.

   Wccftech · 🔥 0

Experience and discussion from the community

Share my r/LocalLLaMA experienceAsk about r/LocalLLaMA