Llama.cpp
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 88
- Reddit user seeks Qwen3.5-9B QLoRA fine-tuning advice
A Reddit user is seeking advice on fine-tuning Qwen3.5-9B with QLoRA using about 500 GB of mixed images, scanned documents, PDFs, Word files, and Excel files. They report running a Q5_K_XL build on an ASUS ESC4000 G3 with 128 GB of RAM and eight NVIDIA T4 GPUs, using separate llama.cpp instances behind a round-robin nginx load balancer. Their internal harness reportedly measured 35–40 tokens per second for an average single user and a 3–5 second time to first token, but the post does not provide a completed fine-tuning workflow or validated answer.
r/LocalLLaMA · 🔥 0 - Hobbyist boosts Strata decoding for Qwen3.8-Flash-Next
A hobbyist running IQ3_S Qwen3.8-Flash-Next on Strata 0.1.43 with an RTX 4090, 48 GB RAM and a Ryzen 9 3900X reported 2,076-token prompt processing and an average decode rate of 95–105 tokens per second at a 200,000-token context. A separate grid-search result rose from 97 to 142 tokens per second with --adapt-async 1, --kv-resident 20480 and STRATA_ADAPT_LAG=2. The author also found that --ple-io ram, a large explicit --expert-cache and --no-prefill-borrow reduced throughput by 90%, 72% and 46%, respectively.
r/LocalLLaMA · 🔥 0 - Developer releases Strata fork for 64GB Jetson AGX Orin
A Reddit developer released a fork of Strata v0.1.45 for the 64GB Jetson AGX Orin, primarily to run Qwen3.6-35B-A3B. The developer reported about 21 tok/s decoding with MTP-3 in llama.cpp, around 30 tok/s in earlier Strata runs, and roughly 40–41 tok/s in controlled short-prompt tests with sampled MTP-2. The author said these were setup-specific results rather than matched llama.cpp-versus-Strata benchmarks. The fork also includes unified INT8 KV caching, prefix checkpoints and optional KV preallocation. The developer disclosed that it was heavily AI-assisted, said it had been tested on their Orin, and invited feedback from other owners.
r/LocalLLaMA · 🔥 0 - TradeMAV’s $109 RAMDeck Hub Crowdfunds Multi-Device AI Inference
TradeMAV is crowdfunding RAMDeck on Indiegogo for $109, offering a low-power hub that coordinates model inference across Windows PCs, Macs, Linux systems and Android phones. It divides model layers among devices rather than merging their RAM at the hardware level, building on llama.cpp’s free RPC backend and adding device discovery, layer placement, failure recovery, a dashboard and an OpenAI-compatible API. Vendor demonstrations showed fast MoE results but much slower dense-model performance of around two tokens per second, while the hub’s processor, memory, ports and power draw remain undisclosed.
TechRadar · 🔥 0 - llama.cpp PR proposes merging Qwen tensors during GGUF conversion
A pull request by am17an proposes merging Qwen tensors when converting models to GGUF in llama.cpp. The supplied Reddit post links to ggml-org/llama.cpp pull request #31028 but does not confirm that it was merged or provide implementation details.
r/LocalLLaMA · 🔥 0 - Yomu brings offline, local-AI manga translation to Android
The Yomu developer has released Yomu, an open-source Android app that detects speech bubbles, translates Japanese manga pages with a local llama.cpp-based LLM, and redraws them in English, Portuguese, Spanish, French, or German. The developer says page data remains on the phone, and the app requires an arm64 device plus a one-time model download of about 1.1 GB. Claude Code wrote most of the Kotlin app, overlay service, Compose UI, and C++ JNI bridge, while the developer handled decisions, review, and phone testing.
r/ClaudeAI · 🔥 0 - llama.cpp CUDA PR fuses FFN operations for reported speed gains
llama.cpp pull request #29948 proposes fusing CUDA FFN gate/up matrix multiplications and GLU in MMQ with interleaved gate/up rows. Its llama-bench results report gains of 1.9%–10.7% over master for listed quantizations of Qwen3.8-27B, Qwen3.6-27B, and Llama-3.1-8B on NVIDIA DGX Spark and RTX 5090. Tests used full GPU offload, Flash Attention, a 16,384-token prompt, and five repetitions.
r/LocalLLaMA · 🔥 0 - llama.cpp PR Proposes DeepSeek-V4.1-Flash Support
llama.cpp Pull Request #31086, titled “model: DSV4.1 flash by am17an,” links to the DeepSeek-V4.1-Flash model page and technical report. The submitter described the model as having 763 billion parameters, but the supplied report does not independently verify that figure.
r/LocalLLaMA · 🔥 0 - TensorSharp adds MiniMax-M3.1-Flash support and reports 8× A40 results
A TensorSharp developer reports adding MiniMax-M3.1-Flash support on the first day of its open-weight release, including GGUF inference, dedicated CUDA kernels for MiniMax Sparse Attention, text and image inputs through an OpenAI-compatible API, tool calling, multi-GPU execution, and DSpark speculative decoding. In early tests on eight NVIDIA A40 GPUs, the developer reports 64K executor-prefill throughput of 2,567 tok/s versus 262 tok/s on a tested llama.cpp branch, but says the results came from one host, one quantization, and an uncommitted build without public raw records.
r/LocalLLaMA · 🔥 0 - Developer reports llama.cpp gains on legacy NVIDIA GPUs
A developer reported improvements to llama.cpp for older NVIDIA GPUs, particularly Kepler and other pre-Ampere architectures. The changes include fixes to cuBLAS compute-type selection that let some hardware that previously failed during initialization run, along with CUDA graphs and Windows NCCL integration. In preliminary testing with MTP Qwen3.6-35B-A3B at Q8_0 quantization on four Tesla K40c GPUs, generation speed rose from about 35 to 55 tokens per second after enabling CUDA graphs under the same reported clock configuration. The developer also reported a 5–10% end-to-end improvement with Windows NCCL, throughput above 60 tokens per second, and prompt-processing speeds ranging from 70 to 100 tokens per second. These were preliminary results from the developer’s own system, not universal benchmarks.
r/LocalLLaMA · 🔥 0 - ik_llama.cpp Fork Update Adds RTX 2060 Inference Results
IceFog72 released an updated ik_llama.cpp fork and invited community feedback on r/LocalLLaMA. The developer self-reported about 45–48 tokens per second on short prompts, about 37 tokens per second at a 28K context length, and prefill speeds of roughly 430–700 tokens per second on an RTX 2060 with 6 GB of VRAM. The update retains the core idea of the earlier approach but has a slightly different implementation and still requires cleanup and broader testing.
r/LocalLLaMA · 🔥 0 - Community post reports 262K-context Qwen-Flash-Next Coder on 8 GB VRAM
A Reddit community post reports running a heavily quantized Qwen-Flash-Next Coder through llama.cpp with 262K context, 8 GB of VRAM and 24 GB of RAM. The setup uses a 3.5-bit-per-weight Coder variant with selective mixture-of-experts offloading, keeping most experts in RAM, smaller experts in VRAM and n-gram data on the fastest SSD. The post reports 70 tokens per second for prompt processing and about 8 tokens per second for generation, with generation limited by PCIe. It also notes that the first run may be slower because of memory mapping.
r/LocalLLaMA · 🔥 0 - AbliteratedOutTheBox Ported to Strata
A developer has ported the AbliteratedOutTheBox inference-time abliteration engine, forked from llama.cpp, to Strata. The developer says it can use abliterated models without large amounts of VRAM or fine-tuning, but the report does not independently validate those benefits. The developer describes the Strata port as experimental and says full-model behavior and response quality still require validation.
r/LocalLLaMA · 🔥 0 - ISTA-DASLab releases 1.75–2.25-bit DeepSeek-V4.1-Flash GGUFs
ISTA-DASLab announced three GSQ-RCO GGUF builds of DeepSeek-V4.1-Flash-1017: IQ1_M at 1.75 bits per weight, IQ2_XXS at 2.00 bpw, and IQ2_XS at 2.25 bpw, with backbones ranging from 120.7GB to 155.2GB. The release post says IQ2_XXS matched the base model on the evaluated benchmarks at reasoning effort 50, while a community discussion questioned the saturated benchmarks and noted that no independent KLD or perplexity measurements were supplied. The models reportedly run through Strata on a single 12GB GPU but cannot yet run in llama.cpp because architecture support has not been merged.
r/LocalLLaMA · 🔥 0 - llama.cpp PR proposes QSA selection without kq_mask
A llama.cpp pull request by sanmai proposes building QSA selection without kq_mask for the qwen4exp branch. The report says the approach saves gigabytes of VRAM when running Qwen Flash Next and describes it as one way Strata improves use of the experts' cache.
r/LocalLLaMA · 🔥 0 - Seven Qwen 3.8 FN Inference Engines Benchmarked on One RTX 5090
The developer of Basalt benchmarked seven Qwen 3.8 FN inference engines on an RTX 5090 with 128 GB of RAM. The report’s results placed Basalt and Infernix ahead overall: Basalt recorded 354 tokens per second for structured output and 190 tokens per second at 25K context, while both reached a 3.94-second time to first token at 25K. Each engine converted NVIDIA’s NVFP4 weights with its own tool and was configured according to its documentation. Every engine received the same seven greedy requests, with one cold run per request and no warm-up.
r/LocalLLaMA · 🔥 0 - Strata now supports Qwen 3.6 35B
A Reddit post reports that Strata now supports Qwen 3.6 35B. In the same post, a user said a Strata build for the gfx906 architecture on a 16GB AMD MI50 performed worse than llama.cpp in decode speed. The report did not provide official confirmation or broader performance benchmarks.
r/LocalLLaMA · 🔥 0 - Developer Reports GGUF Q4 Inference in the Browser with WebGL2 and GLSL
A Reddit developer said an implementation was running GGUF Q4 models inside HTML by paging them into WebGL2 embedded compute buffers and using GLSL. The developer also described tokenizer and vocabulary processing as part of what they called an endless debugging challenge. The report presents the developer's account and includes an unresolved link marker.
r/LocalLLaMA · 🔥 0 - Open-Source REA Brings AI to Software Reverse Engineering
A 36Kr report citing the KuWan Laboratory describes REA, an open-source GitHub project, as a collection of MCP services and skill packages that teaches AI models how to perform reverse-engineering tasks. The report presents projects associated with the REA ecosystem, including PS5 modification, running llama.cpp locally on a hacked PS5, game-engine interoperability, unofficial macOS graphics drivers, and the PSBox project. It also highlights ArtCraft, a separate clean-room effort to develop open-source alternatives to Adobe products, which the report says was only 25% to 45% complete. According to one tester cited in the report, AI still could not analyze software protected by encryption or packing.
36氪 人工智能 · 🔥 0 - Local MoE models show large gaps in agentic coding efficiency
A Reddit author benchmarked Flint 40B-A2.5B, Ornith 1.5 35B-A3B, and Nemotron 3.5 Lightning 30B-A3B as parallel coding workers under Opus 5.5 in Atomic Agent v0.6.10. Flint completed the Three.js task in 7:36 with six fixes and $0.41 in Opus usage, versus 9:04, 14 fixes, and $0.49 for Ornith, and 30:32, 16 fixes, and $0.95 for Nemotron. The results are preliminary because each model ran only once, output quality was not assessed in detail, and the author works on the agent runtime.
r/LocalLLaMA · 🔥 0 - llama.cpp XMX matrix engine refactor nearly doubles B70 speculative decoding speeds
A Reddit user on r/LocalLLaMA reports that llama.cpp pull request #29864, which refactors Intel XMX matrix engines and is attributed to budiony, made a huge improvement to speculative decoding speeds on an Intel B70 GPU running SYCL. In before/after runs on Muse Glimmer Q6_K (30B) with DFlash2 across nine prompts, throughput rose from 38.0 to 73.0 tok/s on code_python and aggregate wall time fell from 46.86s to 26.68s. The poster notes the acceptance rate dropped from 0.7845 to 0.363 because cheaper token verification allowed n-max to be increased; the numbers are a single, unverified user report.
r/LocalLLaMA · 🔥 0 - Ling-3.0-tiny UBTree DFlash draft runs on stock llama.cpp, claims up to 61% code speedup
A Reddit user, /u/Barni275, says he extracted the pure DFlash portion of the newly published UBTree DFlash draft for Ling-3.0-tiny, dropping 7 ubtree_selector tensors, so it can run with stock mainline llama.cpp through the dflash drafter. In his own small tests, prose generation rose from 78.8 to 84.8 tok/s (+8%) and Python code generation from 79.0 to 127.2 tok/s (+61%), results he described as moderate and not independently verified. He made small patches to the llama.cpp GGUF converter, published the converted model and procedure on Hugging Face, and speculated that native UBTree support in llama.cpp would bring larger gains.
r/LocalLLaMA · 🔥 0 - Developer claims unreleased ToRc engine runs ~100GB Qwen model on two £300 ARM mini PCs
A developer posting on r/LocalLLaMA says an inference engine he is writing, codenamed ToRc and not yet released, reached 315 tok/s prompt read and 43 tok/s generation with Qwen3.6 35B A3B across two Snapdragon X mini PCs joined by a single USB4 cable, hardware he says cost about £600 in total. The same post reports 46 tok/s read and 20 tok/s generation for a roughly 95GB Qwen3.8 Flash Next IQ4_NL quant on the two boxes, versus 10.9/5.8 tok/s for llama.cpp over RPC, and 34.3/18.3 tok/s for a larger UD-Q4_K_XL quant. The author, who previously contributed ARM porting work to the Janas engine, says ToRc uses CPU cores plus the Hexagon NPU with no GPU involved; all figures are his own measurements and have not been independently verified.
r/LocalLLaMA · 🔥 0 - Kapitsa adds Flint support, claims SOTA Apple Silicon decode
A contributor to Kapitsa, an open-source inference engine written with custom Metal kernels and targeted at Apple Silicon from M1 through M6, posted that the project (version 0.2.7) has added support for Recursive's newly released Flint model, which Recursive says is optimized for coding on Macs with 32GB or more and beats Qwen's 35B sparse MoEs while nearing Qwen's 27B dense model. On a single MacBook Pro M4 Pro with 48GB, the contributor's own benchmarks put Kapitsa at 127.9 tok/s short-context decode versus 126.6 for Recursive's rec 0.1.0 and 79.6 for llama.cpp b11804, with Kapitsa's lead over rec growing from 1.06x at 8K context to 1.30x at 128K. The author notes only one Mac was tested, that Flint has not been run on an M5, that prefill on M5+ is not yet optimized, and that a KFlash tree drafter trained on eight H200s for Flint Q4, claimed to give an internal 1.4x decode speedup without accuracy loss, is due Monday; running it requires a Mac with at least 32GB and a 22.6GB download.
r/LocalLLaMA · 🔥 0 - Reddit user reports LFM2.5-24-A2B loops and fails a coding task
A Reddit user in r/LocalLLaMA says the open-weight model LFM2.5-24-A2B looped repeatedly and stumbled on simple tasks when run through a llama.cpp build optimized for certain architectures, and that it produced broken code, omitted required functionality and claimed verification without end-to-end tests on a Hermes memory-plugin task. The user says the same setup reached about 130 tokens per second on an RTX A4500, and that switching to LM Studio stabilized the erratic behavior into "correct-ish" chat responses. A community chat-template fix from Hugging Face reportedly did not change the results much. The evaluation of the coding output was mediated by ChatGPT, and no independent testing is provided.
r/LocalLLaMA · 🔥 0 - 2-bit draft plus residual verifier scheme targets models that don't fit VRAM
A Reddit user posting as Happy_Guitar3521 in r/LocalLLaMA proposed keeping a 2-bit copy of a model in GPU VRAM and the 6–8-bit residual weights in system RAM, with the GPU drafting tokens and the CPU verifying them so only activations cross PCIe. The author reports self-measured results from dequantized weights run in PyTorch: a 2-bit plus 8-bit residual setup was closer to bf16 than Q8_0 at about half the KL divergence, a 6-bit residual reached Q8_0-level quality while reading roughly 28% fewer bytes from RAM, and the 2-bit draft reusing the verifier's KV cache accepted 25–30% more tokens per check than Google's official Gemma 4 31B drafter (5.23 vs 4.20). These results are self-reported and not independently verified; the post provides no tok/s figures, says the approach only makes sense when Q8 does not fit in VRAM, and shares a public Hugging Face repository for others to experiment with.
r/LocalLLaMA · 🔥 0 - llama.cpp fork claims live inference-time abliteration without fine-tuning
A Reddit post in r/LocalLLaMA by user AccountAntique9327 introduced AbliteratedOutTheBox, described as a llama.cpp fork that abliterates any loaded model live at inference without fine-tuning or extra files. The author says the approach avoids the hassle of abliteration without requiring additional VRAM, and links to a GitHub repository under heterodoxin. The capability claims come from this single self-post and have not been independently verified.
r/LocalLLaMA · 🔥 0 - Flint MoE model support merged into llama.cpp, Reddit post says
A post on r/LocalLLaMA says a pull request adding Flint (FlintForCausalLM) support to llama.cpp, PR #30928 in the ggml-org repository, was merged after about six hours. The post describes Flint as a decoder-only mixture-of-experts Transformer from developer Recursive with 39.8B total parameters and roughly 2.5B active parameters per token including the output head, spread over 49 layers — 2 dense followed by 47 MoE layers. It lists 256 routed experts with 6 selected per token plus an always-active shared expert, 35 sliding-window and 14 full-attention layers, a documented context configuration of 262,144 tokens, and a stated knowledge cutoff of June 2026; these details come from the post alone and have not been independently confirmed.
r/LocalLLaMA · 🔥 0 - Mica v0.5: a 4B decision model that runs on one consumer GPU
A developer on r/LocalLLaMA released Mica v0.5, a 4B decision model that runs on a single consumer GPU and follows the earlier Mica v0.1. In self-reported measurements on one RTX 6000 Ada, the developer says it takes 19.8 ms per decision at 256 tokens and about 1.2 s at 32k tokens, claiming it is up to 3.8x faster than the other 4B judges measured, 12 to 19x faster than Perplexity's 27B decider and 25 to 340x faster than API decision models. The developer reports 84/111 on JevBench Hard, described as the best of any 4B model found with only Microsoft's 9B Decision-1 higher, and a JDI score of 58.8, while acknowledging that top 4B models on JDI are benchmark-focused and that training used data from the areas the benchmarks cover. Mica v0.5 handles yes/no, multiple choice and scores from 2 to 255 levels, accepts inputs up to 32k tokens, and is released under Apache-2.0 with INT8 and GGUF files, running on 8 GB cards and up.
r/LocalLLaMA · 🔥 0 - Reddit user runs 748B DeepSeek-V4.1-Flash on a used €2,000 PC at 7.2 tok/s
A Reddit user posting as /u/Vladowski on r/LocalLLaMA reported running DeepSeek-V4.1-Flash-1017, a 748B-parameter model, on a used PC bought for under €2,000 with an RTX 3090 24 GB, 128 GB DDR4 and an Intel i9-9900X. The run used an Unsloth UD-IQ1_S GGUF taking 233 GB on disk and a llama.cpp build from open PR #30817 with CUDA 12.8, with the model's 110 GB Engram table kept on NVMe and read row by row on demand, which the user says is how it fit. The user reported generation at 7.2 tok/s and said the answer contained a physics error, adding that they saw little practical use for the setup while Qwen models work well on the same hardware.
r/LocalLLaMA · 🔥 0
Experience and discussion from the community
Share my Llama.cpp experienceAsk about Llama.cpp
Nobody has shared their experience with Llama.cpp yet.