Llama.cpp
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 88
- Reddit user reports mixed Qwen3.8-Flash-Next performance
A Reddit user said Qwen3.8-Flash-Next performed well on coding and adjacent tasks, including evaluating open-source repositories and running models. In general research, however, the user reported confusion, verbose and dense prose, and difficulty maintaining consistent answers. The test used Pi with llama.cpp, UD_Q4_K_M quantization, and MTP, with sampling parameters copied from Qwen3.8 27B.
r/LocalLLaMA · 🔥 5 - Betelgeuse reaches 24 tok/s for Qwen 3.8 Flash Next on dual-channel RAM
In a self-published r/LocalLLaMA report, the Betelgeuse developer tested Qwen 3.8 Flash Next IQ4_XS on an RTX 3060 12GB with 16GB of system RAM. Betelgeuse reached 24.0 tok/s with dual-channel 2x8GB DDR4 and 14.0 tok/s with a single 16GB stick, while the developer reported byte-exact output and no dropped experts; stock llama.cpp and Strata were slower in the same tests.
r/LocalLLaMA · 🔥 6 - Reddit user seeks advice on Threadripper PRO 3995WX LLM server
A Reddit user asked whether to buy a used workstation listed in Iraq with an AMD Ryzen Threadripper PRO 3995WX, 224 GB of DDR4-3200 RAM, and an ASUS Pro-series motherboard. The poster wants to evaluate it as a local LLM inference server with four to eight RTX 3090 GPUs, but the configuration, exact motherboard model, price, and upgrade feasibility remain unverified.
r/LocalLLaMA · 🔥 4 - ik_llama.cpp adds native PQ2_0 support for Ternary Bonsai 2
A community guide reports that ik_llama.cpp now provides native CPU and CUDA support for PrismML’s Ternary Bonsai 2 in PQ2_0 format. It cites merged CUDA kernels, a lossless repacking script for upstream TQ2_0 and TQ1_0 weights, and runtime repacking with `-rtr 1` for faster CPU prompt processing; no independent benchmarks are provided.
r/LocalLLaMA · 🔥 4 - Basalt developer tests lossy MoE routing options
A Basalt developer reported an experiment with Qwen 3.8 Flash Next that tested two opt-in, lossy inference options: routing bias, which favors experts already on the GPU, and tail skip, which omits lower-ranked experts not resident on the GPU. Both reduce transfers from RAM but can change outputs. In a simulated small-card configuration, the shipped settings increased prose throughput by 56%, while ladder and fact scores fell by 5.2 and 2.3 points, respectively. The developer’s deliberately worst settings reduced fact accuracy from 38.1% to 2.7%. The test covered one model on one rig, used one seed for ladders and three for facts, and measured speed with single cold runs. Basalt ships both flags disabled by default and warns when they are enabled.
r/LocalLLaMA · 🔥 6 - Betelgeuse Runs Qwen 3.8 Flash Next on Budget Hardware
Developer /u/zyxciss reported on r/LocalLLaMA that Betelgeuse, a llama.cpp fork, ran the 68GB Qwen 3.8 Flash Next MoE model on an RTX 3060 12GB with 16GB DDR4 RAM. The developer recorded 16.80 tokens per second cold, 22.03 warm, and bursts up to 48 tokens per second, while claiming byte-exact output without pruning any routed experts. Betelgeuse uses a VRAM expert-slot pool, a RAM victim cache, and io_uring paging to reduce storage reads.
r/LocalLLaMA · 🔥 5 - Prospero launches as a resource-light AI coding harness
nullprogram introduced Prospero, an open-source C++ coding harness with a Dear ImGui interface, desktop control, subagents, remote deployment, and operation deadlines. The Community Edition is free, while the Professional Edition adds embedded headless engines for macOS, Windows, and Linux; users must supply a model provider. The author claims Prospero uses around two orders of magnitude fewer resources than competing harnesses, but no independent benchmark is provided.
Hacker News 首页（100+ 分） · 🔥 15 - Local tests find unstable URL generation in Qwen3.8-Flash-Next
The Reddit report says a 1,200-run local reproduction of Cohere's trace showed Qwen3.8-Flash-Next generating highly variable destination URLs, with no match to Cohere's specific Alibaba Cloud bucket. Across two Unsloth GGUF quantization files, one llama.cpp engine, one host and one cut point, small changes in layer placement and numerical execution shifted outputs, providing evidence against a fixed destination while leaving the cause uncertain.
r/LocalLLaMA · 🔥 3 - Developer Reports 110K Qwen3.8-27B Context on RTX 5070 Ti
Community developer u/jayecin shared a llama.cpp configuration that sets a 110,000-token context for a custom-weight Qwen3.8-27B GGUF model on an RTX 5070 Ti. The developer reports generation at about 55 tokens per second, with roughly 1.5GB of VRAM to spare. The setup uses the Ryzen 9800X3D integrated GPU for display output, q8_0 and q4_0 KV caches, and flash attention. The report includes configuration and runtime logs, but the performance figures have not been independently validated.
r/LocalLLaMA · 🔥 3 - 10 coding agents tested on one local Qwen 3.8 27B
A Reddit user compared 10 coding agents using the same Qwen 3.8 27B Q4 model through llama.cpp on a MacBook Pro M5 Max with 64 GB of memory. Across five trap-containing tasks, every agent passed four or five tasks, while token consumption, completion time, cache reuse, and network behavior varied substantially; Pi used the fewest median tokens, while Qwen Code used the most. Codex CLI, Cline, OpenCode, and goose attempted an internet connection on every run, although the sandbox blocked those requests.
r/LocalLLaMA · 🔥 1 - SafeAgents locally screens coding-agent tool calls
SafeAgents is an Apache 2.0-licensed local proxy that checks coding-agent tool calls before the harness receives them. It combines two request-based checks with about 110 deterministic atif-scan detectors to identify undeclared tools, unexpected hosts, sandbox side channels, network bypasses, namespace escapes and reward-hacking attempts. An optional OpenAI-compatible judge can assess milder findings and may run locally. Flagged calls are replaced with a text notice, while --log-only mode records attempts without altering execution. The project supports OpenAI Chat Completions and Anthropic Messages, provides setup guides for 12 harnesses and reports end-to-end CI tests for six.
r/LocalLLaMA · 🔥 0 - OpenBonsai opens BF16-to-ternary GGUF compression pipeline
The OpenBonsai developer announced an Apache-2.0 framework for converting BF16 models and fine-tunes into ternary GGUF files compatible with Prism's PTQ1_0 format. The developer reports that Qwen3.5-9B was compressed from about 18 GB to 1.997 GB, with the pipeline also applied to Qwen3-8B at 1.8 GB. Its initial models scored below Prism's Ternary Bonsai 2 on most reported reasoning and coding tests, and the developer describes the project as a research preview.
r/LocalLLaMA · 🔥 0 - Mattt releases Swift packages for on-device speech and decisions
Mattt announced AnySpeechRecognizer and AnyDecisionModel, two Swift packages for Apple platforms. AnySpeechRecognizer mirrors Apple’s speech-to-text API, supports selectable engines such as NVIDIA Parakeet and Alibaba Qwen3-ASR running locally through Core ML, and adds speaker diarization. AnyDecisionModel returns typed answers and probabilities for yes-or-no, multiple-choice, and rating-scale questions, using on-device models through MLX, Core AI, or llama.cpp, or decision models from Jev or Ollama. The packages can download open models from Hugging Face. Reported speed, memory, and word-error-rate results came from one machine and a few datasets; testing was limited to macOS 27, with no iPhone results.
Hugging Face · 🔥 0 - Ollama v0.40.13 expands LFM3 support and patches Metal crash
Ollama v0.40.13 brings Liquid AI's LFM3 models, including lfm3-audio and lfm3-omni, to Linux and Windows in addition to Apple silicon. The release also includes LFM3 GGUF fixes, llama-server speech changes, and a llama.cpp bump to b11963. A v0.40.13-rc1 tag added a temporary workaround for a Metal unary-library crash regression on older macOS versions.
Ollama Releases · 🔥 0 - llama.cpp PR fuses MoE matrix multiplications and GLU in MMQ
A llama.cpp pull request proposes fusing MoE gate/up matrix multiplications and GLU in the CUDA MMQ path. The PR reports prompt-processing gains of 5.7%–17.2% across DGX Spark and RTX 5090 systems using several Qwen MoE model and quantization configurations. The supplied report does not establish that the change has been merged or released, and the benchmark figures have not been independently verified.
r/LocalLLaMA · 🔥 0 - llama.cpp PR enables LFM3 video input support
Pull request #31283 in the ggml-org/llama.cpp repository, submitted by tdakhran, is titled “mtmd: enable video input support for LFM3 model family.” A Reddit post accompanying the link says, “now you can show your videos to tiny LFM3 models.” The supplied report does not say whether the pull request was merged or provide independent tests, benchmarks, or model-quality results.
r/LocalLLaMA · 🔥 0 - Ornith 1.5 tuned for long-context inference on one Tesla P40
Developer sargetun123 reported running Ornith 1.5 35B-A3B on a single 24GB Tesla P40 using a custom llama.cpp fork, Q4_0 weights, q8_0 KV cache and MTP drafting. The reported setup supports a full 262k context and reached about 60.8 tokens/s decode without MTP at zero context and 29.1 tokens/s at 100k context, while a real server with MTP generated 35–39 tokens/s for a 101k-token prompt. The developer also reported that disabling ECC freed 1.5GB of VRAM and improved decoding by about 9%.
r/LocalLLaMA · 🔥 0 - LiquidAI releases LFM3-Omni-2B-GGUF with llama.cpp support
LiquidAI released the LFM3-Omni-2B-GGUF model, according to a r/LocalLLaMA post. The community report says tdakhran coordinated a same-day llama.cpp release, while a user highlighted support for Polish text-to-speech and speech-to-text and described the 2B model as suitable for on-device deployment.
r/LocalLLaMA · 🔥 0 - Exowatt builds reusable SageMaker fine-tuning pipeline
AWS Machine Learning Blog reports that Exowatt used Amazon SageMaker AI serverless model customization to create a repeatable pipeline for task-specific fine-tuned models. A fine-tuned Qwen3-1.7B handled offline alarm triage, while a fine-tuned Qwen3-14B supported cloud energy-market analysis; AWS claims higher accuracy than frontier models at up to 49× lower cost, though those results were not independently verified here. The pipeline turns operational data into a certified training set, evaluated model and deployment artifact, reducing Exowatt's reported workflow from months to days.
AWS Machine Learning Blog · 🔥 0 - llama.cpp Tensor Splitting Shows Promise Over PCIe 3.0 x1
A community test running Gemma 4 31B Q5 on two RTX 3060 12GB GPUs reported that switching llama.cpp from layer splitting to tensor splitting raised throughput from 14 to 20 tokens per second, a 42% increase, with the second GPU connected through PCIe 3.0 x1. The author said the exact home configuration was not extensively tested while awaiting a PSU upgrade. A separate test on two RTX 5060 Ti 16GB GPUs with PCIe Gen 3 x8 links recorded 20.7 tokens per second for tensor splitting versus 11.7 for layer splitting during prompt generation, while decoding showed transfer rates of roughly 90–110 MB/s in each direction. The author cautioned that the results do not prove PCIe 3.0 x1 has no performance penalty and that average throughput does not capture latency, synchronization overhead, or short-transfer bursts.
r/LocalLLaMA · 🔥 0 - Pumpkin open-sources a graphical manager for llama.cpp
Developer u/gbolgurov published Pumpkin on GitHub as an open-source graphical manager that detects hardware, selects a compatible llama.cpp build and quantization, and finds models installed through LM Studio or Ollama. The developer says Pumpkin can reach a running model in about five minutes and adds chat, web search, tools, MCP servers, Claude-style skills, document questions and the Pumpkin Code coding agent. Image and short-video generation require a separate stable-diffusion.cpp installation, while phone support is claimed but not properly tested.
r/LocalLLaMA · 🔥 0 - Six Inference Engines Benchmarked on RTX 5090
A TensorSharp author benchmarked six inference engines running Qwen3.8 27B on one RTX 5090 32 GB, with and without DFlash2 speculative decoding. NInfer led speculative and aggregate concurrent throughput, while llama.cpp recorded the highest plain single-request decode rate; the tests also covered 32K prompts, image input, and a four-turn tool loop. The report attributes results to complete configurations because checkpoints, drafters, and KV formats differed, and its author disclosed being TensorSharp’s creator.
r/LocalLLaMA · 🔥 0 - Zest launches a GUI for running local models
Zest creator Yudhanjaya introduced Zest as an open-source desktop interface that bundles the Lemonade model server (which installs llama.cpp and needed backends) and the Pi agent, giving people who do not want to deal with command-line servers and parameters a friendly way to run local models. The creator says users download an AppImage, go through a start tour, and get sane defaults baked in, such as a 128k context, plus model recommendations the creator says were tried and tested on systems with 10GB to 26GB VRAM; these claims have not been independently verified. The creator says Zest is not aimed at advanced users and points them to Strata and Opencode in the Help section. A separate product listing describes invoking an agent workflow with a keyboard shortcut and gesture.
r/LocalLLaMA · 🔥 0 - Ollama v0.40.12-rc0 bumps llama.cpp to b11901
Ollama tagged the v0.40.12-rc0 release candidate, and its release entry states that llama.cpp was updated to version b11901. The associated GitHub commit carries GitHub’s verified signature and identifies GPG key B5690EEEBB952194, although loading errors left parts of the captured release page unavailable.
Ollama Releases · 🔥 0 - Hob launches open-source desktop AI for local model workflows
Creator Christian announced Hob, a free MIT-licensed desktop app for Mac, Windows and Linux that runs open-weight models locally and can work with local files. The app reportedly selects a model based on installed hardware and supports sandboxed Python, web search powered by SearXNG, MCP connectors, image recognition and filesystem operations; its privacy, sandboxing and model-selection behavior has not been independently verified.
Hacker News 首页（100+ 分） · 🔥 0 - MiniMax-M3.1-Flash Gets a 2-Bit GGUF for Local Deployment
Unsloth AI and MiniMax report that Dynamic GGUF quantization enables local deployment of the 428B-parameter MiniMax-M3.1-Flash model. MiniMax says the 2-bit version has a 143.9GB footprint and 91.5% top-1 token agreement across seven domains, while Unsloth reports 91.5% retained top-1 accuracy, a 45% size reduction and operation on a system with 150GB of RAM. The model can be self-hosted through Unsloth Desktop and llama.cpp, with an OpenAI-compatible API available for existing applications and agents.
UnslothAI (X) · 🔥 0 - Arm Mali Update Adds Matrix Operations for Unified Edge AI
A 36Kr report says Arm updated the Mali GPU architecture with upgraded matrix-operation resources and unified scheduling for graphics and AI workloads across automotive, mobile and embedded devices. It also covers Imagination’s E-series GPU IP, which the company says delivers 4.4 times the Conv2D performance, 4.8 times the GEMM performance and 4.7 times the prefill performance of the preceding D-series. Imagination is also working to submit an optimized Llama.cpp backend upstream and plans native support for PyTorch and ONNX Runtime.
36氪 人工智能 · 🔥 0 - Swift 1.5 benchmark shows coding-agent token savings on Strix Halo
A r/LocalLLaMA user compared UkisAI’s Swift 1.5 fine-tune of Qwen3.8-Flash-Next with Unsloth’s quantization across Halogen, gufo and llama.cpp on an Asus ROG Flow Z13 with 128 GB of memory and a 70 W power limit. Swift produced 56,000–63,000 output tokens across 11 Pi coding-agent tasks, versus 75,000–99,000 for Qwen3.8-Flash-Next, and the gufo Swift setup completed them in 24 minutes compared with 33 minutes for gufo with Unsloth’s UD-Q4_K_XL. The results were mixed on math and LiveCodeBench, and the author cautioned that they came from a single run, so differences of one or two solved problems could reflect chance.
r/LocalLLaMA · 🔥 0 - Ternary Nemotron 30B-A3B GGUF model repository shared
A Hugging Face repository identified in an r/LocalLLaMA post presents a GGUF model called Ternary Nemotron 30B-A3B. The submitter describes it as a ternary, 1.58-bit mixture-of-experts model that runs on a 12 GB GPU, supports a 368K-token context, and reaches 172 tokens per second decoding and more than 7,000 tokens per second prefill. The post says one file works with stock llama.cpp and the PrismML/Bonsai fork, listing specific builds where each was validated.
r/LocalLLaMA · 🔥 0 - Blind test compares five self-hosted AI agents on Qwen 3.8 27B
Reddit user /u/GapNew4766 reported a scripted blind test of OpenCode, goose, Atomic Agent, Hermes and OpenClaw, all running locally on an M5 Max MacBook Pro with 64GB of memory. Each used an uncensored Qwen 3.8 27B model in llama.cpp and completed the same five tasks, including merging contact lists, renaming a function across three files, fixing a bug, counting log errors and converting a receipt PDF into a spreadsheet. All five passed three tasks, while four passed the bug-fix and receipt-conversion tasks; the agent-to-label mapping remained hidden at publication.
r/MachineLearning · 🔥 0
Experience and discussion from the community
Share my Llama.cpp experienceAsk about Llama.cpp
Nobody has shared their experience with Llama.cpp yet.