Llama.cpp
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 88
- Developer runs DeepSeek V4.1 Flash IQ1_S on RTX 5090 at 53.6 tok/s
A developer posting on r/LocalLLaMA reports running DeepSeek V4.1 Flash 1017 at Unsloth's UD-IQ1_S quantization in Basalt, an inference engine they are developing, on an RTX 5090 plus an RTX 5060 Ti 16 GB and 128 GB of DDR5. With all 107 GiB of experts held in RAM and the 5090 used as an expert cache, the setup reached 53.6 tok/s decode and about 610 tok/s prefill, versus roughly 20 tok/s for a llama.cpp PR #30817 baseline on the same hardware and quantization. Letting the 5060 Ti compute some experts raised decode to 63.5 tok/s, but the author reports no perplexity or KL-divergence quality measurements yet, only greedy-match agreement with llama.cpp on test prompts.
r/LocalLLaMA · 🔥 0 - IBM updates Granite Speech with three open-weight models
IBM released Granite Speech 5.0 2B and revised Granite Speech 5.0 Turbo CTC and Granite Speech 4.1 2B Plus, all under Apache 2.0 with native Transformers support. According to IBM, the new 2B model adds multilingual transcription and translation, language-ID prompting and keyword biasing for up to 2,000 terms, while Turbo CTC gains multilingual transcription and native punctuation; 4.1 2B Plus improves speaker attribution, timestamps and long-speech performance. IBM also reports a 5.06% average English WER for Granite Speech 5.0 2B and 19,870 RTFx for Turbo CTC, though these results are vendor-provided and the former ranking is described as unofficial.
Hugging Face · 🔥 0 - Developer shares fast LoRA kernels for llama.cpp, lifting Strix Halo prompt processing
A developer posting on r/LocalLLaMA as u/woct0rdho shared LoRA kernels for llama.cpp, saying the project currently has no fast LoRA kernels and that running a model with a LoRA on Strix Halo dropped prompt processing from about 1500 to 300 t/s even though the adapter was only 35 MB and touched few modules. The author reports that a general LoRA kernel raised throughput to 1000 t/s and a fused kernel to 1400 t/s. The changes are currently optimized for Strix Halo and have not yet been upstreamed to mainline llama.cpp or strix-llama.cpp, though the author says porting to other GPUs should not be hard and invites others to reuse the code, which is also discussed in a llama.cpp GitHub issue. The post also points to a third-party Qwen3.8-Flash-Next LoRA that produces an uncensored model without re-downloading the full weights, with a tunable LoRA scale.
r/LocalLLaMA · 🔥 0 - Kapitsa 0.2 brings KFlash speculative decoding to Apple Silicon, tops self-run benchmark
Kapitsa, an open-source inference engine for Apple Silicon aimed at Qwen and Nemotron models, released version 0.2 with a new speculative decoding method its developer calls KFlash, announced in a r/LocalLLaMA post. KFlash checks a best-first tree of the draft model's candidate tokens in one pass, whereas the comparison DFlash2 method checks a single chain and stops at the first wrong guess. In the developer's own benchmarks on a MacBook Pro M4 Pro with 48GB, Kapitsa reported 63 tokens/s on Qwen3.8 27B versus 53 tokens/s for Splash 1.4.0 with DFlash2, and 95 versus 62 tokens/s on Qwen3.5 4B on an M1 Pro 16GB machine; the author cautioned readers not to trust benchmarks and to run their own.
r/LocalLLaMA · 🔥 0 - Developer reports 190 tok/s for Qwen3.8-27B on two RTX 3090s
A developer reports that the Geezuz dual-GPU CUDA kernel ran Qwen3.8-27B on two power-limited RTX 3090s at 190 tok/s for one request and an aggregate 622 tok/s across eight requests. The open-source project splits computation across two GPUs over PCIe, supports tensor-core quantization paths and MTP drafting, and includes an OpenAI-compatible server, but its speed and output-quality comparisons have not been independently verified.
r/LocalLLaMA · 🔥 0 - Hugging Face ships AnyLanguageModel 1.0 for Apple platforms
Hugging Face has released AnyLanguageModel 1.0, alongside swift-huggingface 1.0 and swift-xet 1.0, a year after introducing the project. AnyLanguageModel is described as a drop-in replacement for Apple's Foundation Models framework that lets apps choose the model and provider, and the post says tool calling and guided generation now work with every provider, with image support using Foundation Models 27 syntax. Hosted support has expanded to OpenAI's Chat Completions and Responses APIs, Open Responses, Anthropic's Messages API with adaptive thinking and effort options, Google's Gemini API, Ollama, Private Cloud Compute on OS 27, and Hugging Face Inference Providers or other OpenAI-compatible endpoints via a custom base URL; the project also runs on Linux.
Hugging Face · 🔥 0 - CPU low-bit benchmark: bitnet.cpp wrong, Bonsai correct, ik_llama.cpp fastest
A community benchmark posted on r/LocalLLaMA compared low-bit GGUF runtimes against reference outputs on an ARM Neoverse-N1 server and x86 AVX2 VMs. It reports that bitnet.cpp main (commit 0b341e5) is incorrect for BitNet b1.58 2B4T: x86 text looks fine but perplexity is about 6× the HF transformers reference on the same tokens, while ARM prints ????; the author says fixes from issue #588 and an ARM kernel, bundled in PR #642's third version dated Oct 15, are correct on both. Ternary-Bonsai 1.7B files gave correct output wherever they loaded in llama.cpp, the Prism fork and ik_llama.cpp, with prompt-processing speeds varying by file, runtime and CPU — 71 t/s for PQ2_0 in ik_llama.cpp with -rtr versus 28 in Prism — and the Q2_0 file marked recommended on the model card did not load in any of the three runtimes.
r/LocalLLaMA · 🔥 0 - Cohere adds llama.cpp support for North Micro Vision Instruct vision encoder
Cohere staffer Nick (u/nick_frosst) posted on r/LocalLLaMA that the team recently merged llama.cpp support for the vision encoder of North Micro Vision Instruct, which he said means all of Cohere's latest model architectures are now compatible with llama.cpp; the LLM portion shares the architecture of North Mini Code and Command A+, which were already supported. He also said North Micro Vision is being used as the base for several of Cohere's new releases and invited the community to try it and give feedback. In the same post, Cohere said its Cohere Transcribe speech-to-text model is now available for fine-tuning in Unsloth and Unsloth Studio. The compatibility and capability statements come from Cohere's side, with no independent verification or benchmark data provided.
r/LocalLLaMA · 🔥 0 - Tencent Hunyuan releases GGUF quantizations of Hy-Memory-30B-A3B for llama.cpp
Tencent Hunyuan published a GGUF repository for Hy-Memory-30B-A3B, a roughly 30B-parameter long-term-memory writer model with about 3B active parameters, aimed at local inference through llama.cpp. Three quantization levels are offered: Q6_K imatrix at 24.71 GB, Q8_0 at 31.99 GB and Q4_K_M imatrix at 18.24 GB, all under Apache-2.0. The model card reports LoCoMo Judge scores of 58.78% (154/262) for Q6_K, 57.63% for Q8_0 and 56.49% for Q4_K_M on a fixed 20% LoCoMo split using the Mem0 2.0.19 harness, with gpt-5.6-sol as answer and judge model; these figures are self-reported by the publisher and not independently replicated.
腾讯混元 · Hugging Face · 🔥 0 - Show HN: qwen3-4b-deslop fine-tune trains AI writing tics out of Qwen3-4B
A developer released qwen3-4b-deslop, a LoRA fine-tune of Qwen3-4B-Instruct-2507 trained to suppress assistant-style writing habits, distributed as full-precision weights and a 2.5 GB Q4_K_M GGUF build for Ollama and llama.cpp. Using the author's own regex-based slop detector, the score on 40 held-out prompts fell from 121 to 12.5 per 1,000 words, em dashes dropped from 20.5 per 1,000 words to zero, and 68% of answers landed in the range the author associates with human writing; an intermediate version scored 9.9 but produced opinion pieces without a stance and invented studies, which a second task-fit filter addressed at the cost of a slightly higher score. The author reports the model still answered 30 of 30 factual questions correctly and matched requested genre and length as often as the original, but cautions that no blind reader test has verified the score, that the task-fit checks are simple rules and the training data was filtered on the same checks, and that performance on long conversations, other languages and tool use may be no better than the original or worse.
Hacker News · AI（100+ 分） · 🔥 0 - Qwen3.8-27B and Qwen3.8-Flash-Next local inference benchmark roundup
An r/LocalLLaMA user, FeydRowan, compiled single-request decode speeds for Qwen3.8-27B and the 125B MoE Qwen3.8-Flash-Next on consumer hardware, marking rows as “verified” only when backed by a LocalMaxxing run or a paired run on the same machine. The author's own capy-flash rows were measured on 2× RTX 5070 Ti with 96 GB DDR5, while most other figures are as reported by each project; the post also compares tensor parallelism versus layer splitting, expert offload, quant formats and drafters, and calculates tokens per second per €1,000 using EU VAT-inclusive prices. It puts 2× R9700 on top for Flash-Next cost-efficiency and says 96 GB of DDR5 is about 40% of the bill for such builds, but the performance and price data are largely user-compiled and self-reported.
r/LocalLLaMA · 🔥 0 - Kapitsa: open-source Apple Silicon inference engine tuned for Qwen and Nemotron
Developer Ye at Nace.ai posted Kapitsa on r/LocalLLaMA, an Apache 2.0 open-source inference engine for Apple Silicon aimed at running coding agents on local models on Macs. Kapitsa supports only two model families — Qwen (3.5 4B, 3.8 27B, 3.6 35B-A3B and 3.8 Flash-Next) and NVIDIA Nemotron (3.5 Lightning, Bolt, Super-VL plus Dory 4B) — so the team can customize Metal kernels per architecture. The author published decode-speed tables on M1 Pro 16GB, M4 Pro 48GB and M5 Max 128GB machines, claiming it beats mlx-serve, oMLX, Ollama, MLX and llama.cpp in most cases, while trailing mlx-serve and oMLX by roughly 2-3% on Qwen3.8 Flash-Next. He also warns readers not to trust benchmarks, points them to a built-in kapitsa bench command to test themselves, notes that 4-bit quantization costs some quality, and says the new codebase should be expected to have bugs.
r/LocalLLaMA · 🔥 0 - HardenedLinux releases SonicBoom v0.0.3, a compiler-first inference runtime
The HardenedLinux community announced SonicBoom v0.0.3, an open-source GPLv3 inference runtime that it describes as compiler-first and free of Python at build or run time, using one shared library, a C ABI and Guile Scheme as its first-class language. On a single RTX 3050 8GB, the team self-reported that Gemma 4 E4B Q3_K_M reached 1.18× llama.cpp's decode speed and 1.00× prefill, while Qwen2.5-0.5B Q4_K_M reached 1.12× and 1.09×; Qwen decode at depth 512 was 0.98×. The team acknowledges it tested only one GPU and is asking NVIDIA owners to run a one-command benchmark and post results.
r/LocalLLaMA · 🔥 0 - New IQ2_K–IQ6_K quantization types proposed for llama.cpp
A r/LocalLLaMA post highlights five new quantization types proposed in a llama.cpp pull request — IQ2_K, IQ3_K, IQ4_K, IQ5_K and IQ6_K — and links to test models on Hugging Face. According to the post, the files are the same size as Q-quants, KLD improves substantially over Q-quants, inference performance suffers slightly, and creating IQ quants is somewhat slow. The poster cautions that the numbers presented cover only a single model.
r/LocalLLaMA · 🔥 0 - Blockway releases Agens Volundr 32B v1 with an MIT-licensed Agens Engine
Blockway posted Agens Volundr 32B v1 on r/LocalLLaMA, saying it addressed issues the community raised against the Preview: repetition loops on SWE-bench tasks fell from 36/50 to 0/50, SWE-bench Verified (50-task subset) rose from 22/50 to 27/50, τ²-bench from 89 to 92/120, and median SWE task time dropped from 124 to 82 minutes. The team also open-sourced Agens Engine under the MIT licence, an inference engine built for the hybrid architecture, claiming 1.4–1.6× faster single-stream decoding than its own SGLang build, plus a roughly 2.85 bits-per-weight build offering 262K context within a 16 GB memory budget. All figures are vendor self-reported: the 16 GB numbers were measured on a 48 GB card under a 16 GiB cap, GPQA drops to 73.7 at that bit-width versus 80.3 at bf16, and the team says stock vLLM cannot load the architecture while SGLang and llama.cpp still require its own builds.
r/LocalLLaMA · 🔥 0 - ASUS launches first Intel Arc Pro B70 card: 32GB ECC VRAM, 367 TOPS
ASUS has introduced its first Intel discrete GPU, the UGen Intel Arc Pro B70 32GB GDDR6, aimed at AI inference and workstation workloads — marking the vendor's return to Intel's add-in card lineup after skipping several generations. The card uses the Xe2-based Big Battlemage GPU with 32 Xe cores, 256 XMX AI engines, 32GB of GDDR6 ECC memory, 608GB/s bandwidth, PCIe 5.0 x16 and a 310W typical board power; ASUS rates INT8 throughput at up to 367 TOPS. ASUS says it supports frameworks including vLLM, llama.cpp, Ollama, PyTorch, OpenVINO, oneAPI and Vulkan, and that up to four cards can be combined for 128GB of total memory to run larger models. ASUS has not published a price; IT之家 notes comparable Arc Pro B70 cards typically sell for about $1,299 to over $2,000.
IT之家 AI · 🔥 0 - PrismML releases Ternary Bonsai 2 on-device models in 2B, 4B and 9B sizes
PrismML released the Ternary Bonsai 2 series on October 15 in 2B, 4B and 9B sizes, aimed at running AI agents locally on phones, PCs and wearables for reasoning, image understanding and tool calling, IT之家 reported. The models are built on Alibaba's Qwen 3.5 base and use ternary weights encoded only as -1, 0 and +1; PrismML says they occupy roughly 0.43GB, 0.95GB and 2GB of storage. The company says it tested six capability areas across 19 benchmarks, with the 9B model using about 40% fewer tokens on standard instruction-following and everyday math tasks, and reports up to 5x throughput and 3x energy efficiency versus full-precision counterparts on RTX 4090 and M5 Pro, plus up to 34 tokens/s for the 9B and 120 tokens/s for the 2B on iPhone 18 Pro Max. Weights are released under Apache 2.0, runnable via llama.cpp on NVIDIA GPUs and MLX on Mac, iPhone and iPad, joining the existing Bonsai 2 27B.
IT之家 AI · 🔥 0 - nFOX Lab launches AI model management platform, plans to open-source Blockhouse sandbox
AI governance startup nFOX Lab has launched a platform for managing owned large language models and said it plans to open-source Blockhouse, a security sandbox for AI workloads, under the Apache 2.0 license. According to SiliconANGLE, Blockhouse removes privileges by default and restores access through policies enforced at the Linux kernel level, routes communications through a monitored channel called Porthole, and verifies model files with SHA-256 hashes before serving. nFOX says the platform is available now while Blockhouse is in beta for qualified enterprise evaluations; the startup is self-funded, has paying beta pilot customers and is not yet profitable.
SiliconANGLE AI · 🔥 0 - Llama for Windows released: one-click local model downloads
Llama for Windows, the Windows counterpart to the macOS Llama app, is a free, open-source Windows 11 application built on Microsoft's WinUI 3 that lives in the system notification area and opens its control panel from a taskbar icon. It connects directly to the Hugging Face model hub, shows a curated model list with parameter counts, disk usage and vision support, and downloads GGUF-quantized models driven by the llama.cpp ecosystem, all without admin rights, manual compilation or complex launch parameters. The app offers an Alt+Space floating chat window, a built-in web UI for loading and unloading local models, and can start an OpenAI-compatible local server at localhost:9931 that scripts and AI agents can call, with multiple apps able to share one model file. According to the project, prompts and inference stay on the device and no telemetry, analytics events or installation identifiers are collected, though network access is still needed to fetch models and updates; it supports Windows 11 x64 and ARM64 plus NVIDIA's RTX Spark platform, and does not list Windows 10 as supported.
cnBeta · 🔥 0 - Open-source local desktop assistant Jarvis
The isair/jarvis project is a privacy-focused, local-first desktop AI assistant offering voice and text chat, personal memory, dictation and MCP tool integrations. According to the project's documentation, it runs offline on macOS, Windows and Linux through Ollama or an OpenAI-compatible local server, with speech recognition, language models and speech synthesis executing on the user's own hardware. While listening, it keeps a short temporary transcript of nearby speech so it can join an ongoing conversation when addressed by name. The maintainers say it is free for personal use with commercial use requiring contact, and they list known limitations including primary development on macOS, X11-only dictation on Linux, no dictation on macOS 26 and above, and no mobile app.
GitHub Trending · Python · 🔥 0 - Houmo open-sources AI software stack, outlines 2D and 3D SRAM-CIM chip roadmap
At its first ecosystem partner conference in Beijing on October 14, Houmo AI said it is open-sourcing its full "Houmo AI ecosystem software" stack, spanning model quantization and compilation, inference frameworks and operator libraries, domain kits and middleware, and deployment/operations tools, with 15 open-source projects to go online by January; repositories such as postmo-modelzoo, postmo-quantflow, vllm-postmo and adapted llama.cpp and vLLM builds are already visible on GitHub. The company also detailed two paths built on SRAM compute-in-memory: 2D CIM for high capacity and compute (already shipping in M30 and M50) and 3D CIM for high-bandwidth dense models, with its first 3D CIM chip A20 in production and an M6X successor to M50 in development, claimed at 3–6x the M50's overall capability. Houmo says M50 has landed more than 150 product design wins and over 100 customers within a year of mass production, including Lenovo's AI host P7, Deepshard's Truffle 1, UGREEN's HomeAgent HA100 Pro, Liaogu's ClawHouse X1 and Great Wall's N90 Pro laptop; these performance and commercial figures are vendor claims.
新智元 · 🔥 0 - Jailbroken PS5 runs Qwen3.8-27B locally at 20.9 tokens/s via GPU
Developer abhi245y built PS5LM Plus, a native GPU app on top of Mert Cobanov's PS5LM llama.cpp port, letting a jailbroken PS5 call the console GPU through a Mesa RADV driver and Vulkan; usable memory reportedly rose from about 6GB to 11.4GB, enough for a heavily quantized Qwen3.8-27B. Developer-reported tests put that model at about 20.9 tokens/s with GPU utilization at 100% and CPU at 2%, while smaller Gemma 4 E4B and Llama 3.2 3B models reached roughly 61 and 124 tokens/s. PS5LM Plus also offers a LAN web chat interface and OpenAI/Anthropic-compatible APIs, but it requires a jailbreak, carries warranty and account-ban risks, and the 16GB unified memory still limits model size and quality.
Wccftech · 🔥 0 - Liquid AI releases LFM2.5-24B-A2B, a 24B open-weight reasoning model for on-device agents
Liquid AI released LFM2.5-24B-A2B on Oct. 13, an open-weight reasoning model with 24B total parameters and roughly 2B active per token (Liquid AI elsewhere cites 2.45B), aimed at running agents locally on laptops and PCs. It is the largest and final member of the LFM2.5 family, with the context window extended from 32K to 131K tokens, a vocabulary doubled from 65K to 128K for better non-Latin tokenization, pretraining scaled from about 17T to 33T tokens, and a switch to reasoning-only output with an explicit chain of thought. Liquid AI says it matches the larger Qwen3.6-35B-A3B on five of eight agentic and tool-use benchmarks (for example 48.09 versus 38.19 on LiveClaw), though those benchmarks and the speed figures (184 tokens/s on an Apple M5 Max) are vendor-run. Base and post-trained weights are on Hugging Face with day-one support in llama.cpp, MLX, vLLM, SGLang and ONNX.
Liquid AI Blog · 🔥 0 - Janas streams MoE experts from SSD, runs 80B models on a 32GB laptop
Janas is an experimental, GPL-3 open-source inference engine by Maurizio Cammalleri that streams mixture-of-experts weights from an NVMe SSD so models larger than RAM can run on ordinary hardware. Its own janas-bench numbers on a Debian 13 laptop (Intel Core Ultra 9 185H, integrated Arc GPU, 32 GB RAM) put the 48.4 GB Qwen3-Next-80B-A3B at 62 tok/s reading a 1,024-token prompt and 24 tok/s writing, against 16 and 2.8 tok/s for llama.cpp on the same files. Janas reports faster prompt reading and generation than llama.cpp on most of the compared models, with one exception: a 3-bit Qwen3-30B-A3B reads prompts slower than llama.cpp.
Hacker News · AI（100+ 分） · 🔥 0 - AWS publishes guide to deploying decision models with llama.cpp on SageMaker AI
The AWS Machine Learning Blog published a tutorial on deploying decision models with the AWS Deep Learning Container for llama.cpp on Amazon SageMaker AI, serving quantized models on CPU or GPU infrastructure. The standard path exposes OpenAI-compatible chat completions through SageMaker AI InvokeEndpoint, while applications needing scores over a fixed set of choices can build a child image via AWS CodeBuild that swaps the default server for a SemIf (formerly OpenJev) adapter returning probabilities over declared options without decoding an answer token. The post includes request and response schemas, CPU-versus-GPU guidance, and advises applying application-level thresholds and fallbacks rather than treating a model decision as authorization.
AWS Machine Learning Blog · 🔥 0 - Hugging Face Transformers tops GitHub Trending
Hugging Face's Transformers library appeared on GitHub Trending's daily list. The project positions itself as the model-definition framework for state-of-the-art text, vision, audio, video and multimodal models, for both inference and training, and its documentation says there are over 1M Transformers model checkpoints on the Hugging Face Hub. It says the library interoperates with training frameworks such as Axolotl, Unsloth, DeepSpeed, FSDP and PyTorch-Lightning, and with inference engines and modeling libraries including vLLM, SGLang, TGI, llama.cpp and mlx, and requires Python 3.10+ and PyTorch 2.6+.
GitHub Trending（每日） · 🔥 0 - JetBrains releases Mellum2.1 coding model
JetBrains launched Mellum2.1, keeping Mellum2's 12B mixture-of-experts architecture with 2.5B active parameters and the Apache 2.0 license, and focusing the upgrade on agentic coding. JetBrains says reinforcement learning was expanded from a short post-training step into the main training stage, letting the model explore codebases, edit files, identify root causes of failing tests and draft and verify fixes. The model is available on Hugging Face, with GGUF builds for llama.cpp, Ollama and LM Studio and a vLLM MTP speculative-decoding component promised later.
IT之家 AI · 🔥 0 - Google releases EmbeddingGemma 2 for on-device multimodal retrieval
Google released EmbeddingGemma 2, a 740M-parameter model licensed under Apache 2.0 that maps text, code, images, video, and audio into a shared 768-dimensional embedding space. Based on Gemma 4, it has a modular design with a 270M-parameter text component and optional 170M-parameter vision and 300M-parameter audio encoders. It supports an 8K-token context window and Matryoshka Representation Learning, allowing embeddings to be truncated to reduce storage. The reports say models are available through Hugging Face and LiteRT; arXiv metadata lists the accompanying paper separately.
Google Developers Blog · 🔥 3
Experience and discussion from the community
Share my Llama.cpp experienceAsk about Llama.cpp
Nobody has shared their experience with Llama.cpp yet.