ggml-org

Headlines · 3

1. Oct 22
   llama.cpp XMX matrix engine refactor nearly doubles B70 speculative decoding speeds

   A Reddit user on r/LocalLLaMA reports that llama.cpp pull request #29864, which refactors Intel XMX matrix engines and is attributed to budiony, made a huge improvement to speculative decoding speeds on an Intel B70 GPU running SYCL. In before/after runs on Muse Glimmer Q6_K (30B) with DFlash2 across nine prompts, throughput rose from 38.0 to 73.0 tok/s on code_python and aggregate wall time fell from 46.86s to 26.68s. The poster notes the acceptance rate dropped from 0.7845 to 0.363 because cheaper token verification allowed n-max to be increased; the numbers are a single, unverified user report.

   r/LocalLLaMA · 🔥 0
2. Oct 21
   Flint MoE model support merged into llama.cpp, Reddit post says

   A post on r/LocalLLaMA says a pull request adding Flint (FlintForCausalLM) support to llama.cpp, PR #30928 in the ggml-org repository, was merged after about six hours. The post describes Flint as a decoder-only mixture-of-experts Transformer from developer Recursive with 39.8B total parameters and roughly 2.5B active parameters per token including the output head, spread over 49 layers — 2 dense followed by 47 MoE layers. It lists 256 routed experts with 6 selected per token plus an always-active shared expert, 35 sliding-window and 14 full-attention layers, a documented context configuration of 262,144 tokens, and a stated knowledge cutoff of June 2026; these details come from the post alone and have not been independently confirmed.

   r/LocalLLaMA · 🔥 0
3. Oct 15
   Llama for Windows released: one-click local model downloads

   Llama for Windows, the Windows counterpart to the macOS Llama app, is a free, open-source Windows 11 application built on Microsoft's WinUI 3 that lives in the system notification area and opens its control panel from a taskbar icon. It connects directly to the Hugging Face model hub, shows a curated model list with parameter counts, disk usage and vision support, and downloads GGUF-quantized models driven by the llama.cpp ecosystem, all without admin rights, manual compilation or complex launch parameters. The app offers an Alt+Space floating chat window, a built-in web UI for loading and unloading local models, and can start an OpenAI-compatible local server at localhost:9931 that scripts and AI agents can call, with multiple apps able to share one model file. According to the project, prompts and inference stay on the device and no telemetry, analytics events or installation identifiers are collected, though network access is still needed to fetch models and updates; it supports Windows 11 x64 and ARM64 plus NVIDIA's RTX Spark platform, and does not list Windows 10 as supported.

   cnBeta · 🔥 0

Experience and discussion from the community

Share my ggml-org experienceAsk about ggml-org