GGUF
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 10
- Reddit post shares Qwen3.8 27B GGUFs as “statistically lossless”
A Reddit post titled “Statistically Lossless Qwen3.8 27B GGUFs” shared a Hugging Face link for the files. The poster said they were reposting the link because the original analysis was behind a blog paywall. The supplied report does not include methodology, benchmark results, quantization details, or independent validation of the claim.
r/LocalLLaMA · 🔥 0 - Community post labels Qwen3.8 27B GGUFs statistically lossless
A community post labeled Qwen3.8 27B GGUFs as statistically lossless. The poster was unsure whether the EAR99 quantization was related to the IST-DASLab SLQ project and said that, after roughly 30,000 tokens, it felt the same as or better than FP8, while explicitly describing the test as unscientific.
r/LocalLLaMA · 🔥 0 - Nanbeige4.3-33B GGUF Quantizations Span 35.2GB to 13.7GB
A r/LocalLLaMA post presents quantized GGUF builds of Nanbeige4.3-33B ranging from approximately 35.2GB to 13.7GB. It identifies the 20.3GB AD-IQ4_XS build as the preferred option for 32GB hardware and reports a 92.0% top-1 result relative to BF16, though the benchmark methodology and result are not independently verified.
r/LocalLLaMA · 🔥 0 - llama.cpp PR proposes merging Qwen tensors during GGUF conversion
A pull request by am17an proposes merging Qwen tensors when converting models to GGUF in llama.cpp. The supplied Reddit post links to ggml-org/llama.cpp pull request #31028 but does not confirm that it was merged or provide implementation details.
r/LocalLLaMA · 🔥 0 - Developer Reports GGUF Q4 Inference in the Browser with WebGL2 and GLSL
A Reddit developer said an implementation was running GGUF Q4 models inside HTML by paging them into WebGL2 embedded compute buffers and using GLSL. The developer also described tokenizer and vocabulary processing as part of what they called an endless debugging challenge. The report presents the developer's account and includes an unresolved link marker.
r/LocalLLaMA · 🔥 0 - ThinkingCap cuts Gemma 4 reasoning tokens with minimal accuracy loss
Bottlecap AI released ThinkingCap-gemma-4-26B-A4B-it, a drop-in replacement designed to reduce the amount of thinking performed by Google's Gemma 4 26B-A4B-it. In its own evaluation across 12 benchmarks, the model used 38.9% fewer thinking tokens on average while average accuracy fell by 0.15 percentage points, although AA-LCR accuracy dropped by 4 points. Bottlecap also reports lower looping and published the model with GGUF builds.
Hacker News 首页（100+ 分） · 🔥 0 - Developer releases Qwen3-0.6B MoE conversion with GGUF build
Developer bayliner1980 released Qwen3-0.6B-MoE-A0.5B, a mixture-of-experts conversion of Qwen3-0.6B, together with a GGUF version. The developer adapted the process from Qwen 2.5's qwen2moe to Qwen3's qwen3moe and used a pseudo-shared expert to recover some quality lost because Qwen3 lacks a shared expert. The developer's evaluation found it worse than the dense model on every reported quality metric, with no GPU speed improvement at this size, while the converted model's code was broken and its thinking mode was not trained.
r/LocalLLaMA · 🔥 0 - XHToken releases Spark-X2.6-4B for on-device AI
XHToken introduced Spark-X2.6-4B, a compact general-purpose model intended for on-device deployment and built on the Spark-X2.5-4B architecture. The publisher says improved post-training strengthens coding, tool use, long-context, multilingual and reasoning capabilities, while three configurable thinking levels let users balance quality against latency and inference cost; its comparisons with similarly sized open-source models are not independently verified in the supplied report.
r/LocalLLaMA · 🔥 0 - Llama for Windows released: one-click local model downloads
Llama for Windows, the Windows counterpart to the macOS Llama app, is a free, open-source Windows 11 application built on Microsoft's WinUI 3 that lives in the system notification area and opens its control panel from a taskbar icon. It connects directly to the Hugging Face model hub, shows a curated model list with parameter counts, disk usage and vision support, and downloads GGUF-quantized models driven by the llama.cpp ecosystem, all without admin rights, manual compilation or complex launch parameters. The app offers an Alt+Space floating chat window, a built-in web UI for loading and unloading local models, and can start an OpenAI-compatible local server at localhost:9931 that scripts and AI agents can call, with multiple apps able to share one model file. According to the project, prompts and inference stay on the device and no telemetry, analytics events or installation identifiers are collected, though network access is still needed to fetch models and updates; it supports Windows 11 x64 and ARM64 plus NVIDIA's RTX Spark platform, and does not list Windows 10 as supported.
cnBeta · 🔥 0 - AWS publishes guide to deploying decision models with llama.cpp on SageMaker AI
The AWS Machine Learning Blog published a tutorial on deploying decision models with the AWS Deep Learning Container for llama.cpp on Amazon SageMaker AI, serving quantized models on CPU or GPU infrastructure. The standard path exposes OpenAI-compatible chat completions through SageMaker AI InvokeEndpoint, while applications needing scores over a fixed set of choices can build a child image via AWS CodeBuild that swaps the default server for a SemIf (formerly OpenJev) adapter returning probabilities over declared options without decoding an answer token. The post includes request and response schemas, CPU-versus-GPU guidance, and advises applying application-level thresholds and fallbacks rather than treating a model decision as authorization.
AWS Machine Learning Blog · 🔥 0
Experience and discussion from the community
Share my GGUF experienceAsk about GGUF
Nobody has shared their experience with GGUF yet.