Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

unsloth

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 50

  1. LookAtMyQuant 2 Compresses Qwen and Inkling Models

    LookAtMyQuant 2 is an open-source quantization compiler that measures how compression affects a model’s output and assigns codes and bit budgets to stored model components. Its recipe uses post-training quantization, trellis codes, Fisher information for per-row bit allocation, and distillation. The project released a 1.23 GB Qwen3.5-4B artifact and reports compressing Inkling-2-Mini from 18.89 GB of text weights to 5.38 GB and 8.87 GB. The method and MIT-licensed code are available through the project site. The project claims its Gemma 4 E2B GGUF has 12× lower KL divergence than Q4_K_M at the same file size, but the supplied report contains no independent corroboration.

    Hacker News 首页(100+ 分) · 🔥 26
  2. Thinking Machines Lab releases Inkling 2 Mini open-weight model

    Thinking Machines Lab released Inkling 2 Mini, a 26.4B-parameter open-weight multimodal MoE model designed for local agentic workloads. Its 24 decoder layers include 12 middle layers that run three times per token, while the checkpoint has 3.4B active parameters and a reported 1M-token context window. Artificial Analysis independently placed it at 20 on its Intelligence Index, while Thinking Machines Lab reported leading results among open-weight models below 100B total parameters.

    TestingCatalog · 🔥 55
  3. Learned predictors improve LLM fine-tuning duration estimates

    Dharma-AI R&D analyzed 272 production supervised fine-tuning jobs totaling about 7,382 GPU-hours on NVIDIA H200 clusters. In five-fold cross-validation, its CatBoost predictor achieved R² of 0.98 and a mean absolute error of 1.71 hours, compared with R² of 0.31 and 12.41 hours for the tested FLOP-based baseline. The study covers only SFT workloads on H200 infrastructure, and the full paper is planned for arXiv.

    Hugging Face · 🔥 0
  4. SupraLabs releases 65M-parameter Supra2.5 Base and Instruct models

    SupraLabs announced the Supra2.5-65M Base and Instruct model family, with the models available on Hugging Face. The company says it trained the model for four days on two RTX A6000 GPUs using 100 billion tokens of high-quality data, with F16, Q8_0, and Q4_K_M GGUF files available for llama.cpp, Ollama, and Unsloth. SupraLabs claims the 65M-parameter model outperforms its earlier 50M- and 100M-parameter models, but the supplied report includes no independent benchmark verification.

    r/LocalLLaMA · 🔥 0
  5. Reddit User Reports Qwen Unsloth Q8 Outperforms Q3 in Financial PDF Test

    A Reddit user described a self-hosted test of four models processing a large set of bank, credit-card, mortgage, brokerage, and retirement PDF statements. The user said Qwen FN Unsloth Q8 extracted all data accurately and produced a polished, insightful frontend, but ran for about eight hours. Qwen FN Q3 produced a nice frontend while extracting PDF data incorrectly and providing basic analytics. Orca nano1 finished in about two hours with mostly correct data but some clear errors and a basic frontend. Maya-S24 reportedly stalled in PDF parsing and validation, and the user stopped waiting after more than 10 hours.

    r/LocalLLaMA · 🔥 0
  6. Reddit user reports Strata update degrades Qwen3.8-Flash-Next output

    A Reddit user reports that updating Strata from version 0.1.42.0 made Qwen3.8-Flash-Next faster to start but less capable, with longer runtimes and more failures on a detailed voxel-art prompt. The user says version 0.1.42.0 completed similar work in roughly 15–30 minutes, or 50–60 minutes with xhigh, while the latest version often ran for at least 90 minutes, exhausted the 262K context, or failed. The report is a single user account and does not establish that the regression is reproducible across setups.

    r/LocalLLaMA · 🔥 0
  7. Developer Reports Completing Game With Local Qwen 3.8 Flash Next

    A Reddit developer reports completing all code for the free-to-play game Hyper Clashers with a locally run, Unsloth-quantized Qwen 3.8 Flash Next, without assistance from frontier models. The project used TypeScript and Three.js on a Windows machine with Ryzen AI 395+ hardware, a custom Windows fork of Gufo and the developer’s own harness. The developer recorded 1,911 prompts, 31,883 tool calls and 20,370,450 output tokens. They say the model handled coding effectively but required clear task scope, close human oversight, measurable tests, logging and debugging tools.

    r/LocalLLaMA · 🔥 0
  8. Author reports lower KL divergence for Hessian-refined Qwen3.8-27B GGUF

    A Reddit user released a Hessian-refined Qwen3.8-27B UD_4_XL GGUF that applies the first two MACaQ refinement techniques without requiring inference-software patches. On a small test set using one RTX 5090, the author reports 13.23% lower KL divergence than the Unsloth version, equal inference speed, and a 4% LiveCodeBench improvement that the author describes as a single, unreliable data point. The third MACaQ technique was not used because it requires inference-software updates.

    r/LocalLLaMA · 🔥 0
  9. Developer Turns DiffusionGemma into an Image Generation Pipeline

    A community developer built a text-to-image and image-manipulation pipeline for DiffusionGemma using LoRA, a trained image output head, and a vision decoder. The author says the adapted model produced recognizable images and improved visual manipulation when its generated visual latents were fed back into the next forward pass, and released the adapter, decoder weights, and code. The project is a self-reported experiment heavily assisted by LLMs, and the author describes the resulting system as rough and largely impractical.

    r/LocalLLaMA · 🔥 0
  10. Five Coding Agents Compared on One Local Qwen Model

    Reddit user seti_at_home ran Claude Code, OpenCode, GitHub Copilot Desktop, Pi, and Codex once each with the same pixel-art Halloween prompt, using a locally served Qwen3.8 Flash Next model. The reported runs varied widely in elapsed time, request count, output tokens, prefill speed, and decode speed, while Pi encountered a memory stall and Codex was eventually aborted. A side-by-side video showed 30 seconds of each result, but the author did not score visual quality or conduct a controlled benchmark with identical token streams.

    r/LocalLLaMA · 🔥 0
  11. Huawei releases open-source MACaQ quantization method

    Huawei released MACaQ, an open-source quantization method that supports text, image, video and audio models across llama.cpp, vLLM, Diffusers, PyTorch and Ascend backends. The project says MACaQ can quantize a 122B-parameter model in 64 minutes with 32.5 GB of device memory, while its reported Qwen3 GGUF tests show lower mean KL divergence than Unsloth and Bartowski at similar file sizes. These performance claims come from the project and published results cited by a Reddit submitter, who said they had not independently tested the 122B-model result.

    Hacker News 首页(100+ 分) · 🔥 0
  12. Hugging Face introduces a decision-model interface

    Hugging Face presented decision models as an alternative to generating text for classification-like tasks, returning structured decisions and probabilities through a typed interface. The article demonstrates the Transformers decision pipeline with GEV-26B-Decide and explains Choice, Score, and Noul primitives, while also describing custom training and a forthcoming TRL integration whose pull request remained open at publication.

    Hugging Face · 🔥 0
  13. Local Qwen model reportedly inserted unexpected URL tool calls

    A Reddit user says a local Qwen model inserted two URL-fetch tool calls while generating a GitHub release-checker script, alongside web searches. The setup was identified as unsloth/Qwen3.8-Flash-Next-GGUF:IQ4_XS running through plain llama.cpp, and the user shared an exported Open WebUI chat as evidence.

    r/LocalLLaMA · 🔥 0
  14. LocalLLaMA user asks how to restrict model web searches

    A LocalLLaMA user asked how local-model applications can limit liability from unrestricted web searches performed through a user's internet connection. They suggested network tools such as OpenSnitch and uMatrix as possible mitigations and sought local apps that can whitelist domains or URLs, while saying Unsloth did not appear to support such controls.

    r/LocalLLaMA · 🔥 0
  15. Reddit user tests Unsloth Qwen Flash-Next quants at 12 GB VRAM

    A r/LocalLLaMA post says Unsloth's new Dynamic v3.1 quantizations for Qwen Flash-Next and Qwen-27B, alongside ISTA-DASLab's advanced quantizations, make the models more accessible while keeping quality high. The author reports testing Unsloth's Iq2_XS quantization for Qwen Flash-Next with the Strata engine, running on 12 GB VRAM and 32 GB RAM at 1K prefill and about 50 tokens per second average decode across a 131K context window, with minimal quality loss under Strata's default auto-tuner. The post also says configurations targeting 8 GB VRAM with 32 GB system RAM are now in view, and that Qwen 4 was expected the same week.

    r/LocalLLaMA · 🔥 0
  16. Developer releases Saber Rust inference engine for AMD R9700

    A developer released Saber, an LLM inference engine written in Rust and targeting AMD R9700 GPUs. It supports tensor parallelism settings of 1, 2 or 4, as well as Qwen3.8 27B. The developer reported 6,645 tokens per second for prefill and a 316 tokens per second decode peak on Qwen3.8 27B, using two R9700 GPUs, an AMD Ryzen 9900X and a Taichi X870E. The developer also claims Saber beats existing available engines on Qwen3.8 27B and Flash Next.

    r/LocalLLaMA · 🔥 0
  17. Delta Inference Engine v0.93 expands local DeepSeek inference support

    The developer released Delta Inference Engine v0.93 with support for 64, 96 and 128 GB of RAM, Unsloth quantization formats, EXL3 and basic multi-GPU configurations. On one Windows system with an RTX 3090, the reported Unsloth UD-IQ2_XXS configuration reached 41 tokens per second with 192 GB RAM and scored 57 out of 99 on the developer’s benchmark, compared with 52 for the previous default quantization and 59 for the full model via API. The update also includes public CI builds, faster prompt handling and fixes for long contexts and large model files.

    r/LocalLLaMA · 🔥 0
  18. Unsloth releases Dynamic v3.1 GGUFs for Qwen3.8-Flash-Next

    Unsloth released Dynamic v3.1 GGUF quantizations for Qwen3.8-Flash-Next, including BF16, Q8_0, Q5_1, and IQ4_NL, along with a new engram-switching process. The vendor claims 10–15% higher accuracy than GSQ-RCO on DeepSWE, Terminal Bench, Aider Polyglot, and other benchmarks, using 256K context for agentic tests and 128K for the rest. Unsloth says it ran 26.2 billion benchmark tokens and repeated several tests multiple times, while acknowledging benchmark noise.

    r/LocalLLaMA · 🔥 0
  19. Qwen3.8 Flash Next Quant Report Nears BF16 Performance

    A Reddit user reported that Unsloth’s benchmark chart placed the IQ_3S GSQ RCO quantization of Qwen3.8 Flash Next at 99.0% of the BF16 baseline across 12 benchmarks. The report also cited Unsloth’s claim that its UD v3.1 quantization is about 15% better than GSQ RCO at roughly the same file size, although that comparison is a vendor claim and the underlying chart is not independently verified here.

    r/LocalLLaMA · 🔥 0
  20. Basalt 0.2.0 adds a new NVFP4-DQ8 build and faster Qwen 3.8 Flash Next inference

    The Basalt developer posted on r/LocalLLaMA that version 0.2.0 of the Blackwell-only inference engine for Qwen 3.8 Flash Next adds a new NVFP4-DQ8 quantization build on Hugging Face, with KLD of 0.0561 versus 0.0624 for the default NVFP4-DQ6_K build at roughly 7% slower prose throughput. Quality numbers are measured against Unsloth's BF16 GGUF served through llama.cpp (reference PPL 3.2749) and scored with basalt-bench --quality, with Basalt Q8 at 0.0200 KLD and 95.34% same top-1. On a rig with an RTX 5090 and RTX 5060 Ti 16GB plus 128GB DDR5 RAM, the developer reports about 603.7 to 594.2 tokens per second for structured output, 295.0 to 299.9 tokens per second for prose, and roughly 10,221 to 10,229 tokens per second for 64K prefill. The .basalt file format is now version 2, and older files can be migrated rather than redownloaded.

    r/LocalLLaMA · 🔥 0
  21. Reddit user reports Strata raises Qwen decode speed to 150 tok/s

    A Reddit user reported that switching from llama-server to Strata raised Qwen 3.8 Flash Next decode speed from 70–80 tok/s to 150 tok/s. The user listed the setup as “1 pro 6000,” two RTX 3090s, 256 GB DDR4 and a 160k context window. The same user reported 70 tok/s decode for DeepSeek V4.1 Flash at 2.25-bit, with an Unsloth 6-bit quant, and said they did not know what implementation method Strata used.

    r/LocalLLaMA · 🔥 0
  22. Unsloth DeepSeek GGUF Benchmark Results Face Scrutiny

    Unsloth released new DeepSeek-V4.1-Flash-1017 GGUF builds and reported that its Dynamic V3.1 quantizations led across several benchmarks at 512K context. The 1-bit build scored 68.1 on DeepSWE and 72.2 on Terminal-Bench. A r/LocalLLaMA analysis questioned why all four Unsloth quantizations and the Q8 baseline improved by roughly 2–3 percentage points on LiveCodeBench v6. Unsloth said the earlier chart represented one run, followed by additional evaluations under unchanged settings. The analysis depends on statistical assumptions and has not independently reproduced Unsloth's results; the reports did not provide the underlying raw data.

    UnslothAI (X) · 🔥 0
  23. MiniMax highlights local quantization work for M3.1 Flash

    MiniMax highlighted community efforts to run its 428B-parameter MiniMax M3.1 Flash model locally using lower-bit weights, specialized kernels, and expert placement across GPUs and system RAM. The projects include Unsloth's 143.9 GB 2-bit Dynamic GGUF weights, ViC305's SAGE-EXL3 recipe for one DGX Spark, PeasantSmith's Project Maya quantizations, and additional work from MiaAI Lab, oMLX, and others. A LocalLLaMA review noted that the reported sizes, fidelity scores, and speed figures used separate evaluation sets and are not directly comparable.

    MiniMax_AI (X) · 🔥 0
  24. Infernix updates Qwen support and adds a beta WebUI launcher

    The Infernix developer announced a release optimized for recent Qwen-architecture models and NVIDIA RTX 5090 systems on Windows and Linux. It adds Qwen3.8-Flash-Next variants, calibrated automatic speculative decoding, a reproducible lossless mode, and a beta WebUI launcher for model downloads, configuration, statistics, logs, and test chats. The developer also claims performance advantages over Strata and Basalt, but no independent benchmarks were supplied.

    r/LocalLLaMA · 🔥 0
  25. Community benchmark finds near-linear Qwen scaling to eight streams

    A Reddit user reported spending about $40 on vast.ai instances to benchmark RAM-heavy inference servers with Qwen 3.8 Flash Next in Unsloth Q8. On systems with two AMD EPYC 7642 processors and 251GB or 504GB of DDR4 memory, the poster reported nearly linear per-stream scaling through eight concurrent streams at a 124k context window. Raw benchmark data was posted in the Reddit comments, and concurrency beyond eight streams was not tested.

    r/LocalLLaMA · 🔥 0
  26. Unsloth adds DeepSeek support in Strata v0.2.9

    Niko Veit announced that Unsloth now supports DeepSeek models. The post listed improvements on Strix, parallel request handling, NVIDIA DGX Spark and Intel B-series hardware, alongside a link to the Strata v0.2.9 release.

    UnslothAI (X) · 🔥 0
  27. ISTA disputes Unsloth's DeepSeek-V4.1-Flash-1017 benchmark comparison

    ISTA said its DeepSeek-V4.1-Flash-1017 benchmarks used a 128K context window, reasoning effort of 50 and an output budget of up to 128K tokens. It attributes the discrepancy with Unsloth's published comparisons to different constraints, saying Unsloth imposed a strict 32K reasoning budget, and denies using hidden hyperparameter adjustments to favor its models. ISTA said its code is available on GitHub, its papers are on arXiv and additional comparisons were underway. It invited independent reproduction using its stated settings.

    r/LocalLLaMA · 🔥 0
  28. Unsloth releases preview GGUFs for DeepSeek-V4.1-Flash-1017

    Unsloth released preview Dynamic v3.1 GGUF quantizations for DeepSeek-V4.1-Flash-1017 and said they can run on devices with 128GB of RAM or unified memory, or with 64GB of RAM using engram offloading. The vendor reported stronger results than ISTA's GSQ-RCO GGUFs on nearly all listed metrics across seven benchmarks, but said it could not reproduce ISTA's published AIME25 result. Unsloth averaged at least three runs per benchmark, with 16 runs for AIME25.

    r/LocalLLaMA · 🔥 0
  29. Mistral Releases Mistral Large 4 Open Weights

    Mistral AI released the open weights for Mistral Large 4, also known as Le Chonk. It is a native multimodal, multilingual mixture-of-experts model with about 1 trillion total parameters, 49 billion active parameters and a 1-million-token context window. Official FP8 and NVFP4 variants are available on Hugging Face, alongside community-created GGUFs from Unsloth and the llama.cpp team. The model is offered under the Mistral Ecosystem Open-Weight License, which permits use, modification, fine-tuning and commercial use, while requiring a separate commercial license for model-as-a-service businesses. Mistral says the model leads open-weight coding and defensive-security benchmarks and offers three reasoning modes. Reported external results include 19.7% on Cognition's private FrontierCode 2.0 benchmark and 48% on Artificial Analysis's Coding Agent Index, matching GLM-5.3 while using 47% fewer tokens per task on average.

    Mistral AI News · 🔥 0
  30. Unsloth Announces Local LLM Browser Control

    Unsloth announced that local LLMs can use its tools to control a browser and complete tasks autonomously. The company said Qwen3.8-27B completed browser-based chess and painting tasks without human intervention, and shared a GitHub repository and usage guide.

    UnslothAI (X) · 🔥 0

Next →

Experience and discussion from the community

Share my unsloth experienceAsk about unsloth

unsloth: A desktop app for running and training models locally.

What problem it solves Running and training models locally requires tools for model execution, fine-tuning, agent integration, and deployment across different model formats and hardware. The project brings these workflows into a desktop app, a web UI, and a code-based version. Key features Run and train models — Run and train language, diffusion, decision, embedding, audio, image, and video models…

github.comhttps://github.com/unslothai/unsloth