Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

FP8

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 3

  1. 4-bit GPT training speeds up on a single Nvidia B200

    VizuaraAI's author reports training GPT models end to end on one Nvidia B200 with an NVFP4 forward pass, an FP8 backward pass using stochastic rounding, and high-precision master weights. For the 840M-parameter model, 2,000 steps took about 4.3 minutes versus 5.5 minutes in BF16, while final validation losses were 1.691 and 1.688 respectively. Reported per-step speedups ranged from 18% for the 219M model to 38% for the 3.3B model, although the largest run covered only 1,000 steps.

    r/LocalLLaMA · 🔥 0
  2. Quantized LLM Serving Gains Depend on Batch Size

    A meta-analysis of public vLLM and SGLang measurements reports that 4-bit quantization delivers roughly 1.35–2.12× speedups at batch size 1, while W4A16 gains fall to about 1.01–1.02× in the cited batch-256 tests. One H20 measurement found FP8 retained a 1.68× gain at a long prompt, but the analysis rates the large-batch W4A16 finding low confidence and notes that FP8 was not measured in that comparison. It also finds that all reported HumanEval differences fall within estimated benchmark noise, while selected AWQ and GPTQ cases show GSM8K losses of about 4 points. The underlying timings come from heterogeneous engine versions, GPUs, metrics and workloads rather than a unified benchmark.

    掘金 · 人工智能 · 🔥 0
  3. Mistral Releases Mistral Large 4 Open Weights

    Mistral AI released the open weights for Mistral Large 4, also known as Le Chonk. It is a native multimodal, multilingual mixture-of-experts model with about 1 trillion total parameters, 49 billion active parameters and a 1-million-token context window. Official FP8 and NVFP4 variants are available on Hugging Face, alongside community-created GGUFs from Unsloth and the llama.cpp team. The model is offered under the Mistral Ecosystem Open-Weight License, which permits use, modification, fine-tuning and commercial use, while requiring a separate commercial license for model-as-a-service businesses. Mistral says the model leads open-weight coding and defensive-security benchmarks and offers three reasoning modes. Reported external results include 19.7% on Cognition's private FrontierCode 2.0 benchmark and 48% on Artificial Analysis's Coding Agent Index, matching GLM-5.3 while using 47% fewer tokens per task on average.

    Mistral AI News · 🔥 0

Experience and discussion from the community

Share my FP8 experienceAsk about FP8

FreeVideo: A local inference engine for generating MiniMax H3 videos on consumer GPUs.

What problem it solves FreeVideo enables local video generation on consumer hardware while coordinating VRAM, system memory, disk, weight placement, compute precision, and attention kernels. It provides a ComfyUI workflow and launchers that simplify setup across Windows, macOS, and Linux. Key features Hardware-adaptive execution — The engine selects GPU-specific weight formats and probes available…

github.comhttps://github.com/FlashML-org/FreeVideo

unsloth: A desktop app for running and training models locally.

What problem it solves Running and training models locally requires tools for model execution, fine-tuning, agent integration, and deployment across different model formats and hardware. The project brings these workflows into a desktop app, a web UI, and a code-based version. Key features Run and train models — Run and train language, diffusion, decision, embedding, audio, image, and video models…

github.comhttps://github.com/unslothai/unsloth

llama.cpp: Plain C/C++ implementation for local and cloud LLM and VLM inference.

What problem it solves Running large language and vision-language models can require substantial setup and may depend on the available hardware. The project aims to make inference accessible across a wide range of local and cloud hardware with minimal setup. Key features Minimal-setup inference — The project provides installation options for common platforms, including scripts, Docker, pre-built b…

github.comhttps://github.com/ggml-org/llama.cpp