Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

4-bit GPT training speeds up on a single Nvidia B200

AI summary

VizuaraAI's author reports training GPT models end to end on one Nvidia B200 with an NVFP4 forward pass, an FP8 backward pass using stochastic rounding, and high-precision master weights. For the 840M-parameter model, 2,000 steps took about 4.3 minutes versus 5.5 minutes in BF16, while final validation losses were 1.691 and 1.688 respectively. Reported per-step speedups ranged from 18% for the 219M model to 38% for the 3.3B model, although the largest run covered only 1,000 steps.

In-depth insight

The experiment provides a single-GPU test of end-to-end low-precision GPT training, showing faster steps and nearly identical validation loss to BF16 for small models trained on TinyStories.

Thinking Machines wrote last week that Inkling 2 Small was trained with a native 4-bit forward pass. I wanted to check this for myself. What I ran: Results: Two things to improve: Not tested: multiple GPUs, MoE dispatch and combine, Four Over Six, MXFP8 block scales in the backward pass. The width-8,192 run is only 1,000 steps and undertrained. Write-up with every figure: https://4bit.vizuara.ai Code: https://github.com/VizuaraAI/4bit-training-b200 3-hour walkthrough from first principles, if you want the theory: https://youtu.be/a3nV32Guxcc submitted by /u/OtherRaisin3426 [link] [comments]

#Nvidia B200#NVFP4#FP8#BF16#TinyStories#PyTorch 2.8#cuBLASLt#Modal#VizuaraAI

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.