4-bit GPT training speeds up on a single Nvidia B200
VizuaraAI's author reports training GPT models end to end on one Nvidia B200 with an NVFP4 forward pass, an FP8 backward pass using stochastic rounding, and high-precision master weights. For the 840M-parameter model, 2,000 steps took about 4.3 minutes versus 5.5 minutes in BF16, while final validation losses were 1.691 and 1.688 respectively. Reported per-step speedups ranged from 18% for the 219M model to 38% for the 3.3B model, although the largest run covered only 1,000 steps.
The experiment provides a single-GPU test of end-to-end low-precision GPT training, showing faster steps and nearly identical validation loss to BF16 for small models trained on TinyStories.
| Thinking Machines wrote last week that Inkling 2 Small was trained with a native 4-bit forward pass. I wanted to check this for myself. What I ran: Results: Two things to improve: Not tested: multiple GPUs, MoE dispatch and combine, Four Over Six, MXFP8 block scales in the backward pass. The width-8,192 run is only 1,000 steps and undertrained. Write-up with every figure: https://4bit.vizuara.ai Code: https://github.com/VizuaraAI/4bit-training-b200 3-hour walkthrough from first principles, if you want the theory: https://youtu.be/a3nV32Guxcc submitted by /u/OtherRaisin3426 [link] [comments] |
#Nvidia B200#NVFP4#FP8#BF16#TinyStories#PyTorch 2.8#cuBLASLt#Modal#VizuaraAI