Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

TorchTitan trains a roughly 1T-parameter MoE on 384 GH200 GPUs

AI summary

A report describes training a roughly 1T-parameter mixture-of-experts model with TorchTitan on 384 NVIDIA GH200 GPUs in the JUPITER supercomputer, reaching about 1,300 tokens per second per GPU. The setup used 384-way FSDP and four-way expert parallelism. Reported optimizations included running Newton-Schulz on the GPU, which raised throughput by 35%; balancing attention cost, which added 13%; group-limited routing, which added 7%; FP8 expert all-gather, which added 18%; and moving allocations to the compute stream, which added 21%. The authors said the fixes went upstream to PyTorch and TorchTitan, and that the model and training recipes would be released openly. The long-term quality effect of group-limited routing and the impact of FP8 expert GEMMs remained unresolved.

In-depth insight

The report shows that a shared, publicly funded Hopper-generation system with ordinary user access can host frontier-scale MoE training, subject to inter-node bandwidth and host-memory limits. It also documents performance fixes made available upstream to PyTorch and TorchTitan.

Hi everyone!

Over the last few months we've been in the trenches with the TorchTitan and PyTorch maintainers, trying to find out how far GH200 (the GPUs inside the JUPITER supercomputer) can be pushed. Short answer: far enough to train a ~1T-parameter MoE at ~1,300 tokens/s per GPU on 384 GPUs. That's 14X where we started, and our starting point was, let's say, humbling.

Along the way we collected a handful of lessons. If you've ever trained a big model, or just stared at a CUDA out of memory, some of this will feel familiar.

Here's the tour. If you want the full thing with all the plots, it's on the Domyn blog, and there's a condensed version on the PyTorch blog if you're short on coffee.

Hope you like it!

The setup: FSDP + EP, and why I'm not a PP fan

A 1T MoE doesn't fit on one GPU (shocking, I know), so the first question is how to shard it. This choice ends up dictating basically every optimization that follows, so it pays to think about it. The usual suspects are:

#TorchTitan#PyTorch#NVIDIA GH200#JUPITER supercomputer#FlexShard#FSDP#Muon#JSC

Original · 7 min readRead the original
Group chat

No comments yet. Start the conversation.