TorchTitan在384块GH200上训练约万亿参数MoE
一篇报告称,研究团队使用TorchTitan在JUPITER超级计算机的384块NVIDIA GH200上训练了一个约万亿参数的混合专家模型,吞吐量达到每GPU每秒约1300个token。训练采用384路FSDP和4路专家并行。报告中的优化包括在GPU上运行Newton-Schulz,使吞吐量提高35%;均衡注意力成本,增幅13%;采用组限制路由,增幅7%;采用FP8专家全收集,增幅18%;以及将相关分配移至计算流,增幅21%。作者表示,相关修复已提交至PyTorch和TorchTitan上游,模型及训练配方将开放发布。组限制路由的长期质量影响以及FP8专家GEMM的实际影响仍未确定。
报告显示,在考虑节点间带宽和主机内存限制的情况下,一台共享、公立资助且提供普通用户访问权限的Hopper代系统能够承载前沿规模的MoE训练。报告还记录了已向PyTorch和TorchTitan上游提供的性能修复。
Hi everyone!
Over the last few months we've been in the trenches with the TorchTitan and PyTorch maintainers, trying to find out how far GH200 (the GPUs inside the JUPITER supercomputer) can be pushed. Short answer: far enough to train a ~1T-parameter MoE at ~1,300 tokens/s per GPU on 384 GPUs. That's 14X where we started, and our starting point was, let's say, humbling.
Along the way we collected a handful of lessons. If you've ever trained a big model, or just stared at a CUDA out of memory, some of this will feel familiar.
Here's the tour. If you want the full thing with all the plots, it's on the Domyn blog, and there's a condensed version on the PyTorch blog if you're short on coffee.
Hope you like it!
The setup: FSDP + EP, and why I'm not a PP fan
A 1T MoE doesn't fit on one GPU (shocking, I know), so the first question is how to shard it. This choice ends up dictating basically every optimization that follows, so it pays to think about it. The usual suspects are:
#TorchTitan#PyTorch#NVIDIA GH200#JUPITER supercomputer#FlexShard#FSDP#Muon#JSC