Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

FSDP

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 1

  1. TorchTitan trains a roughly 1T-parameter MoE on 384 GH200 GPUs

    A report describes training a roughly 1T-parameter mixture-of-experts model with TorchTitan on 384 NVIDIA GH200 GPUs in the JUPITER supercomputer, reaching about 1,300 tokens per second per GPU. The setup used 384-way FSDP and four-way expert parallelism. Reported optimizations included running Newton-Schulz on the GPU, which raised throughput by 35%; balancing attention cost, which added 13%; group-limited routing, which added 7%; FP8 expert all-gather, which added 18%; and moving allocations to the compute stream, which added 21%. The authors said the fixes went upstream to PyTorch and TorchTitan, and that the model and training recipes would be released openly. The long-term quality effect of group-limited routing and the impact of FP8 expert GEMMs remained unresolved.

    r/MachineLearning · 🔥 0

Experience and discussion from the community

Share my FSDP experienceAsk about FSDP

torchtitan: A PyTorch-native platform for pretraining and post-training generative AI models at scale.

What problem it solves Training generative AI models at scale requires coordinating multiple parallelism strategies while supporting pretraining, fine-tuning, and reinforcement learning in one codebase. Large-scale training also needs reproducible configurations, checkpointing, debugging tools, and observability that can be adapted for research and model development. Key features Composable model …

github.comhttps://github.com/pytorch/torchtitan