torchtitan: A PyTorch-native platform for pretraining and post-training generative AI models at scale.
What problem it solves
Training generative AI models at scale requires coordinating multiple parallelism strategies while supporting pretraining, fine-tuning, and reinforcement learning in one codebase. Large-scale training also needs reproducible configurations, checkpointing, debugging tools, and observability that can be adapted for research and model development.
Key features
Composable model parallelism — Models can stack FSDP2, tensor, pipeline, context, and expert parallelism into N-D layouts through DeviceMesh and DTensor without rewriting the model.
Source: README
Research-oriented Python recipes — Each run is represented by a readable Python recipe that returns a complete configuration and can be copied or modified.
Source: README, torchtitan_recipes
Pretraining through reinforcement learning — The same model definitions and kernels can be used across pretraining, supervised fine-tuning, TitanRL training, and vLLM generation.
Source: README
Checkpointing and observability — Distributed checkpointing, direct Hugging Face safetensors support, metrics, profiling, memory snapshots, deterministic mode, and loss comparison tools support training workflows.
Source: README
Quick start
Copied verbatim from the project's README.
git clone https://github.com/pytorch/torchtitan && cd torchtitan
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu132
pip install -r requirements.txtNGPU=1 ./run_train.sh # Llama 3 debug model, 10 stepsMODULE=my_recipes CONFIG=llama3_8b_tp2 ./run_train.shWho it may suit
Inference: may suit PyTorch users researching scalable generative model training, building reproducible distributed training recipes, or extending pretraining and reinforcement-learning workflows.
Topics to discuss
Which parallelism combination should be the basis for the next shared recipe?
How should recipe-level configuration and component overrides be designed for easier experimentation?
What debugging and reproducibility workflows should receive priority for large-scale distributed training?
Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.
Read the original README https://github.com/pytorch/torchtitan#readme
ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.
Comments
…