Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

torchtitan

torchtitan: A PyTorch-native platform for pretraining and post-training generative AI models at scale.

What problem it solves

Training generative AI models at scale requires coordinating multiple parallelism strategies while supporting pretraining, fine-tuning, and reinforcement learning in one codebase. Large-scale training also needs reproducible configurations, checkpointing, debugging tools, and observability that can be adapted for research and model development.

Key features

  • Composable model parallelism — Models can stack FSDP2, tensor, pipeline, context, and expert parallelism into N-D layouts through DeviceMesh and DTensor without rewriting the model.

    Source: README

  • Research-oriented Python recipes — Each run is represented by a readable Python recipe that returns a complete configuration and can be copied or modified.

    Source: README, torchtitan_recipes

  • Pretraining through reinforcement learning — The same model definitions and kernels can be used across pretraining, supervised fine-tuning, TitanRL training, and vLLM generation.

    Source: README

  • Checkpointing and observability — Distributed checkpointing, direct Hugging Face safetensors support, metrics, profiling, memory snapshots, deterministic mode, and loss comparison tools support training workflows.

    Source: README

Quick start

Copied verbatim from the project's README.

git clone https://github.com/pytorch/torchtitan && cd torchtitan
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu132
pip install -r requirements.txt
NGPU=1 ./run_train.sh   # Llama 3 debug model, 10 steps
MODULE=my_recipes CONFIG=llama3_8b_tp2 ./run_train.sh

Who it may suit

Inference: may suit PyTorch users researching scalable generative model training, building reproducible distributed training recipes, or extending pretraining and reinforcement-learning workflows.

Topics to discuss

  • Which parallelism combination should be the basis for the next shared recipe?

  • How should recipe-level configuration and component overrides be designed for easier experimentation?

  • What debugging and reproducibility workflows should receive priority for large-scale distributed training?


Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.

Read the original README https://github.com/pytorch/torchtitan#readme

ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.

github.comhttps://github.com/pytorch/torchtitan

Comments

…