TorchTitan Adds New Model Support, TitanRL and Scaling Improvements
PyTorch reports that TorchTitan, a PyTorch-native platform for generative AI pretraining and post-training, now supports Muse Spark Open, Muse Glimmer, Qwen3-VL, Qwen3.5, Kimi K2.7, Kimi K3 and DeepSeek-V4 through full training-stack integrations. The update adds the TitanRL reinforcement-learning stack, asynchronous rollouts, FlexShard and Dist-MoE integrations, and distributed checkpointing improvements. PyTorch also reports internal benchmarks for these features, but the announcement is a single vendor source and does not provide an independent comparison with other training frameworks.
The update expands TorchTitan’s reusable PyTorch-native toolkit with recent model architectures, reinforcement-learning support and large-scale training infrastructure. TitanRL is designed to keep the same model implementation across pretraining, post-training and vLLM, reducing the need to align separate training and inference codebases.
TorchTitan is a PyTorch-native platform for rapid experimentation and large-scale pre- and post-training of generative AI models. It is designed as a minimal, clean-room implementation of PyTorch-native techniques, with flexible extension points for users’ needs. This has enabled TorchTitan to find widespread adoption across research and production in academia and industry.
The TorchTitan team landed massive improvements in the past year, without losing its minimal, extensible ethos. Enhancements include support for SOTA models, dramatic improvements of out-the-box performance at scale, RL support and new communication features.
TorchTitan is for everyone – easy to use, providing SOTA performance and allowing users to start small and scale up seamlessly. We will present all these new improvements in the following sections, and we look forward to your feedback!
New model support
SOTA model support is critical for the success of training frameworks, and TorchTitan aims to always provide the latest advancements when it comes to open-weights architecture. TorchTitan added PyTorch-native support for Muse Spark Open, Muse Glimmer, Qwen3-VL and Qwen3.5, Kimi K2.7, Kimi K3, and DeepSeek-V4. These were full training-stack integrations with support across TorchTitan’s 5D parallelism stack. This work also introduced a composable configuration system built around typed Python recipes, model-config transforms, and targeted component overrides. Users can change parallelization strategies, model components, or hardware-specific kernels without maintaining separate model forks. The same interfaces integrate optimized kernels from Attention Gym, including Gated DeltaNet, Kimi Delta Attention, and sparse-attention kernels, as well as opt-in Helion kernels such as fused RoPE, while remaining compatible with torch.compile.
#TorchTitan#TitanRL#Muse Spark Open#Muse Glimmer#Qwen3-VL#Qwen3.5#Kimi K2.7#Kimi K3#DeepSeek-V4#PyTorch#vLLM#FlexShard