torchtitan
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 7
- LLM Pretraining Engineering: Frameworks, Parallelism, and Scaling Laws
The supplied article surveys large-model pretraining, covering data, tokenization, optimizers, memory, distributed parallelism, checkpoints, and model evaluation. It compares Megatron-LM, DeepSpeed, and torchtitan; explains the different configuration guidance in Kaplan’s and Chinchilla’s scaling laws; and discusses cost estimation, loss spikes, checkpoint recovery, and resuming distributed training. Framework and performance characterizations are the article’s assessments, while its cost calculation is presented without accompanying source calculations.
掘金 · 人工智能 · 🔥 0 - Trainium3 Advances Native PyTorch Training, but CUDA Gap Remains
SemiAnalysis reports that Anthropic is running production inference and reinforcement-learning rollouts on AWS Trainium2 and Trainium3, while AWS Neuron is moving Trainium toward native PyTorch training with eager mode, torch.compile and standard APIs. Its engineers found the Trainium3 direction promising after testing the TorchTitan and TorchNeuron stack, but encountered unspecified bugs, long compilation times and a platform that remains far less mature than NVIDIA's CUDA ecosystem. AWS says upcoming TorchNeuron releases will use precompiled NKI kernels and a redesigned compile stack to target faster time to first step, though that target has not yet been demonstrated.
SemiAnalysis · 🔥 0 - PyTorch Conference announces native backends for Google TPU and AWS Trainium
According to the PyTorch Blog, PyTorch Conference North America 2026 drew more than 3,500 attendees to San Jose on October 20–21 for discussions on open intelligence, research, models, infrastructure, hardware and production. The event introduced TorchTPU, an open-source Google and Meta collaboration providing a native PyTorch backend for Google TPUs, and launched an Apache 2.0 public beta of TorchNeuron, a native backend for AWS Trainium. Other updates covered compiler development for IBM Spyre and an optional backend for AMD MI350-series GPUs. TorchTitan also expanded model support across its 5D parallelism stack and added reinforcement learning through TitanRL.
PyTorch Blog · 🔥 0 - AWS walkthrough: training Qwen3 on Trainium with torchtitan
An AWS Machine Learning Blog post walks through training Qwen3 on a single trn2.48xlarge Trainium instance using torchtitan, saying TorchNeuron registers Trainium as a native PyTorch device so torchtitan can use it through standard PyTorch interfaces. The post describes three configurations that share the same torchrun launch pattern and config structure: Hugging Face Transformers v5 as a modeling backend running Hub models through AutoModelForCausalLM with published safetensors, torchtitan's native Qwen3 selected by changing config.model_spec, and an NKI mixture-of-experts path using kernels from the torchtitan-neuron recipes attached via torchtitan's override registry. AWS says the setups support both pre-training from random initialization and fine-tuning published weights, and pins versions including torch 2.14.0, torch_neuronx 2.14.3, transformers 5.16.0 and torchtitan-neuron release-2.33.
AWS Machine Learning Blog · 🔥 0 - TorchTitan Adds New Model Support, TitanRL and Scaling Improvements
PyTorch reports that TorchTitan, a PyTorch-native platform for generative AI pretraining and post-training, now supports Muse Spark Open, Muse Glimmer, Qwen3-VL, Qwen3.5, Kimi K2.7, Kimi K3 and DeepSeek-V4 through full training-stack integrations. The update adds the TitanRL reinforcement-learning stack, asynchronous rollouts, FlexShard and Dist-MoE integrations, and distributed checkpointing improvements. PyTorch also reports internal benchmarks for these features, but the announcement is a single vendor source and does not provide an independent comparison with other training frameworks.
PyTorch Blog · 🔥 0 - Google open-sources TorchTPU, giving PyTorch a native TPU backend
Google announced Tuesday that TorchTPU, its version of PyTorch for TPUs, is now available as an open-source project, adding a native torch.device("tpu") backend so existing PyTorch training scripts need only minimal device-initialization changes. Google and Meta co-developed TPU-native execution passes inside TorchTitan, and a TorchTitan TPU fork is published on GitHub; Google says it worked with Hugging Face, Inferact (vLLM) and RadixArk (SGLang) on tool compatibility. Google reports vendor benchmarks including about 57% MFU for Llama 3 405B from 128 to 1,024 TPU v7x chips and 32.42% MFU for Qwen3 235B on a single 1,024-chip slice, while Amazon released its own PyTorch variant, TorchNeuron, for Trainium and AWS a day earlier.
Google Developers Blog · 🔥 0 - PyTorch's Zephon brings elastically deterministic data loading to TorchTitan
A PyTorch blog post introduces Zephon, a data loader for TorchTitan that aims to produce identical global batches even when the number of GPUs changes. In experiments training a 1B-parameter model from scratch on 20B tokens across 8 to 64 GPUs, a standard loader let DCLM Core v1 scores vary by up to 0.82 points and FineWeb scores by 0.64 points — roughly four times the thresholds those benchmarks' authors used — while Zephon's spread was 0.05 and 0.011 points respectively. Zephon tokenizes, packs, shuffles and mixes data on the fly inside the training job, using lanes, single-threaded accumulators and periodic flush sentinels to reproduce state, with checkpoints of a few MiB that can be resumed on a different number of data-parallel ranks; the authors say it keeps pace with Google Grain on a vision-language pipeline.
PyTorch Blog · 🔥 0
Experience and discussion from the community
Share my torchtitan experienceAsk about torchtitan
Nobody has shared their experience with torchtitan yet.