TorchTitan

Headlines · 1

1. 19h
   PyTorch's Zephon brings elastically deterministic data loading to TorchTitan

   A PyTorch blog post introduces Zephon, a data loader for TorchTitan that aims to produce identical global batches even when the number of GPUs changes. In experiments training a 1B-parameter model from scratch on 20B tokens across 8 to 64 GPUs, a standard loader let DCLM Core v1 scores vary by up to 0.82 points and FineWeb scores by 0.64 points — roughly four times the thresholds those benchmarks' authors used — while Zephon's spread was 0.05 and 0.011 points respectively. Zephon tokenizes, packs, shuffles and mixes data on the fly inside the training job, using lanes, single-threaded accumulators and periodic flush sentinels to reproduce state, with checkpoints of a few MiB that can be resumed on a different number of data-parallel ranks; the authors say it keeps pace with Google Grain on a vision-language pipeline.

   PyTorch Blog · 🔥 6

Experience and discussion from the community

Share my TorchTitan experienceAsk about TorchTitan