Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

DiLoCo

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 2

  1. Streaming DiLoCo Brings WAN-Based LLM Training to MaxText

    Google Developers Blog says MaxText v0.2.5 introduces Streaming DiLoCo, which lets independently located learner groups train across datacenters while reducing cross-slice communication through periodic global updates and fine-grained parameter streaming. In tests using Qwen3-32B on two TPU v5p-32 slices, Google reported about 64–65 times less cross-slice traffic than data parallelism; the worked example says the 10 Gbit/s link required roughly 1.3 Gbit/s for Streaming DiLoCo. The feature is available through MaxText v0.2.5 on PyPI, its Docker images and the GitHub main branch.

    Google Developers Blog · 🔥 0
  2. Apex-3 from-scratch MoE checkpoint released with training observations

    A Reddit author reported training Apex-3, a 20.99B-parameter mixture-of-experts model with 3.36B active parameters per token, from scratch on 70B tokens without external pretrained weights. The model uses 64 routed experts, one shared expert, SuperBPE tokenization, Muon and DiLoCo, while its logs showed a clear in-context-learning phase transition around 1.3B to 1.5B tokens. A stable checkpoint and training materials were published for continued work, but the token-efficiency estimate and capability interpretations are the author's claims.

    r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my DiLoCo experienceAsk about DiLoCo

Nobody has shared their experience with DiLoCo yet.