Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

GH200

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 2

  1. Hugging Face releases a practical MoE training handbook for Megatron-LM

    Hugging Face authors released a practical handbook for training Mixture-of-Experts models with Megatron-LM, along with 55 figures, scripts, and a consolidated block of recommended flags. It covers hardware testing, image building, dense-baseline training, a one-node MoE comparison, routing and load balancing, serving, and reinforcement learning. The authors say its lessons draw on about 800,000 GH200 GPU-hours of MoE training on CSCS's Alps cluster; this figure was not independently verified in the supplied reports.

    r/LocalLLaMA · 🔥 0
  2. Apex-3 from-scratch MoE checkpoint released with training observations

    A Reddit author reported training Apex-3, a 20.99B-parameter mixture-of-experts model with 3.36B active parameters per token, from scratch on 70B tokens without external pretrained weights. The model uses 64 routed experts, one shared expert, SuperBPE tokenization, Muon and DiLoCo, while its logs showed a clear in-context-learning phase transition around 1.3B to 1.5B tokens. A stable checkpoint and training materials were published for continued work, but the token-efficiency estimate and capability interpretations are the author's claims.

    r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my GH200 experienceAsk about GH200

Nobody has shared their experience with GH200 yet.