GH200
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 2
- Hugging Face releases a practical MoE training handbook for Megatron-LM
Hugging Face authors released a practical handbook for training Mixture-of-Experts models with Megatron-LM, along with 55 figures, scripts, and a consolidated block of recommended flags. It covers hardware testing, image building, dense-baseline training, a one-node MoE comparison, routing and load balancing, serving, and reinforcement learning. The authors say its lessons draw on about 800,000 GH200 GPU-hours of MoE training on CSCS's Alps cluster; this figure was not independently verified in the supplied reports.
r/LocalLLaMA · 🔥 0 - Apex-3 from-scratch MoE checkpoint released with training observations
A Reddit author reported training Apex-3, a 20.99B-parameter mixture-of-experts model with 3.36B active parameters per token, from scratch on 70B tokens without external pretrained weights. The model uses 64 routed experts, one shared expert, SuperBPE tokenization, Muon and DiLoCo, while its logs showed a clear in-context-learning phase transition around 1.3B to 1.5B tokens. A stable checkpoint and training materials were published for continued work, but the token-efficiency estimate and capability interpretations are the author's claims.
r/LocalLLaMA · 🔥 0
Experience and discussion from the community
Share my GH200 experienceAsk about GH200
Nobody has shared their experience with GH200 yet.