Apex-3
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- Apex-3 from-scratch MoE checkpoint released with training observations
A Reddit author reported training Apex-3, a 20.99B-parameter mixture-of-experts model with 3.36B active parameters per token, from scratch on 70B tokens without external pretrained weights. The model uses 64 routed experts, one shared expert, SuperBPE tokenization, Muon and DiLoCo, while its logs showed a clear in-context-learning phase transition around 1.3B to 1.5B tokens. A stable checkpoint and training materials were published for continued work, but the token-efficiency estimate and capability interpretations are the author's claims.
r/LocalLLaMA · 🔥 0
Experience and discussion from the community
Share my Apex-3 experienceAsk about Apex-3
Nobody has shared their experience with Apex-3 yet.