Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Apex-3

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 1

  1. Apex-3 from-scratch MoE checkpoint released with training observations

    A Reddit author reported training Apex-3, a 20.99B-parameter mixture-of-experts model with 3.36B active parameters per token, from scratch on 70B tokens without external pretrained weights. The model uses 64 routed experts, one shared expert, SuperBPE tokenization, Muon and DiLoCo, while its logs showed a clear in-context-learning phase transition around 1.3B to 1.5B tokens. A stable checkpoint and training materials were published for continued work, but the token-efficiency estimate and capability interpretations are the author's claims.

    r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my Apex-3 experienceAsk about Apex-3

Nobody has shared their experience with Apex-3 yet.