Apex-3 from-scratch MoE checkpoint released with training observations
A Reddit author reported training Apex-3, a 20.99B-parameter mixture-of-experts model with 3.36B active parameters per token, from scratch on 70B tokens without external pretrained weights. The model uses 64 routed experts, one shared expert, SuperBPE tokenization, Muon and DiLoCo, while its logs showed a clear in-context-learning phase transition around 1.3B to 1.5B tokens. A stable checkpoint and training materials were published for continued work, but the token-efficiency estimate and capability interpretations are the author's claims.
The released checkpoint and detailed logs provide artifacts for examining how a from-scratch sparse language model develops, although the reported findings have not been independently verified.
Due to budget constraints, training only went up to 70B tokens. With SuperBPE tokenization providing roughly +27% token efficiency, this is effectively equivalent to training on around 100B tokens. A stable checkpoint was also released so anyone can continue training from it.
https://huggingface.co/YOON1v/Apex-3-59B-stable
I trained Apex-3, a 20.99B-parameter MoE (3.36B active per token) from scratch on 70B tokens. No external pretrained weights. It uses 64 routed experts plus 1 shared expert (top-4), a SuperBPE tokenizer, and was trained with Muon and DiLoCo on GH200s.
Model card: https://huggingface.co/YOON1v/Apex-3 Code and training notes: https://github.com/DW-dev-UE/LLM-from-scratch
This post covers what showed up in the training logs, not the final SFT scores. Evidence comes from the A, B, and C runs in run_logs/apex3_20261026/ (icl, arith, val, and moe logs) plus the 70B base model benchmarks. Numbers below are from run A. B and C were nearly identical. Final arithmetic full accuracy was 0.665 for A, 0.655 for B, and 0.655 for C.
- In-context learning (induction head) — the only clear phase transition