Apex-3从零训练MoE开放检查点
一名Reddit作者称,其从零训练了Apex-3,一款总参数20.99B、每Token激活3.36B参数的混合专家模型,使用70B Token且未采用外部预训练权重。该模型包含64个路由专家、1个共享专家,采用SuperBPE、Muon和DiLoCo;训练日志显示,上下文学习在约1.3B至1.5B Token处出现明显相变。作者已发布稳定检查点及训练材料,但Token效率估算和能力解读仍属其本人陈述。
公开的检查点和训练日志为研究从零训练稀疏语言模型的能力变化提供了可查验材料,但相关结论尚未得到独立验证。
Due to budget constraints, training only went up to 70B tokens. With SuperBPE tokenization providing roughly +27% token efficiency, this is effectively equivalent to training on around 100B tokens. A stable checkpoint was also released so anyone can continue training from it.
https://huggingface.co/YOON1v/Apex-3-59B-stable
I trained Apex-3, a 20.99B-parameter MoE (3.36B active per token) from scratch on 70B tokens. No external pretrained weights. It uses 64 routed experts plus 1 shared expert (top-4), a SuperBPE tokenizer, and was trained with Muon and DiLoCo on GH200s.
Model card: https://huggingface.co/YOON1v/Apex-3 Code and training notes: https://github.com/DW-dev-UE/LLM-from-scratch
This post covers what showed up in the training logs, not the final SFT scores. Evidence comes from the A, B, and C runs in run_logs/apex3_20261026/ (icl, arith, val, and moe logs) plus the 70B base model benchmarks. Numbers below are from run A. B and C were nearly identical. Final arithmetic full accuracy was 0.665 for A, 0.655 for B, and 0.655 for C.
- In-context learning (induction head) — the only clear phase transition