Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Muon

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 7

  1. CPU-Based Kimi K3 Reproduction Adds Muon P2P Optimizer

    A Juejin AI article presents RossiLH’s lightweight, CPU-based reproduction of Kimi K3 training components, including training loops, evaluation, checkpointing, and a Gradio demo. It also details a Muon optimizer that gathers parameter updates through peer-to-peer communication, applies Newton–Schulz orthogonalization, and scatters the results back; the project reports 28 local tests.

    掘金 · 人工智能 · 🔥 3
  2. CPU Demo Explores KIMI K3 Optimizer Components

    RossiLH published a CPU-only Gradio demo based on KIMI K3 optimizer code, with 15 tabs covering Newton-Schulz, Muon, parameter grouping, learning-rate scheduling, Hybrid, and P2P components. It includes seven visualization scripts and two end-to-end demos. In a small Transformer comparison, the author reports Muon loss lower than AdamW by 0.3% to 1.3%. The experiment used a two-layer model, 200 steps by default, and no hyperparameter sweep; the author cautions that the result may not generalize.

    掘金 · 人工智能 · 🔥 0
  3. Reddit user questions OKLS step efficiency versus Muon

    A Reddit post asks why OKLS, described as a zero-staleness implementation of KL-Shampoo, may be outperformed by Muon in step efficiency. The submitter claims that SOAP is more step-efficient than Shampoo, KL-Shampoo is more step-efficient than SOAP, and OKLS only surpasses Muon in parameter efficiency during very long training runs. The supplied post does not include benchmark measurements, implementation details, or responses that resolve these comparisons.

    r/MachineLearning · 🔥 0
  4. TorchTitan trains a roughly 1T-parameter MoE on 384 GH200 GPUs

    A report describes training a roughly 1T-parameter mixture-of-experts model with TorchTitan on 384 NVIDIA GH200 GPUs in the JUPITER supercomputer, reaching about 1,300 tokens per second per GPU. The setup used 384-way FSDP and four-way expert parallelism. Reported optimizations included running Newton-Schulz on the GPU, which raised throughput by 35%; balancing attention cost, which added 13%; group-limited routing, which added 7%; FP8 expert all-gather, which added 18%; and moving allocations to the compute stream, which added 21%. The authors said the fixes went upstream to PyTorch and TorchTitan, and that the model and training recipes would be released openly. The long-term quality effect of group-limited routing and the impact of FP8 expert GEMMs remained unresolved.

    r/MachineLearning · 🔥 0
  5. FOAM Halves SOAP Optimizer State and Matches Reported Baselines

    A paper by Jaemyung Yu, Byeongho Heo, Sangdoo Yun, and Dongyoon Han introduces FOAM, a lightweight SOAP variant designed to reduce optimizer-state memory and eigendecomposition overhead. FOAM uses input-only preconditioning and replaces SOAP’s dense rotated second moment with a factored Adam-moment estimator based on row-by-column statistics. The authors report that, in LLM pre-training experiments from 124M to 760M parameters, FOAM matches strong SOAP and Muon baselines in validation loss, uses half of SOAP’s optimizer state, and runs at Muon-class wall-clock time. The supplied report presents these results as the authors’ findings and does not independently verify them.

    Hugging Face · Papers · 🔥 0
  6. Apex-3 from-scratch MoE checkpoint released with training observations

    A Reddit author reported training Apex-3, a 20.99B-parameter mixture-of-experts model with 3.36B active parameters per token, from scratch on 70B tokens without external pretrained weights. The model uses 64 routed experts, one shared expert, SuperBPE tokenization, Muon and DiLoCo, while its logs showed a clear in-context-learning phase transition around 1.3B to 1.5B tokens. A stable checkpoint and training materials were published for continued work, but the token-efficiency estimate and capability interpretations are the author's claims.

    r/LocalLLaMA · 🔥 0
  7. arXiv preprint: long-horizon RL training recipe leans on the Muon optimizer

    An arXiv paper submitted on 14 October 2026, "Long-Horizon RL Needs Muon," by Kai Ruan, Jinghao Lin, Tao Chen and three co-authors, examines reinforcement learning for long-horizon agents that must coordinate reasoning, tool use and environment feedback across extended interactions. The authors describe a single-rollout asynchronous recipe that combines the Muon optimizer, token-averaged loss, truncated importance sampling, and auxiliary length and tool-error penalties, and they study optimizer choices and recipe components for repository-level issue resolution across two model families. They report higher training success with the full recipe than with baselines, though the 7-page work-in-progress paper presents these results without independent replication.

    Hugging Face · Papers · 🔥 0

Experience and discussion from the community

Share my Muon experienceAsk about Muon

Nobody has shared their experience with Muon yet.