Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Shampoo

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 2

  1. Reddit user questions OKLS step efficiency versus Muon

    A Reddit post asks why OKLS, described as a zero-staleness implementation of KL-Shampoo, may be outperformed by Muon in step efficiency. The submitter claims that SOAP is more step-efficient than Shampoo, KL-Shampoo is more step-efficient than SOAP, and OKLS only surpasses Muon in parameter efficiency during very long training runs. The supplied post does not include benchmark measurements, implementation details, or responses that resolve these comparisons.

    r/MachineLearning · 🔥 0
  2. FOAM Halves SOAP Optimizer State and Matches Reported Baselines

    A paper by Jaemyung Yu, Byeongho Heo, Sangdoo Yun, and Dongyoon Han introduces FOAM, a lightweight SOAP variant designed to reduce optimizer-state memory and eigendecomposition overhead. FOAM uses input-only preconditioning and replaces SOAP’s dense rotated second moment with a factored Adam-moment estimator based on row-by-column statistics. The authors report that, in LLM pre-training experiments from 124M to 760M parameters, FOAM matches strong SOAP and Muon baselines in validation loss, uses half of SOAP’s optimizer state, and runs at Muon-class wall-clock time. The supplied report presents these results as the authors’ findings and does not independently verify them.

    Hugging Face · Papers · 🔥 0

Experience and discussion from the community

Share my Shampoo experienceAsk about Shampoo

Nobody has shared their experience with Shampoo yet.