Adam
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- FOAM Halves SOAP Optimizer State and Matches Reported Baselines
A paper by Jaemyung Yu, Byeongho Heo, Sangdoo Yun, and Dongyoon Han introduces FOAM, a lightweight SOAP variant designed to reduce optimizer-state memory and eigendecomposition overhead. FOAM uses input-only preconditioning and replaces SOAP’s dense rotated second moment with a factored Adam-moment estimator based on row-by-column statistics. The authors report that, in LLM pre-training experiments from 124M to 760M parameters, FOAM matches strong SOAP and Muon baselines in validation loss, uses half of SOAP’s optimizer state, and runs at Muon-class wall-clock time. The supplied report presents these results as the authors’ findings and does not independently verify them.
Hugging Face · Papers · 🔥 0
Experience and discussion from the community
Share my Adam experienceAsk about Adam
Nobody has shared their experience with Adam yet.