FOAM Halves SOAP Optimizer State and Matches Reported Baselines
A paper by Jaemyung Yu, Byeongho Heo, Sangdoo Yun, and Dongyoon Han introduces FOAM, a lightweight SOAP variant designed to reduce optimizer-state memory and eigendecomposition overhead. FOAM uses input-only preconditioning and replaces SOAP’s dense rotated second moment with a factored Adam-moment estimator based on row-by-column statistics. The authors report that, in LLM pre-training experiments from 124M to 760M parameters, FOAM matches strong SOAP and Muon baselines in validation loss, uses half of SOAP’s optimizer state, and runs at Muon-class wall-clock time. The supplied report presents these results as the authors’ findings and does not independently verify them.
FOAM addresses the substantial optimizer-state memory and eigendecomposition overhead associated with SOAP. The reported results suggest that retaining SOAP-like training performance may not require SOAP’s full optimizer-state cost, although the findings are author-reported and limited to the tested model scale.
Computer Science > Machine Learning
arXiv:2611.16905 (cs)
[Submitted on 12 Nov 2026]
Title:Factored One-sided Adam-Moment for Practical and Scalable SOAP
Authors:Jaemyung Yu, Byeongho Heo, Sangdoo Yun, Dongyoon Han
View a PDF of the paper titled Factored One-sided Adam-Moment for Practical and Scalable SOAP, by Jaemyung Yu and 3 other authors