Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

FOAM Halves SOAP Optimizer State and Matches Reported Baselines

AI summary

A paper by Jaemyung Yu, Byeongho Heo, Sangdoo Yun, and Dongyoon Han introduces FOAM, a lightweight SOAP variant designed to reduce optimizer-state memory and eigendecomposition overhead. FOAM uses input-only preconditioning and replaces SOAP’s dense rotated second moment with a factored Adam-moment estimator based on row-by-column statistics. The authors report that, in LLM pre-training experiments from 124M to 760M parameters, FOAM matches strong SOAP and Muon baselines in validation loss, uses half of SOAP’s optimizer state, and runs at Muon-class wall-clock time. The supplied report presents these results as the authors’ findings and does not independently verify them.

In-depth insight

FOAM addresses the substantial optimizer-state memory and eigendecomposition overhead associated with SOAP. The reported results suggest that retaining SOAP-like training performance may not require SOAP’s full optimizer-state cost, although the findings are author-reported and limited to the tested model scale.

Computer Science > Machine Learning

arXiv:2611.16905 (cs)

[Submitted on 12 Nov 2026]

Title:Factored One-sided Adam-Moment for Practical and Scalable SOAP

Authors:Jaemyung Yu, Byeongho Heo, Sangdoo Yun, Dongyoon Han

View a PDF of the paper titled Factored One-sided Adam-Moment for Practical and Scalable SOAP, by Jaemyung Yu and 3 other authors

#FOAM#SOAP#Shampoo#Adam#AdamW#Muon#K-FAC

Original · 3 min readRead the original
Group chat

No comments yet. Start the conversation.