K-FAC
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 2
- CurvDelta targets safer LLM adaptation and selective unlearning
A research paper submitted by Zeguan Xiao and five coauthors introduces CurvDelta, a framework that embeds capability-preservation constraints into model parameter updates. CurvDelta uses low-curvature geometry, K-FAC and factorized projection, while CurvLoRA provides a low-rank variant; the authors report competitive task performance with better general-capability retention on mathematical reasoning, code generation, TOFU and MUSE.
Hugging Face · Papers · 🔥 0 - FOAM Halves SOAP Optimizer State and Matches Reported Baselines
A paper by Jaemyung Yu, Byeongho Heo, Sangdoo Yun, and Dongyoon Han introduces FOAM, a lightweight SOAP variant designed to reduce optimizer-state memory and eigendecomposition overhead. FOAM uses input-only preconditioning and replaces SOAP’s dense rotated second moment with a factored Adam-moment estimator based on row-by-column statistics. The authors report that, in LLM pre-training experiments from 124M to 760M parameters, FOAM matches strong SOAP and Muon baselines in validation loss, uses half of SOAP’s optimizer state, and runs at Muon-class wall-clock time. The supplied report presents these results as the authors’ findings and does not independently verify them.
Hugging Face · Papers · 🔥 0
Experience and discussion from the community
Share my K-FAC experienceAsk about K-FAC
Nobody has shared their experience with K-FAC yet.