创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

苹果研究:双语数据混合优于超参数调优

AI 总结

苹果机器学习研究团队发表论文《Mix, Don't Tune》,在约1000次预训练实验中系统比较了两种缓解数据受限的方法:激进超参数调优(如高权重衰减)与混入高资源语言数据。实验覆盖150M至1.43B四个模型规模,以阿拉伯语为目标语言、英语为辅助语言,结果显示数据混合在验证损失和下游任务准确率上的提升均大于超参数调优,且差距随模型规模扩大。作者称,混合带来的增益相当于2–3倍独有目标数据(验证损失)和2–13倍(下游准确率),并指出目标语言验证损失会系统性低估混合的价值。

深度见解

作者指出,目标语言的验证损失只反映混合的正则化效果,无法体现其带来的新知识,因此会系统性低估数据混合的价值。

research area Speech and Natural Language Processing

content type paperpublished October 2026

Mix, Don’t Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings

AuthorsPaul Jeha†**, Anastasiia Sedova, Louis Béthune, Skyler Seto, Jes Frellsen†‡, Pierre Ablin‡, Natalie Schluter‡

View publication

For most languages of the world, language model pre-training operates in a data-constrained regime where models must repeat their training data many times, degrading generalization. Two remedies exist: aggressive hyperparameter tuning such as high weight decay, and mixing in data from a high-resource auxiliary language to directly aid the low-resource target. While hyperparameter tuning regularizes the model by shrinking weights to restrict network capacity, auxiliary data mixing uses a tunable mixing ratio to expand the training distribution and diversify the training signal with new knowledge. Both offer a principled way to improve training in a data-constrained domain. We compare these levers systematically across four model scales from 150M to 1.43B parameters, using Arabic as the low-resource target and English as the auxiliary, over approximately 1000 pre-training runs. Three findings emerge. First, mixing yields larger improvements than hyperparameter tuning on both validation loss and downstream task accuracy, and the gap grows with model size. Second, we quantify how much mixing helps: it boosts performance by an amount equivalent to 2–3× the unique target data on validation loss and 2–13× on downstream task accuracy, with the gain scaling steeply with model size. Third, this divergence reveals that target-language validation loss systematically underestimates mixing’s value. Mixing regularizes by diversifying the training signal and contributes knowledge the repeated target corpus cannot supply; validation loss captures only the first effect. Our practical recommendations are: mix in a high-resource language, prioritize the mixing ratio over hyperparameter tuning, and transfer hyperparameters from a small proxy model via μP.

#Apple#Apple Machine Learning Research

原文约 2 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。