创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

Soft-MTP改善多词元预测信号

AI 总结

Jayden Teoh的预印本《Multi-Token Prediction Done Right》提出Soft-MTP,通过梯度恢复方法将KL自蒸馏信号与标准多词元预测训练中的噪声残差分离。论文称,Soft-MTP只需一行代码修改,支持DeepSeek和Meta风格的MTP,不增加参数,并在167M至1.2B规模的实验中报告了更低的困惑度、更快的推测解码和更高的零样本准确率。上述性能提升主要由作者自报,尚未获得独立验证。

深度见解

Soft-MTP提供了一种低开销、无需新增参数的MTP训练改进方案,但报告的性能优势仍需独立复现。

Sharing one of our latest papers on Multi-Token Prediction (MTP). MTP is now a standard recipe for frontier pre-training but turns out, it can be done much better. We show the standard MTP gradient hides a KL self-distillation signal (which we want for improving speculative decoding) behind a noisy residual (that harms next-token prediction). I think the coolest part was the KL gradient–recovery trick to extract the KL gradient. You just need a one-line code change: h_ntp += h_mtp.detach() - h_mtp Soft Multi-Token Prediction (Soft-MTP) is essentially a free lunch that improves MTP across a range of tasks. I'm super excited about this work. Please do check it out below: 💻 Code: https://github.com/JaydenTeoh/Soft-MTP 📝 Paper: https://arxiv.org/abs/2610.38331 🤗 HuggingFace: https://huggingface.co/collections/JaydenTeoh/soft-multi-token-prediction submitted by /u/jayden_teoh_ [link] [comments]

#Soft-MTP#Multi-Token Prediction#DeepSeek#Meta#Jayden Teoh

原文约 1 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。