Soft-MTP改善多词元预测信号
AI 总结
Jayden Teoh的预印本《Multi-Token Prediction Done Right》提出Soft-MTP,通过梯度恢复方法将KL自蒸馏信号与标准多词元预测训练中的噪声残差分离。论文称,Soft-MTP只需一行代码修改,支持DeepSeek和Meta风格的MTP,不增加参数,并在167M至1.2B规模的实验中报告了更低的困惑度、更快的推测解码和更高的零样本准确率。上述性能提升主要由作者自报,尚未获得独立验证。
深度见解
Soft-MTP提供了一种低开销、无需新增参数的MTP训练改进方案,但报告的性能优势仍需独立复现。
| Sharing one of our latest papers on Multi-Token Prediction (MTP). MTP is now a standard recipe for frontier pre-training but turns out, it can be done much better. We show the standard MTP gradient hides a KL self-distillation signal (which we want for improving speculative decoding) behind a noisy residual (that harms next-token prediction). I think the coolest part was the KL gradient–recovery trick to extract the KL gradient. You just need a one-line code change: h_ntp += h_mtp.detach() - h_mtp Soft Multi-Token Prediction (Soft-MTP) is essentially a free lunch that improves MTP across a range of tasks. I'm super excited about this work. Please do check it out below: 💻 Code: https://github.com/JaydenTeoh/Soft-MTP 📝 Paper: https://arxiv.org/abs/2610.38331 🤗 HuggingFace: https://huggingface.co/collections/JaydenTeoh/soft-multi-token-prediction submitted by /u/jayden_teoh_ [link] [comments] |