Soft-MTP Recovers a Cleaner Signal for Multi-Token Prediction
Jayden Teoh's preprint “Multi-Token Prediction Done Right” introduces Soft-MTP, a gradient-recovery method that separates a KL self-distillation signal from noisy residual components in standard multi-token prediction training. The paper reports that Soft-MTP can be implemented as a one-line change, supports DeepSeek- and Meta-style MTP, adds no parameters, and showed lower perplexity, faster speculative decoding, and higher zero-shot accuracy across model sizes from 167M to 1.2B. These gains are reported by the paper's author and have not been independently corroborated in the supplied sources.
Soft-MTP offers a simple, reportedly low-overhead way to improve MTP training while preserving speculative-decoding benefits, but its reported performance remains an author claim pending independent validation.
| Sharing one of our latest papers on Multi-Token Prediction (MTP). MTP is now a standard recipe for frontier pre-training but turns out, it can be done much better. We show the standard MTP gradient hides a KL self-distillation signal (which we want for improving speculative decoding) behind a noisy residual (that harms next-token prediction). I think the coolest part was the KL gradient–recovery trick to extract the KL gradient. You just need a one-line code change: h_ntp += h_mtp.detach() - h_mtp Soft Multi-Token Prediction (Soft-MTP) is essentially a free lunch that improves MTP across a range of tasks. I'm super excited about this work. Please do check it out below: 💻 Code: https://github.com/JaydenTeoh/Soft-MTP 📝 Paper: https://arxiv.org/abs/2610.38331 🤗 HuggingFace: https://huggingface.co/collections/JaydenTeoh/soft-multi-token-prediction submitted by /u/jayden_teoh_ [link] [comments] |