Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Soft-MTP Recovers a Cleaner Signal for Multi-Token Prediction

AI summary

Jayden Teoh's preprint “Multi-Token Prediction Done Right” introduces Soft-MTP, a gradient-recovery method that separates a KL self-distillation signal from noisy residual components in standard multi-token prediction training. The paper reports that Soft-MTP can be implemented as a one-line change, supports DeepSeek- and Meta-style MTP, adds no parameters, and showed lower perplexity, faster speculative decoding, and higher zero-shot accuracy across model sizes from 167M to 1.2B. These gains are reported by the paper's author and have not been independently corroborated in the supplied sources.

In-depth insight

Soft-MTP offers a simple, reportedly low-overhead way to improve MTP training while preserving speculative-decoding benefits, but its reported performance remains an author claim pending independent validation.

Sharing one of our latest papers on Multi-Token Prediction (MTP). MTP is now a standard recipe for frontier pre-training but turns out, it can be done much better. We show the standard MTP gradient hides a KL self-distillation signal (which we want for improving speculative decoding) behind a noisy residual (that harms next-token prediction). I think the coolest part was the KL gradient–recovery trick to extract the KL gradient. You just need a one-line code change: h_ntp += h_mtp.detach() - h_mtp Soft Multi-Token Prediction (Soft-MTP) is essentially a free lunch that improves MTP across a range of tasks. I'm super excited about this work. Please do check it out below: 💻 Code: https://github.com/JaydenTeoh/Soft-MTP 📝 Paper: https://arxiv.org/abs/2610.38331 🤗 HuggingFace: https://huggingface.co/collections/JaydenTeoh/soft-multi-token-prediction submitted by /u/jayden_teoh_ [link] [comments]

#Soft-MTP#Multi-Token Prediction#DeepSeek#Meta#Jayden Teoh

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.