Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

JetBrains Adds MTP Head to Mellum2.1 Thinking

AI summary

A JetBrains model card reposted on Reddit presents Mellum2.1 Thinking with a multi-token prediction (MTP) head, an additional transformer block trained to predict the token following the sampled one. The model card says inference engines can use it for speculative drafting while verifying every token, claiming 1.4–1.8x faster decoding and about 90% draft acceptance; support is listed for vLLM 0.33+, Transformers 5.20+, and llama.cpp.

In-depth insight

The release adds a speculative-decoding mechanism to a concrete open model, but its reported performance gains have not been independently validated.

Mellum2.1 Thinking with the MTP head Multi-token prediction head The multi-token prediction (MTP) head is one extra transformer block, trained together with the model, that predicts the token after the one just sampled. Inference engines use it as a speculative draft: the model verifies every drafted token, so the output is unchanged while decoding runs 1.4–1.8x faster, with about 90% of the drafted tokens accepted. The head lives in model-mtp.safetensors under the mtp. prefix, announced by num_nextn_predict_layers: 1 in config.json; everything else is identical to Mellum2.1 Thinking. submitted by /u/pmttyji [link] [comments]

#JetBrains#Mellum2.1 Thinking#MTP head#vLLM#Transformers#llama.cpp

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.