Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Kestrel_VLLM moves its MTP draft head to CUDA

AI summary

Kestrel_VLLM author Kestrel_VLLM reports moving the MTP draft head from CPU to CUDA as an independent slot with its own one-layer KV cache, cursor, and packed weight keys. The implementation shares a layer-independent attention core with the main model, uses non-fatal optional binding with the feature disabled by default and fail-closed behavior, and reportedly passed 55 device-level comparison rows across 11 stages on an RTX 5090 with Qwen3.5-27B. The report provides no inference-speed results, and the dedicated MTP probe was not independently reproduced.

In-depth insight

The report documents a substantive implementation of device-side MTP drafting in an LLM inference engine, with independent state management, shared attention code, optional integration safeguards, and stage-level validation.

31 · MTP 头上 GPU:一个独立槽位,与一条抽出来共用的注意力

一句话:把 MTP 草稿头从 CPU 搬到 CUDA——它必须是主模型之外的一个独立槽位(自带一层 KV、自带游标),接入必须走非致命绑定 + 默认关 + fail-closed,attention 核心从「主模型的第 L 层」里抽出来两边共用;实测设备级逐级对拍 T=5、11 阶段 × 5 token 共 55 行全 OK(maxrel 6.2e-7 ~ 1.5e-5)。 前置:先读 第 30 篇(MTP 模块、15 张量、四个口径点、无 oracle)。 环境:x86-64 · NVIDIA RTX 5090 · Qwen3.5-27B(q4 打包)· CUDA 后端。

一、问题与结论

指标数值判定
槽位形态主模型之外的独立槽位,自带 1 层 KV(kv_width = 1024)实测
键落位8 个打包 q4 大矩阵走同一口 rq 池;7 个 f32 小张量单独上载实测
池增量(开档 vs 关档)+8 entry / +227.8 MB(401 → 409)实测
设备级逐级对拍T=5,11 阶段 × 5 token = 55 行全 OK,maxrel 6.2e-7 ~ 1.5e-5(容差 1e-4)实测
缺键 / 错形状自检均 rc=-1;两次失败都没破坏已挂好的槽位实测

#Kestrel_VLLM#Qwen3.5-27B#NVIDIA RTX 5090#CUDA

Original · 13 min readRead the original
Group chat

No comments yet. Start the conversation.