Kestrel_VLLM moves its MTP draft head to CUDA
Kestrel_VLLM author Kestrel_VLLM reports moving the MTP draft head from CPU to CUDA as an independent slot with its own one-layer KV cache, cursor, and packed weight keys. The implementation shares a layer-independent attention core with the main model, uses non-fatal optional binding with the feature disabled by default and fail-closed behavior, and reportedly passed 55 device-level comparison rows across 11 stages on an RTX 5090 with Qwen3.5-27B. The report provides no inference-speed results, and the dedicated MTP probe was not independently reproduced.
The report documents a substantive implementation of device-side MTP drafting in an LLM inference engine, with independent state management, shared attention code, optional integration safeguards, and stage-level validation.
31 · MTP 头上 GPU:一个独立槽位,与一条抽出来共用的注意力
一句话:把 MTP 草稿头从 CPU 搬到 CUDA——它必须是主模型之外的一个独立槽位(自带一层 KV、自带游标),接入必须走非致命绑定 + 默认关 + fail-closed,attention 核心从「主模型的第 L 层」里抽出来两边共用;实测设备级逐级对拍 T=5、11 阶段 × 5 token 共 55 行全 OK(maxrel 6.2e-7 ~ 1.5e-5)。 前置:先读 第 30 篇(MTP 模块、15 张量、四个口径点、无 oracle)。 环境:x86-64 · NVIDIA RTX 5090 · Qwen3.5-27B(q4 打包)· CUDA 后端。
一、问题与结论
| 指标 | 数值 | 判定 |
|---|---|---|
| 槽位形态 | 主模型之外的独立槽位,自带 1 层 KV(kv_width = 1024) | 实测 |
| 键落位 | 8 个打包 q4 大矩阵走同一口 rq 池;7 个 f32 小张量单独上载 | 实测 |
| 池增量(开档 vs 关档) | +8 entry / +227.8 MB(401 → 409) | 实测 |
| 设备级逐级对拍 | T=5,11 阶段 × 5 token = 55 行全 OK,maxrel 6.2e-7 ~ 1.5e-5(容差 1e-4) | 实测 |
| 缺键 / 错形状自检 | 均 rc=-1;两次失败都没破坏已挂好的槽位 | 实测 |