Kestrel_VLLM adds CUDA Graphs to a 64-layer hybrid inference path
The Kestrel_VLLM author implemented an opt-in CUDA Graph path for a 64-layer hybrid forward pass, moving token-dependent RoPE positions, KV-cache append indices, and attention visibility lengths from kernel arguments to device memory. On an NVIDIA RTX 5090 with Qwen3.5-27B q4, six interleaved A/B rounds reportedly achieved a 1.1335× speedup and saved 1.878 ms/token, while element-level memcmp found byte-identical outputs between the graph and streaming paths. An nsys measurement identified a 2.761 ms/token launch gap as an upper bound rather than the realized saving.
The report provides a concrete implementation and regression-testing approach for reducing launch overhead in hybrid LLM inference, although its performance results are limited to one author's hardware, model, and toolchain.
29 · CUDA Graph 接到 hybrid 路径
一句话:把 26 篇只停留在「乐观(上限)外插」的 CUDA Graph 候选真正落地——先用 nsys 直接量出发射空档 2.761 ms/token(16.87%),得到天花板;再把 CUDA Graph 接到 64 层异构前向路径上,难点不在 graph API 本身,而在「把随 token 变化的位置量从内核参数搬到设备」;同会话交错 A/B 六轮 1.1335×(省 1.878 ms/token,是下界),两档数值经元素级
memcmp确认逐字节相同。 前置:建议先读 第 26 篇(发射空档的两个机制)与 第 28 篇(hybrid 接进引擎)。 环境:x86-64 + NVIDIA RTX 5090;模型 Qwen3.5-27B q4;工具 nsys / ncu;出厂档build-x64\vllm_cuda.dll。 三个数:① nsys 逐核起止区间合并取补集,发射空档 2.761 ms/token(16.87%)、1090 次发射/token ⇒ 2.53 µs/次(第 2.2 节,是 graph 的上界);② 同会话交错 A/B 六轮 1.1335×、省 1.878 ms/token(15.944 → 14.066 ms/token,第 6 节实测,是下界);③ graph 档与流式档的h64/hfinal/logits元素级 memcmp 逐字节相同(第 10.1 节)。 怎么读:结论看 一、问题与结论 与 十二、小结(必读);机制看 三、接线:把位置量从内核参数搬进设备、四、设计依据与两个坑、五、数值零回归与口径边界、七、与天花板对账:归因分级(想深挖再看);二、侦察:先把空档量出来、六、同会话交错 A/B、八、零回归全项与产物、十、收口(10.4–10.10)与文末附录属过程与原始数据(可跳)。
Some source content is omitted. Continue reading on the original site.