Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Kestrel_VLLM Cuts GDN Kernel Launches by Moving the Batch M Loop Inside

AI summary

Kestrel_VLLM developer measurements show that moving the batch M loop into CUDA kernels reduced per-layer launches from 27 to 9 at M=4 and from 51 to 9 at M=8. In three interleaved M=4 rounds, the merged path averaged 24.063 ms versus 28.182 ms for the per-M path, a reported 14.6% reduction, while memcmp tests found bitwise-identical outputs across real mixed batches from M=2 through M=8. The merged route became the default with an environment-variable fallback, but the evidence comes from one self-published account and some performance figures were not reproduced in the final checklist.

In-depth insight

The report describes a substantial AI-inference optimization that decouples GDN kernel-launch count from batch size, with the developer reporting lower latency without changing numerical outputs.

35 · 把 m 循环搬进内核:一次「先量发射、再合并发射」的完整回合

一句话:第 34 篇把批量 M 行建出来了,但只证明算术是对的。第一次真正量它的性能时,实测比模型预测高约 35%——本篇把这笔账从对不上走到结清:逐相分解 → 定位到 GDN 子层 → 造两个诊断档把「发射」与「工作」分开 → 坐实「每行成本 = 发射主导」→ 把 6 次/行的发射合并成 1 次 → 默认化。拿到了阶段五单笔最大的一刀(M=4 的 verify 段 2.130× → 2.495×,默认化后整步 2.521×)。 前置:先读 第 34 篇(批量 M 行的五个块);对照 第 29 篇(「发射次数多 ≠ 发射开销大」)。 环境:x86-64 · NVIDIA RTX 5090 · Qwen3.5-27B(q4 打包)· CUDA 后端;同会话交错 3 轮。

一、问题与结论

指标数值 / 结果判定
「高了 35%」先变成分相账缺口落在 ② 子层,病态的是 GDN 子层(attn-64 0.930 / gdn-64 3.260 / mixed 3.501 ms/行)实测
门禁:LF2 每行增量0.38–0.54 ms/行,低于 attn 档的 0.930,远低于 real 的 3.26–3.50实测
落地:每层发射数M=4:27 → 9;M=8:51 → 9(与 M 无关)实测
M=4 verify 段28.182 → 24.063 ms,省 4.119 ms(−14.6%);提速 2.130× → 2.495×实测
M=8省 9.34 ms(2.460× → 3.207×)实测
默认化后整步M=4 2.521× / M=8 3.174×实测

#Kestrel_VLLM#Qwen3.5-27B#CUDA#NVIDIA RTX 5090

Original · 13 min readRead the original
Group chat

No comments yet. Start the conversation.