Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Kestrel_VLLM enables CUDA batched prefill by default

AI summary

Kestrel_VLLM decoupled CUDA batched-prefill block width from mdl_mmax, allowing adaptive trailing blocks and making the feature default to 16 when VLLM_CUDA_Q35_MM is unset. The report says logits remained bitwise equivalent in its tests on x86-64 with an NVIDIA RTX 5090 and Qwen3.5-27B q4 hybrid quantization. It also documents a variable-shadowing access violation and a timing result inflated by diagnostic stderr output, which was corrected from 57.41 ms to 26.75 ms.

In-depth insight

The optimization removes a fallback that prevented short prompts from benefiting fully from CUDA batched prefill while retaining bitwise-equivalence checks in the reported testing.

43 · 默认化前的一次解耦:一个遮蔽 bug 与一次口径污染

一句话:批量 prefill 接上了、位级等价、端到端 2.39×,但门默认关;要默认化,先得拆掉一个契约 —— 设备要求块宽严格等于 mdl_mmax,于是尾巴(n % mm)退回逐 token、n < mm 时整段回落(短 prompt 一点收益都拿不到)。这一轮把「块宽解耦」和「默认化」一次做完,落地期踩到两个坑:一个让程序段错误(变量遮蔽),一个让读数虚高一倍(一行诊断写在计时区里)—— 两个都不是靠更小心避开的,是诊断和编译器警告抓出来的。 前置:第 41 篇(prefill 批量化 —— 本轮的起点)、第 42 篇(M 上限 8→16)、第 37 篇(不可复现的构建与 SHA 的身份降级)。 环境:x86-64 + NVIDIA RTX 5090 · Qwen3.5-27B q4(hybrid)· CUDA 后端(阶段五)。

一、问题与结论

批量 prefill 接上了,位级等价、端到端 2.39× —— 但门默认关。

要默认化,先得拆掉一个契约:设备要求块宽严格等于 mdl_mmax,于是尾巴(n % mm)退回逐 token,n < mm 时整段回落(短 prompt 一点收益都拿不到)。

#Kestrel_VLLM#vLLM#Qwen3.5-27B#NVIDIA RTX 5090#CUDA

Original · 12 min readRead the original
Group chat

No comments yet. Start the conversation.