Kestrel_VLLM
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 4
- Kestrel_VLLM enables CUDA batched prefill by default
Kestrel_VLLM decoupled CUDA batched-prefill block width from mdl_mmax, allowing adaptive trailing blocks and making the feature default to 16 when VLLM_CUDA_Q35_MM is unset. The report says logits remained bitwise equivalent in its tests on x86-64 with an NVIDIA RTX 5090 and Qwen3.5-27B q4 hybrid quantization. It also documents a variable-shadowing access violation and a timing result inflated by diagnostic stderr output, which was corrected from 57.41 ms to 26.75 ms.
掘金 · 人工智能 · 🔥 0 - Kestrel_VLLM Cuts GDN Kernel Launches by Moving the Batch M Loop Inside
Kestrel_VLLM developer measurements show that moving the batch M loop into CUDA kernels reduced per-layer launches from 27 to 9 at M=4 and from 51 to 9 at M=8. In three interleaved M=4 rounds, the merged path averaged 24.063 ms versus 28.182 ms for the per-M path, a reported 14.6% reduction, while memcmp tests found bitwise-identical outputs across real mixed batches from M=2 through M=8. The merged route became the default with an environment-variable fallback, but the evidence comes from one self-published account and some performance figures were not reproduced in the final checklist.
掘金 · 人工智能 · 🔥 0 - Kestrel_VLLM moves its MTP draft head to CUDA
Kestrel_VLLM author Kestrel_VLLM reports moving the MTP draft head from CPU to CUDA as an independent slot with its own one-layer KV cache, cursor, and packed weight keys. The implementation shares a layer-independent attention core with the main model, uses non-fatal optional binding with the feature disabled by default and fail-closed behavior, and reportedly passed 55 device-level comparison rows across 11 stages on an RTX 5090 with Qwen3.5-27B. The report provides no inference-speed results, and the dedicated MTP probe was not independently reproduced.
掘金 · 人工智能 · 🔥 0 - Kestrel_VLLM adds CUDA Graphs to a 64-layer hybrid inference path
The Kestrel_VLLM author implemented an opt-in CUDA Graph path for a 64-layer hybrid forward pass, moving token-dependent RoPE positions, KV-cache append indices, and attention visibility lengths from kernel arguments to device memory. On an NVIDIA RTX 5090 with Qwen3.5-27B q4, six interleaved A/B rounds reportedly achieved a 1.1335× speedup and saved 1.878 ms/token, while element-level memcmp found byte-identical outputs between the graph and streaming paths. An nsys measurement identified a 2.761 ms/token launch gap as an upper bound rather than the realized saving.
掘金 · 人工智能 · 🔥 0
Experience and discussion from the community
Share my Kestrel_VLLM experienceAsk about Kestrel_VLLM
Nobody has shared their experience with Kestrel_VLLM yet.