Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Kestrel_VLLM

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 4

  1. Kestrel_VLLM enables CUDA batched prefill by default

    Kestrel_VLLM decoupled CUDA batched-prefill block width from mdl_mmax, allowing adaptive trailing blocks and making the feature default to 16 when VLLM_CUDA_Q35_MM is unset. The report says logits remained bitwise equivalent in its tests on x86-64 with an NVIDIA RTX 5090 and Qwen3.5-27B q4 hybrid quantization. It also documents a variable-shadowing access violation and a timing result inflated by diagnostic stderr output, which was corrected from 57.41 ms to 26.75 ms.

    掘金 · 人工智能 · 🔥 0
  2. Kestrel_VLLM Cuts GDN Kernel Launches by Moving the Batch M Loop Inside

    Kestrel_VLLM developer measurements show that moving the batch M loop into CUDA kernels reduced per-layer launches from 27 to 9 at M=4 and from 51 to 9 at M=8. In three interleaved M=4 rounds, the merged path averaged 24.063 ms versus 28.182 ms for the per-M path, a reported 14.6% reduction, while memcmp tests found bitwise-identical outputs across real mixed batches from M=2 through M=8. The merged route became the default with an environment-variable fallback, but the evidence comes from one self-published account and some performance figures were not reproduced in the final checklist.

    掘金 · 人工智能 · 🔥 0
  3. Kestrel_VLLM moves its MTP draft head to CUDA

    Kestrel_VLLM author Kestrel_VLLM reports moving the MTP draft head from CPU to CUDA as an independent slot with its own one-layer KV cache, cursor, and packed weight keys. The implementation shares a layer-independent attention core with the main model, uses non-fatal optional binding with the feature disabled by default and fail-closed behavior, and reportedly passed 55 device-level comparison rows across 11 stages on an RTX 5090 with Qwen3.5-27B. The report provides no inference-speed results, and the dedicated MTP probe was not independently reproduced.

    掘金 · 人工智能 · 🔥 0
  4. Kestrel_VLLM adds CUDA Graphs to a 64-layer hybrid inference path

    The Kestrel_VLLM author implemented an opt-in CUDA Graph path for a 64-layer hybrid forward pass, moving token-dependent RoPE positions, KV-cache append indices, and attention visibility lengths from kernel arguments to device memory. On an NVIDIA RTX 5090 with Qwen3.5-27B q4, six interleaved A/B rounds reportedly achieved a 1.1335× speedup and saved 1.878 ms/token, while element-level memcmp found byte-identical outputs between the graph and streaming paths. An nsys measurement identified a 2.761 ms/token launch gap as an upper bound rather than the realized saving.

    掘金 · 人工智能 · 🔥 0

Experience and discussion from the community

Share my Kestrel_VLLM experienceAsk about Kestrel_VLLM

Nobody has shared their experience with Kestrel_VLLM yet.