Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

CUDA Graph

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 2

  1. Kestrel_VLLM adds CUDA Graphs to a 64-layer hybrid inference path

    The Kestrel_VLLM author implemented an opt-in CUDA Graph path for a 64-layer hybrid forward pass, moving token-dependent RoPE positions, KV-cache append indices, and attention visibility lengths from kernel arguments to device memory. On an NVIDIA RTX 5090 with Qwen3.5-27B q4, six interleaved A/B rounds reportedly achieved a 1.1335× speedup and saved 1.878 ms/token, while element-level memcmp found byte-identical outputs between the graph and streaming paths. An nsys measurement identified a 2.761 ms/token launch gap as an upper bound rather than the realized saving.

    掘金 · 人工智能 · 🔥 0
  2. MoE Inference Optimization: Batching, DeepEP, DeepGEMM and Load Balancing

    The sixth installment in an inference-infrastructure methodology series combines MoE batch-size theory with end-to-end measurements on eight H20-96 GPUs. It examines expert coverage, all-to-all communication, tile padding and expert-load imbalance, then analyzes how DeepEP, DeepGEMM and EPLB address distinct bottlenecks. The report cautions that its bench_one_batch_server sampling is not equivalent to online-service P99 latency and says an older throughput model should no longer be used for capacity planning. In the sampled runs, batch size 64 met the 30 ms TPOT limit at 2,828 tokens per second, while batch size 128 recorded 30.9 ms; the largest compliant batch size was not identified.

    掘金 · 人工智能 · 🔥 0

Experience and discussion from the community

Share my CUDA Graph experienceAsk about CUDA Graph

Nobody has shared their experience with CUDA Graph yet.