CUDA Graph
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 2
- Kestrel_VLLM adds CUDA Graphs to a 64-layer hybrid inference path
The Kestrel_VLLM author implemented an opt-in CUDA Graph path for a 64-layer hybrid forward pass, moving token-dependent RoPE positions, KV-cache append indices, and attention visibility lengths from kernel arguments to device memory. On an NVIDIA RTX 5090 with Qwen3.5-27B q4, six interleaved A/B rounds reportedly achieved a 1.1335× speedup and saved 1.878 ms/token, while element-level memcmp found byte-identical outputs between the graph and streaming paths. An nsys measurement identified a 2.761 ms/token launch gap as an upper bound rather than the realized saving.
掘金 · 人工智能 · 🔥 0 - MoE Inference Optimization: Batching, DeepEP, DeepGEMM and Load Balancing
The sixth installment in an inference-infrastructure methodology series combines MoE batch-size theory with end-to-end measurements on eight H20-96 GPUs. It examines expert coverage, all-to-all communication, tile padding and expert-load imbalance, then analyzes how DeepEP, DeepGEMM and EPLB address distinct bottlenecks. The report cautions that its bench_one_batch_server sampling is not equivalent to online-service P99 latency and says an older throughput model should no longer be used for capacity planning. In the sampled runs, batch size 64 met the 30 ms TPOT limit at 2,828 tokens per second, while batch size 128 recorded 30.9 ms; the largest compliant batch size was not identified.
掘金 · 人工智能 · 🔥 0
Experience and discussion from the community
Share my CUDA Graph experienceAsk about CUDA Graph
Nobody has shared their experience with CUDA Graph yet.