llm-d

Headlines · 1

1. Oct 19
   vLLM cuts Elastic EP scaling downtime to under 2 seconds

   vLLM published a technical post describing how asynchronous preparation and the NIXL EP backend let a running Elastic Expert Parallelism deployment add or remove GPUs with near-zero serving downtime. In its DeepSeek V4 Pro benchmark on GB200 GPUs, scaling from 16 to 20 GPUs previously paused serving for 100 seconds under the earlier Elastic EP approach and 197 seconds with a full restart, while the new flow cut the pause to 0.82 seconds; scaling back to 16 GPUs fell from 56 seconds to 1.96 seconds. The reconfiguration flow splits into background preparation — initializing joining workers, building the target communication groups, transferring weights over NVLink or RDMA, and warming up kernels and CUDA graphs — and a short commit phase that briefly pauses schedulers; vLLM says the changes are in the latest vLLM and NIXL releases.

   vLLM Blog · 🔥 0

Experience and discussion from the community

Share my llm-d experienceAsk about llm-d