Dynamo

Headlines · 2

1. Oct 23
   vLLM Reports 954K Tokens per Second per GPU for DeepSeek V4.1 Flash

   In a vLLM-published benchmark, DeepSeek V4.1 Flash reached 954,000 tokens per second per GPU on a single NVIDIA GB300 NVL72 rack using P/D disaggregation and Wide-EP, with P90 throughput of 196 tokens per second per user. On GB200, vLLM’s decode throughput was within 6% of DeepInfer’s published results across batch sizes 8–256. A specialized decode path achieved 826 P90 tokens per second for one user on four B300 GPUs. In an AgentX prefill test at concurrency 256, DEP2 delivered 64,400 tokens per second per GPU, compared with 53,300 for DEP4. The report said the configurations differed and did not characterize the difference as an engine-to-engine speedup; replay tests attributed a 13.6% DEP4 throughput loss to data-parallel imbalance.

   vLLM Blog · 🔥 0
2. Oct 19
   vLLM cuts Elastic EP scaling downtime to under 2 seconds

   vLLM published a technical post describing how asynchronous preparation and the NIXL EP backend let a running Elastic Expert Parallelism deployment add or remove GPUs with near-zero serving downtime. In its DeepSeek V4 Pro benchmark on GB200 GPUs, scaling from 16 to 20 GPUs previously paused serving for 100 seconds under the earlier Elastic EP approach and 197 seconds with a full restart, while the new flow cut the pause to 0.82 seconds; scaling back to 16 GPUs fell from 56 seconds to 1.96 seconds. The reconfiguration flow splits into background preparation — initializing joining workers, building the target communication groups, transferring weights over NVLink or RDMA, and warming up kernels and CUDA graphs — and a short commit phase that briefly pauses schedulers; vLLM says the changes are in the latest vLLM and NIXL releases.

   vLLM Blog · 🔥 0

Experience and discussion from the community

Share my Dynamo experienceAsk about Dynamo