vLLM Reports 954K Tokens per Second per GPU for DeepSeek V4.1 Flash
In a vLLM-published benchmark, DeepSeek V4.1 Flash reached 954,000 tokens per second per GPU on a single NVIDIA GB300 NVL72 rack using P/D disaggregation and Wide-EP, with P90 throughput of 196 tokens per second per user. On GB200, vLLM’s decode throughput was within 6% of DeepInfer’s published results across batch sizes 8–256. A specialized decode path achieved 826 P90 tokens per second for one user on four B300 GPUs. In an AgentX prefill test at concurrency 256, DEP2 delivered 64,400 tokens per second per GPU, compared with 53,300 for DEP4. The report said the configurations differed and did not characterize the difference as an engine-to-engine speedup; replay tests attributed a 13.6% DEP4 throughput loss to data-parallel imbalance.
The study provides detailed measurements and optimization lessons for serving a large MoE model on NVIDIA infrastructure, including the use of small DEP instances for prefill and a specialized path for low-latency decoding. Its headline benchmark results come from a single vLLM-published study rather than independent confirmation.
TL;DR: On one NVIDIA GB300 NVL72 rack, vLLM with P/D disaggregation and Wide-EP serves DeepSeek-V4.1-Flash on SemiAnalysis AgentX at 954K tokens/s per GPU at a P90 of 196 tokens/s per user, 7.9× our single-node result from the first post and $38.96 per GPU-hour at DeepSeek's off-peak prices. A specialized decode path reaches 826 tokens/s for a single user. Against DeepInfer, DeepSeek's own engine, vLLM's decode throughput is within 6% across batch sizes 8–256, and its prefill tracks DeepInfer's published curve.
The efficiency comes from deliberate engineering on components that vLLM already has, built on the kernels DeepSeek open-sourced. This post walks through our performance study, breaks down each optimization, and shares the lessons we learned:
- A general-purpose engine can match a model-specific one
- Understand the model before optimizing
- Small DEP instances work best for prefill1
- Wide-EP decode scales well with EPLB
- CPU overhead becomes critical at NVL72 scale
- A specialized decode path unlocks low latency
#DeepSeek V4.1 Flash#vLLM#DeepInfer#NVIDIA GB300 NVL72#NVIDIA B300#SemiAnalysis AgentX#Dynamo