vLLM scales GLM-5.3 to 1M-token contexts on 16 H200 GPUs
vLLM Blog reports a GLM-5.3 serving deployment that uses 16 NVIDIA H200 GPUs, divided into separate prefill and decode nodes. Prefill Context Parallelism splits prompts by token, Decode Context Parallelism shards the KV cache, and Hybrid HiSparse keeps decode KV data in host memory. On AgentX, the tested PCP8 × DCP8 configuration with Hybrid HiSparse reported 1.7–3.2× per-user speed at equal throughput to data-parallel serving and was the only tested configuration that served 1M-token contexts; however, these results came from the project's own contributors.
The report provides a concrete serving design for fitting and accelerating GLM-5.3 workloads with very long contexts, a relevant constraint for memory-constrained AI deployments.
TL;DR: Long agentic contexts push GLM-5.3 against two limits on H200s: KV cache memory and prefill speed. We address both on 16 H200s, split into one prefill node and one decode node:
- Prefill Context Parallelism (PCP) splits each prompt across GPUs by token instead of by head, which suits GLM-5.3's sparse attention far better than tensor parallelism.
- Decode Context Parallelism (DCP) shards the prefill engine's KV cache by token, so a 1M context fits on the prefill node.
- Hybrid HiSparse lets decode keep KV in host memory, so a 1M-token request fits.
- On AgentX, PCP8 × DCP8 prefill with Hybrid HiSparse decode gives 1.7–3.2× the per-user speed of a data-parallel deployment at the same throughput, and it is the only configuration we tested that serves 1M-token contexts.
Serving long contexts in memory-constrained deployments
Agentic workloads have long contexts that grow with every turn, which puts pressure on both GPU memory and prefill. On H200s, GLM-5.3's weights take up most of each GPU's memory, leaving little room for the KV cache of those long contexts. And because every turn adds more tool output and file contents to prefill, under load the prefill engines can become the bottleneck for the whole deployment.
#GLM-5.3#vLLM#NVIDIA H200#AgentX#Prefill Context Parallelism#Decode Context Parallelism#Hybrid HiSparse#Model Runner V2#NIXL