Model Runner V2

Headlines · 2

1. Oct 23
   vLLM scales GLM-5.3 to 1M-token contexts on 16 H200 GPUs

   vLLM Blog reports a GLM-5.3 serving deployment that uses 16 NVIDIA H200 GPUs, divided into separate prefill and decode nodes. Prefill Context Parallelism splits prompts by token, Decode Context Parallelism shards the KV cache, and Hybrid HiSparse keeps decode KV data in host memory. On AgentX, the tested PCP8 × DCP8 configuration with Hybrid HiSparse reported 1.7–3.2× per-user speed at equal throughput to data-parallel serving and was the only tested configuration that served 1M-token contexts; however, these results came from the project's own contributors.

   vLLM Blog · 🔥 0
2. Oct 6
   vLLM v0.31.0 expands serving, speculation, and model support

   vLLM v0.31.0 adds broad serving and model-support updates, including DeepSeek-V4.1-Flash optimizations, Model Runner V2 speculative decoding, larger-scale expert parallelism, and improved multimodal support. It also introduces the `vllm preload` weight-cache daemon for faster engine restarts and experimental CRIU-based initialized-engine snapshots, alongside several security and breaking changes.

   vLLM Releases · 🔥 0

Experience and discussion from the community

Share my Model Runner V2 experienceAsk about Model Runner V2