zai-org
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- vLLM adds bi-directional KV transfer to cut TTFT in multi-turn agent workloads
The vLLM blog describes bi-directional KV cache transfer for prefill-decode (P-D) disaggregated serving: on later turns of a conversation, a decode node can return previously computed KV to the prefill node so shared context is not recomputed. In multi-turn benchmarks on a 1P1D deployment using Amazon EC2 P5en instances with AWS Elastic Fabric Adapter, the feature reduced TTFT by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8, with results depending on workload and configuration. The capability was originally built and validated on AWS Trainium clusters and contributed to vLLM as an accelerator-agnostic feature.
vLLM Blog · 🔥 0
Experience and discussion from the community
Share my zai-org experienceAsk about zai-org
Nobody has shared their experience with zai-org yet.