Amazon EC2 P5en

Headlines · 2

1. 5d
   vLLM adds push-based KV transfer for disaggregated inference

   A vLLM blog post introduces a push-based NIXL connector for prefill-decode disaggregated inference that has the decode instance register its destination blocks in advance and starts the transfer from a background thread, moving post-prefill coordination and setup off the time-to-first-token critical path. In the authors' benchmark using Qwen3-32B on a 1P1D deployment of Amazon EC2 P5en instances connected by AWS EFA, push mode delivered up to 3x better TTFT and up to 30% better time-per-output-token than the default pull-based mode, with results depending on workload and configuration. At 8 QPS and 16k input, TTFT was about 540 ms in push mode versus about 1,744 ms in pull mode. The feature was built and validated on AWS Trainium clusters with EFA and contributed to vLLM as an accelerator-agnostic capability.

   vLLM Blog · 🔥 0
2. Oct 12
   vLLM adds bi-directional KV transfer to cut TTFT in multi-turn agent workloads

   The vLLM blog describes bi-directional KV cache transfer for prefill-decode (P-D) disaggregated serving: on later turns of a conversation, a decode node can return previously computed KV to the prefill node so shared context is not recomputed. In multi-turn benchmarks on a 1P1D deployment using Amazon EC2 P5en instances with AWS Elastic Fabric Adapter, the feature reduced TTFT by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8, with results depending on workload and configuration. The capability was originally built and validated on AWS Trainium clusters and contributed to vLLM as an accelerator-agnostic feature.

   vLLM Blog · 🔥 0

Experience and discussion from the community

Share my Amazon EC2 P5en experienceAsk about Amazon EC2 P5en