AWS EFA

Headlines · 1

1. 5d
   vLLM adds push-based KV transfer for disaggregated inference

   A vLLM blog post introduces a push-based NIXL connector for prefill-decode disaggregated inference that has the decode instance register its destination blocks in advance and starts the transfer from a background thread, moving post-prefill coordination and setup off the time-to-first-token critical path. In the authors' benchmark using Qwen3-32B on a 1P1D deployment of Amazon EC2 P5en instances connected by AWS EFA, push mode delivered up to 3x better TTFT and up to 30% better time-per-output-token than the default pull-based mode, with results depending on workload and configuration. At 8 QPS and 16k input, TTFT was about 540 ms in push mode versus about 1,744 ms in pull mode. The feature was built and validated on AWS Trainium clusters with EFA and contributed to vLLM as an accelerator-agnostic capability.

   vLLM Blog · 🔥 0

Experience and discussion from the community

Share my AWS EFA experienceAsk about AWS EFA