AWS shows multi-node inference with Ray LLM Deep Learning Containers
An AWS Machine Learning Blog post walks through deploying one large language model across two GPU nodes on Amazon EKS using the multi-node Ray LLM Deep Learning Container, with separate prefill and decode nodes behind a single OpenAI-compatible endpoint. The container ships Ray Serve LLM, NVIDIA's NIXL library and the Elastic Fabric Adapter stack, and the example uses Qwen3.5-9B to show prefill/decode disaggregation configured entirely through YAML with no application code. The post also warns that NIXL silently falls back to a TCP-speed transport when it cannot use libfabric, so it recommends checking worker logs to confirm the backend.
According to AWS, the container supplies a tested inference stack so teams can run multi-node serving without assembling GPU and EFA dependencies themselves.
This post shows how to deploy one large language model (LLM) across two GPU nodes on Amazon Elastic Kubernetes Service (Amazon EKS) with the multi-node Ray LLM Deep Learning Containers. It uses separate prefill and decode nodes behind one OpenAI-compatible endpoint. This kind of multi-node setup is common for tensor, pipeline, and data parallelism, as well as prefill/decode disaggregation inference workloads. In a previous post, we introduced the Ray Serve DLC and used it to stand up a small model behind an HTTP endpoint on a single GPU node. You had a working inference endpoint, and the container keeps the GPU stack compatible, including CUDA libraries, so you did not need to write boilerplate code.
Ray is a distributed compute framework, so Ray Serve deployments can span multiple nodes. Customers use multi-node Ray Serve for several reasons. Some want to run foundation or open-weight models that are too large for a single GPU. Others need more throughput than one replica can deliver. Still others want to separate the two phases of inference to improve performance. In each case, customers must build the dependency chain themselves: choosing compatible libraries for the GPU stack and debugging issues when a component drifts out of alignment. This is undifferentiated work that takes time and effort away from delivering models.
#AWS#Ray#Ray Serve#Amazon EKS#Qwen3.5-9B#NVIDIA NIXL#Elastic Fabric Adapter#KubeRay#vLLM