SageMaker HyperPod Inference支持NVIDIA Dynamo
AWS Machine Learning Blog称,Amazon SageMaker HyperPod Inference现可支持NVIDIA Dynamo,包括多节点模型服务、预填充与解码分离以及KV感知路由。托管附加组件可安装Dynamo组件,并提供端点配置、GPU调度、节点恢复和自动扩缩容,文中还给出了在vLLM上部署GLM-5.3的示例。
该集成为AWS客户提供了在SageMaker HyperPod Inference上部署高级分布式推理方案的托管路径,但性能与运维收益仍主要来自厂商陈述。
Serving large language models (LLMs) efficiently requires sufficient GPU memory and careful coordination of prompt processing and token generation. GLM-5.3’s FP8 weights, for example, exceed the 640 GB of GPU memory on a single ml.p5.48xlarge instance, before accounting for KV cache and runtime overhead. Long prompts can also slow token generation when both share the same GPUs.
Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, combining Dynamo’s distributed inference capabilities with the infrastructure management of HyperPod to address these challenges. Dynamo is an open source distributed inference framework that supports multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines such as vLLM, SGLang, and TensorRT-LLM. HyperPod supports these deployments with high-performance Elastic Fabric Adapter (EFA) networking, coordinated GPU scheduling, and node health monitoring and recovery. In this post, we illustrate multi-node and disaggregated serving through a deployment example, share benchmark results, and outline best practices for optimizing inference on HyperPod.
How the integration works
The HyperPod Inference add-on installs the Dynamo custom resource definitions and these serving and scheduling components:
- HyperPod Inference Operator manages endpoint deployment through an InferenceEndpointConfig (IEC) custom resource. Its
dynamoSpecsection supplies the Dynamo configuration. - Dynamo operator manages resources defined in a DynamoGraphDeployment (DGD), including the frontend, model workers, replica counts, and resource requirements.
- Grove coordinates related groups of pods, including multi-node workers, and their scaling relationships.
- Kubernetes AI (KAI) Scheduler places pods on GPU resources, using gang scheduling to schedule the pods required for a multi-node worker together.
#Amazon SageMaker HyperPod Inference#NVIDIA Dynamo#AWS#NVIDIA#GLM-5.3#vLLM#SGLang#TensorRT-LLM