Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

SageMaker HyperPod Inference Adds Managed NVIDIA Dynamo Support

AI summary

AWS Machine Learning Blog says Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo for multi-node model serving, disaggregated prefill and decode, and KV-aware routing. The managed add-on installs Dynamo components and provides endpoint configuration, GPU scheduling, node recovery, and autoscaling, with deployment examples using GLM-5.3 on vLLM.

In-depth insight

The integration gives AWS customers a managed path to deploy advanced distributed inference patterns on SageMaker HyperPod Inference, though its performance and operational benefits remain vendor-reported.

Serving large language models (LLMs) efficiently requires sufficient GPU memory and careful coordination of prompt processing and token generation. GLM-5.3’s FP8 weights, for example, exceed the 640 GB of GPU memory on a single ml.p5.48xlarge instance, before accounting for KV cache and runtime overhead. Long prompts can also slow token generation when both share the same GPUs.

Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, combining Dynamo’s distributed inference capabilities with the infrastructure management of HyperPod to address these challenges. Dynamo is an open source distributed inference framework that supports multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines such as vLLM, SGLang, and TensorRT-LLM. HyperPod supports these deployments with high-performance Elastic Fabric Adapter (EFA) networking, coordinated GPU scheduling, and node health monitoring and recovery. In this post, we illustrate multi-node and disaggregated serving through a deployment example, share benchmark results, and outline best practices for optimizing inference on HyperPod.

How the integration works

The HyperPod Inference add-on installs the Dynamo custom resource definitions and these serving and scheduling components:

  • HyperPod Inference Operator manages endpoint deployment through an InferenceEndpointConfig (IEC) custom resource. Its dynamoSpec section supplies the Dynamo configuration.
  • Dynamo operator manages resources defined in a DynamoGraphDeployment (DGD), including the frontend, model workers, replica counts, and resource requirements.
  • Grove coordinates related groups of pods, including multi-node workers, and their scaling relationships.
  • Kubernetes AI (KAI) Scheduler places pods on GPU resources, using gang scheduling to schedule the pods required for a multi-node worker together.

#Amazon SageMaker HyperPod Inference#NVIDIA Dynamo#AWS#NVIDIA#GLM-5.3#vLLM#SGLang#TensorRT-LLM

Original · 7 min readRead the original
Group chat

No comments yet. Start the conversation.