创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

SageMaker HyperPod Inference支持NVIDIA Dynamo

AI 总结

AWS Machine Learning Blog称,Amazon SageMaker HyperPod Inference现可支持NVIDIA Dynamo,包括多节点模型服务、预填充与解码分离以及KV感知路由。托管附加组件可安装Dynamo组件,并提供端点配置、GPU调度、节点恢复和自动扩缩容,文中还给出了在vLLM上部署GLM-5.3的示例。

深度见解

该集成为AWS客户提供了在SageMaker HyperPod Inference上部署高级分布式推理方案的托管路径,但性能与运维收益仍主要来自厂商陈述。

Serving large language models (LLMs) efficiently requires sufficient GPU memory and careful coordination of prompt processing and token generation. GLM-5.3’s FP8 weights, for example, exceed the 640 GB of GPU memory on a single ml.p5.48xlarge instance, before accounting for KV cache and runtime overhead. Long prompts can also slow token generation when both share the same GPUs.

Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, combining Dynamo’s distributed inference capabilities with the infrastructure management of HyperPod to address these challenges. Dynamo is an open source distributed inference framework that supports multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines such as vLLM, SGLang, and TensorRT-LLM. HyperPod supports these deployments with high-performance Elastic Fabric Adapter (EFA) networking, coordinated GPU scheduling, and node health monitoring and recovery. In this post, we illustrate multi-node and disaggregated serving through a deployment example, share benchmark results, and outline best practices for optimizing inference on HyperPod.

How the integration works

The HyperPod Inference add-on installs the Dynamo custom resource definitions and these serving and scheduling components:

  • HyperPod Inference Operator manages endpoint deployment through an InferenceEndpointConfig (IEC) custom resource. Its dynamoSpec section supplies the Dynamo configuration.
  • Dynamo operator manages resources defined in a DynamoGraphDeployment (DGD), including the frontend, model workers, replica counts, and resource requirements.
  • Grove coordinates related groups of pods, including multi-node workers, and their scaling relationships.
  • Kubernetes AI (KAI) Scheduler places pods on GPU resources, using gang scheduling to schedule the pods required for a multi-node worker together.

#Amazon SageMaker HyperPod Inference#NVIDIA Dynamo#AWS#NVIDIA#GLM-5.3#vLLM#SGLang#TensorRT-LLM

原文约 7 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。