创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

SageMaker HyperPod推出全球推理网关

AI 总结

AWS推出Amazon SageMaker HyperPod Global Inference Gateway,可通过一个托管端点将推理请求路由至同一区域或多个区域内的已注册集群。该网关依据KV缓存余量、队列深度、区域距离和错误率选择集群,并支持自动故障转移、SigV4与IAM身份验证及PrivateLink连接。AWS称该功能已在13个区域提供,但性能和可靠性收益仍主要来自厂商陈述。

深度见解

此次发布为生产级AI推理部署提供了托管方式,可统一多集群和跨区域端点,并将流量避开容量饱和或发生故障的集群。

During a traffic peak, a request can sit in a queue on a saturated HyperPod cluster while GPUs on another cluster, sometimes in another AWS Region, sit idle. You have already paid for that capacity, but your requests can’t reach it.

In Introducing Amazon SageMaker HyperPod Inference Gateway, we showed how the Inference Gateway gives each Amazon SageMaker HyperPod cluster a single, GPU-aware endpoint. It reads real-time signals such as key-value (KV) cache utilization and queue depth to place every request on the best-suited pod. That solves routing inside one cluster, but not across clusters.

Today, SageMaker HyperPod introduces the Global Inference Gateway. It is one SageMaker-managed gateway that routes across every cluster you register, in one Region or across Regions. The gateway picks the cluster and carries the request over the AWS backbone, with automatic failover and authentication through AWS Signature Version 4 (SigV4) and AWS Identity and Access Management (IAM). Inside each cluster, the Inference Gateway keeps picking the pod, unchanged. The Global Inference Gateway is the second tier of the architecture we described in part 1.

#AWS#Amazon SageMaker HyperPod#Amazon SageMaker HyperPod Global Inference Gateway#Amazon SageMaker HyperPod Inference Gateway#AWS PrivateLink#AWS Identity and Access Management

原文约 11 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。