Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

AWS adds cross-Region inference routing to SageMaker AI

AI summary

AWS announced that cross-Region inference routing is generally available in Amazon SageMaker AI for real-time and asynchronous inference in most commercial AWS Regions. Its inference router provides one invocation path across registered endpoints and inference components in different Regions and accounts, preferring healthy local capacity before shifting new requests and gradually restoring local traffic after recovery. AWS says the routing feature adds no fee beyond standard SageMaker AI and CloudWatch charges, while response streaming is unsupported and some quotas and regional exclusions apply.

In-depth insight

The release gives SageMaker AI users a managed way to route inference requests around regional capacity shortfalls without building and operating their own failover logic, though AWS's reliability claims are vendor-reported.

Serving a model from more than one AWS Region is a well-established pattern for highly available inference: when one Region runs short of GPU capacity, requests move to another. The pattern is simple to describe and hard to operate. Until now, if you served models on Amazon SageMaker AI managed endpoints, cross-Region routing was a piece you still had to build yourself.

Today, we’re introducing cross-Region inference routing in Amazon SageMaker AI. It uses a new resource, the inference router, that gives your application one invocation path across the endpoints and inference components you already run, in one or more AWS Regions and AWS accounts. You register the destinations that serve a model, and your application invokes the router instead of a specific endpoint. The router keeps requests in its own Region while the local destination is healthy, shifts new requests to another registered destination when it isn’t, and returns traffic gradually after recovery.

In this post, we explain how cross-Region inference routing decides where each request runs, walk through an example with one model in two Regions, and cover monitoring, limits, and what you pay. For a cross-account deployment that serves several models, see How Salesforce is advancing global-scale model serving with cross-Region inference routing in Amazon SageMaker AI.

#AWS#Amazon SageMaker AI#SageMaker AI inference router#Amazon CloudWatch#AWS PrivateLink

Original · 15 min readRead the original
Group chat

No comments yet. Start the conversation.