Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Amazon SageMaker Inference adds NVIDIA MIG GPU partitioning for endpoints

AI summary

Amazon SageMaker Inference now supports GPU partitioning that uses NVIDIA Multi-Instance GPU (MIG) to split a physical GPU into independent partitions, each with a fixed share of compute units and dedicated memory isolated in hardware. According to the AWS Machine Learning Blog, this lets customers run more models than an instance has physical accelerators and choose a fraction of a newer accelerator instead of a whole GPU. The feature works on inference component endpoints and is available on P4d, P5, G7 and G7e instance families across seven AWS Regions, subject to instance and quota availability; single-model endpoints, serverless endpoints, heterogeneous instance pools and asynchronous invocation are not supported.

In-depth insight

AWS says GPU partitioning removes the constraint that a managed endpoint must be sized around whole accelerators, allowing multiple isolated models per GPU and right-sized compute slices without running MIG on self-managed Kubernetes. Customers are still billed for the full GPU instance, and the announcement comes from a single vendor blog post with stated limits.

When you deploy a model to a GPU endpoint, the smallest unit of GPU you can request is one whole accelerator.

This constraint can cost you twice. First, you can’t host more models on an instance than it has accelerators. An ml.p4d.24xlarge has eight A100s, so it holds eight models, whether each one needs 40 GB or 14 GB. Second, and less obvious, it makes every deployment come down to a choice of which full GPU to deploy to. As your concurrency and prompt lengths grow, a full GPU on an older instance stops being enough, but a full GPU on a newer one is more than your model can use. You end up either under-served or paying for capability you never reach.

Today, Amazon SageMaker Inference supports GPU partitioning, which uses NVIDIA Multi-Instance GPU (MIG) to divide a physical GPU into independent partitions. Each partition receives a fixed share of the GPU’s compute units and a dedicated block of its memory, with isolation enforced in hardware. That changes both constraints: you can place a different model in each partition, and you can choose a fraction of a newer accelerator instead of a whole one of either generation.

#Amazon SageMaker Inference#NVIDIA Multi-Instance GPU (MIG)#AWS#NVIDIA

Original · 15 min readRead the original
Group chat

No comments yet. Start the conversation.