SageMaker Inference 支持 NVIDIA MIG GPU 分区
Amazon SageMaker Inference 现已支持 GPU 分区,可利用 NVIDIA Multi-Instance GPU(MIG)将一块物理 GPU 划分为多个独立分区,每个分区获得固定份额的计算单元和专属显存,并由硬件提供隔离。据 AWS 机器学习博客称,这使用户可在实例上部署多于物理加速器数量的模型,并可选择新一代加速器的部分算力而非整卡。该功能适用于 inference component 端点,涵盖 P4d、P5、G7、G7e 实例族及七个 AWS 区域;单模型端点、serverless 端点、异构实例池和异步调用均不支持。
据 AWS 介绍,GPU 分区打破了托管推理端点必须按整块加速器选型的限制,可在单卡上以硬件隔离运行多个模型,也无需自行在 Kubernetes 上配置 MIG;但用户仍按整卡实例付费,且该消息仅来自单一厂商博客,限制条件较多。
When you deploy a model to a GPU endpoint, the smallest unit of GPU you can request is one whole accelerator.
This constraint can cost you twice. First, you can’t host more models on an instance than it has accelerators. An ml.p4d.24xlarge has eight A100s, so it holds eight models, whether each one needs 40 GB or 14 GB. Second, and less obvious, it makes every deployment come down to a choice of which full GPU to deploy to. As your concurrency and prompt lengths grow, a full GPU on an older instance stops being enough, but a full GPU on a newer one is more than your model can use. You end up either under-served or paying for capability you never reach.
Today, Amazon SageMaker Inference supports GPU partitioning, which uses NVIDIA Multi-Instance GPU (MIG) to divide a physical GPU into independent partitions. Each partition receives a fixed share of the GPU’s compute units and a dedicated block of its memory, with isolation enforced in hardware. That changes both constraints: you can place a different model in each partition, and you can choose a fraction of a newer accelerator instead of a whole one of either generation.
#Amazon SageMaker Inference#NVIDIA Multi-Instance GPU (MIG)#AWS#NVIDIA