AWS CDK简化SageMaker端点扩缩更新
AWS Machine Learning Blog 记录了 CloudFormation 或 AWS CDK 在 Application Auto Scaling 仍处于注册状态时更新 SageMaker AI 实时端点所导致的部署故障。文章介绍一个可复用的 CDK 构件,通过三个自定义资源按依赖顺序注销并重新注册扩缩机制,从而在一次部署中更新端点配置和扩缩参数。
该厂商构件处理了一个可复现的基础设施管理问题,否则端点可能在无报错的情况下失去自动扩缩,或使 CloudFormation 堆栈需要人工恢复。
Updating the Amazon Sagemaker AI real-time endpoint configuration (changing the instance type, instance count, or environment variables) causes deployments through AWS CloudFormation or the AWS Cloud Development Kit (AWS CDK) to fail. SageMaker AI requires you to deregister the scalable target before updating the endpoint, but when you deregister it, CloudFormation can’t track the AWS Application Auto Scaling resources.
In this post, we show you how to use a reusable CDK construct with AWS CloudFormation custom resources to orchestrate deregister and register auto scaling automatically during endpoint updates. You can update your endpoint configuration and auto scaling parameters in a single deployment without manual intervention.
Why this problem occurs
When you deploy a SageMaker AI endpoint with native CloudFormation auto scaling resources, then change the endpoint configuration and redeploy, several failure modes emerge. The specific failure depends on what you change and how you work around the issue.
Changing the endpoint configuration with auto scaling registered
CloudFormation tries to update the endpoint while the scalable target is still registered. SageMaker AI rejects the update with the following error:
#Amazon SageMaker AI#AWS CDK#AWS CloudFormation#AWS Application Auto Scaling