AWS 发布基于实测的 SageMaker TTS 自动扩缩容配置方法
AWS 机器学习博客发文介绍如何通过 AWS CDK 堆栈,将 8200 万参数的非自回归语音合成模型 Kokoro-82M 以自定义容器部署到 Amazon SageMaker AI 实时端点。作者不再凭猜测设定自动扩缩容目标值,而是在真实端点上测量两个数字:Q_max,即单实例在不突破 3000 毫秒延迟 SLO 前提下能承受的最高并发;以及 T_total,即一次完整扩容所需时间。文章给出的结果是 ml.g5.xlarge 上 Q_max 为 50 并发(并发 60 时突破 SLO),实测 T_total 为 260.2 秒,加上有界的策略检测延迟后按 320.2 秒规划,再由这两个数字用算术推导出扩容阈值、缩容阈值和两个冷却时间。
文章认为这套四参数模型同样适用于其他对延迟敏感、受并发约束的 SageMaker AI 实时端点,并指出基于 AWS 自身发布的 ConcurrentRequestsPerModel 指标扩缩容,可避免自定义指标配置看似正确却无人上报的失效情况。
A traffic spike hits your self-hosted text-to-speech (TTS) endpoint, and requests start queueing. Latency climbs past your service level objective (SLO) right when the most callers are listening. The auto scaling policy behind that endpoint was supposed to prevent this, but the target value it scales on was a guess, and guesses like that fail silently until real traffic proves them wrong.
Get that number wrong, and it’s paid for in one of two ways, and neither shows up as an alarm. Guess too conservative, and the fleet runs GPU instances that sit mostly idle. That cost lands on every invoice but never trips an incident report. Guess too aggressive, and the fleet carries a risk nobody sees. Each instance holds more concurrent requests than it can serve inside its latency promise. Nothing looks wrong until a real traffic spike pushes queueing time past the SLO. That happens right when the most callers are listening. That’s the hidden gap. A policy that looks fine on the console can already be failing the people it’s supposed to serve.