AWS shows how to set SageMaker TTS auto scaling targets from live measurements
An AWS Machine Learning Blog post describes deploying Kokoro-82M, an 82-million-parameter non-autoregressive text-to-speech model, as a custom container on an Amazon SageMaker AI real-time endpoint via AWS CDK stacks. Instead of guessing auto scaling target values, the authors measure two numbers against the live endpoint: Q_max, the highest concurrency one instance sustains before breaching a 3,000 ms latency SLO, and T_total, the end-to-end scale-out time. The post reports Q_max of 50 concurrent requests on ml.g5.xlarge (concurrency 60 breached the SLO) and a measured T_total of 260.2 seconds, planned against as 320.2 seconds with bounded policy detection lag, and derives scale-out and scale-in thresholds and cooldowns arithmetically from those values.
The post argues the four-number model generalizes to other latency-sensitive, concurrency-bound real-time SageMaker AI endpoints, and that scaling on the AWS-published ConcurrentRequestsPerModel metric avoids custom-metric configurations that look correct but publish nothing.
A traffic spike hits your self-hosted text-to-speech (TTS) endpoint, and requests start queueing. Latency climbs past your service level objective (SLO) right when the most callers are listening. The auto scaling policy behind that endpoint was supposed to prevent this, but the target value it scales on was a guess, and guesses like that fail silently until real traffic proves them wrong.
Get that number wrong, and it’s paid for in one of two ways, and neither shows up as an alarm. Guess too conservative, and the fleet runs GPU instances that sit mostly idle. That cost lands on every invoice but never trips an incident report. Guess too aggressive, and the fleet carries a risk nobody sees. Each instance holds more concurrent requests than it can serve inside its latency promise. Nothing looks wrong until a real traffic spike pushes queueing time past the SLO. That happens right when the most callers are listening. That’s the hidden gap. A policy that looks fine on the console can already be failing the people it’s supposed to serve.