停止猜测 Amazon SageMaker AI 上的 TTS 自动缩放目标

人工智能

停止猜测 Amazon SageMaker AI 上的 TTS 自动缩放目标

您自办的文本转语音( TTS) 端点，请求开始排队。 当大多数呼叫者在监听时，延迟会超过你的服务级别目标( SLO)。 这个终点背后的自动缩放政策本来是用来防止这种情况的，但是它所衡量的目标值只是一个猜测，这样的猜测在真实流量证明错误之前会默默失败。

把号码弄错了，估计太保守了，舰队运行着GPU实例，大多闲置。 这花费了获得在每张发票上，但从来没有旅行事故报告。 看来太冲动了舰队有危险没人看见每个案例的并行请求都多于其长期承诺。 除非交通堵塞把排队时间推过SLO，否则没什么不对的当大多数来电者在听的时候情况就是这样这是隐藏的缺口。 在操纵台上看好的政策

在此帖子中，我们部署了8200万参数TTS型号的Kokoro-82M，作为Amazon SageMaker AI 实时端点上的定制容器。 AWS云开发 Kit(AWS CDK)堆栈处理部署。 我们用两个测量结果来取代猜测在突破SLO前，我们从这两个数字中推算出缩放阈值缩放阈值以及两个冷却值该员额的每个数字都来自该员额中提到的aws样本库的实际部署和测量运行。

为什么默认的自动缩放指标对 TTS 无效

大部分关于Amazon SageMaker AI自动缩放的公开指导都针对短同步请求/应答呼叫，或者大语言模型(LLM)服务，其中自然度量是每秒生成的符号。 TTS合成不符合两种模式：单项请求保持连接开放数秒，没有符号生成循环，可以附加一个公尺。 Kokoro-82M是非自动回转的，是一种StyleTTS2衍生物，带有ISTFTNet解码器，经过了对抗性训练，而不是通过自动回转的下位预测或迭代解码。 它从一个连续的通行证中产生一个完整的说法。 首先，决定一个 TTS 实例是否超载比它的声音简单：有多少个请求被排队，并且现在就执行。

SageMaker AI的原生指标Amazon SageMaker AI, Amazon SageMaker AI的缩放会删除整类失败

同时要求PerModel在10秒窗口上发布一个最大统计数据，与飞行中的平均计数客户计量标准直接不同。 工具说明这一点。 在货币阶梯的每跑一跑，它都根据相同的负载测量数量，并记录它们之间的比例。 然后它使用这个比率将客户端测量的阈值转换成Amazon Cloud Watch单元，然后将其写入可部署配置。 对于Kokoro-82M在本文中运行，测量的比例在每跑时都出到完全1.0，因此客户端测量的和Amazon Cloud Watch报告的阈值直接吻合。

解决方案概述

这个解决方案在Amazon SageMaker AI 实时端点上的自定义容器中部署Kokoro-82M，然后将一个猜测的自动缩放目标替换为从对活端点进行的两次测量得出的一个：QQmax，一个实例的货币上限在SLO内部服务，T total，一个实际缩放需要多长时间。 一个合成请求流经四层：一个客户端，Amazon SageMaker AI端点，该端点后面一个GPU实例上的定制容器，以及容器内部的Kokoro-82M模型。

容器支持两条电线运输：反应流线(Invoke Endpoint With ResponseStream)，在合成时将音频回流，在发音完成前开始回放，以及双向流线(Invoke Endpoint With BiformeStream)，它有一个长寿的会话开放给多个消息。

在定制的容器里运行Kokoro-82M，而不是在管理下的推论工具包后面运行，是使得其余的这个员额成为可能的原因。 你所控制的容器可以执行自己的入境限制，并揭露Amazon SageMaker AI已经公布的确切货币信号，部署这个端点的AWS CDK堆栈，只有在模型存在一个真实的，测量的计划时，才会附加自动缩放资源，因此一个猜测的配置永远无法默默到达生产。 此帖余为度量。

先决条件

要继续走下去你需要：

- AWS账户，授权创建AWS身份和访问管理(IAM)角色，Amazon简单存储服务(Amazon S3)桶，Amazon弹性集装箱登记(Amazon ECR)寄存器，以及Amazon SageMaker AI资源。

- 注意：样品容器将较早的PyTorch贴上已知的咨询(PYSEC-2026-2286，固定在火炬2.1.0中)。 在任何无价值防伪用途之前审查并升级标针。

- AWS CDK v2，安装和拖入目标账户和区域。

- Python 3.11或以后，uv，和Docker来构建容器图像。

- 在运行该职位的升级度量之前，至少有一个空档位超过基线。

- 来自GitHub寄存器的部署代码在此帖子中引用。

走过去

到这一步结束，你将会有一个活的 Amazon SageMaker AI 自动缩放策略，用于 Kokoro-82M，完全来自对自己的端点进行的两次测量而不是猜测。

部署端点堆栈

端点通过三个AWS CDK堆栈部署，按顺序应用：SpeechFoundationStack创建了IAM角色和缓存模型重量的Amazon S3桶，SpeechModelCacheStack将一个模型的Hugging Face重量同步到该桶中一次，SpeechEndpointStack构建容器图像，推向Amazon ECR，并创建了Amazon SageMaker AI模型，端点配置，以及端点。

第三堆上的闸门保持默认的，未测量的缩放配置无法到达生产。 缩放启用是一个静态旗，当最大内置配置大于 min 内置时为真，称本身对它背后的数字是否曾被测量过没有任何信息。 堆栈新增在合成时间一秒独立检查：这个模型是否存在真实可行的计划文物？只有当两个条件都存在时，堆栈才会合成缩放资源：一个将缩放策略和警报接线的EndpointAutoscaling构造，以及一个EndpointObservation构造，用来监测它的仪表板和警报的新增。 一个有缩放启用=True但后面没有测量的模型，不会得到与有缩放启用=的模型相同的缩放资源输出。 虚假的。

项目的 CLI 将三个堆栈包在一个命令后面，按依赖顺序应用：

SpeechModelCacheStack运行AWS代码构建项目，将模型的Hugging Face重量同步到缓存的Amazon S3桶中一次。 冷落的起点不需要从互联网上拉出来。 由前作的缩放启用的检查，SpeechEndpointStack构建容器图像，将其推向Amazon ECR，并创建Amazon SageMaker AI模型，端点配置，以及端点。

作为命令行工具链，按固定顺序运行的方法包：

步骤1：检查漂移

漂浮在第一，每一次，在任何花费钱或持有终点负载。 这是在实际部署(可扩展目标、缩放政策、警报状态)与配置文件称应部署内容之间，早期模式的孤儿政策，或停止或取消的qmax或ttotal run，否则会静悄悄地移动能力中度测量并腐蚀所测量的数字。

一个干净的结果报告没有出现有飞行前状态的错配和退出，为qmax扫清了道路。 停止或取消的 qmax 或 ttotal 运行是报告错配的最常见来源，因为它可以在单个实例上留下自动缩放冻结。 运行 tts-bench 解冻 -- endpoint speech-kokoro-82m 在您开始前恢复已部署的政策，因为 qmax 和 ttotal 都假设已部署状态与配置文件完全匹配。

第2步：度量系数，货币上限

QQmax 是最大的货币，请求队列加请求执行一次，其中请求仍以 95 百分位数到达 SLO 内的第一个音频字节。 使用闭路负载梯子测量：精确地持有固定的货币N，闭环避免了协调的遗漏，即固定到达率测试中的缺陷，即一个慢速服务器使得客户端发送的请求较少而不是表面显示真正的排队延迟。

- 干燥运行打印时间表和一个墙钟的估计零AWS的电话，所以你可以预算大约36分钟的费用一个直播运行先。 现场指令会冻结自动缩放，并在持续时间里将容量固定在一个实例中，在退出时自动恢复，所以它产生的数字是真实的每英寸上限。

负载梯级记录如下：

max = 50，且是括号内的结果：rung 60 实际跨越 SLO，同样的运行测量了未中断的服务时间，一个请求花费的时间完全没有排队：61.3 ms,p95 66.1 ms. 该数字驱动了第二个独立警报在本文后面描述。

步骤 3：措施 T 总计，实际退出时间

总而言之，墙钟需要多长时间，从要求更多的容量到该容量服务于健康的流量。 它的衡量方法是直接强制扩大规模，绕过已部署的政策本身的检测逻辑，然后给每个可区分的阶段打上时间戳。

完全需要 qmax 文物作为输入，因为它的回收判断读出该梯子自己的测量径

强制扩大分为八个衡量阶段：

主导阶段是Amazon弹性计算云(Amazon EC2)实例提供加容器图像牵引，为计划总量的181秒56%。 减少T total影响最大的方式不是应用代码，而是减少图像大小或使用热容量池。 在“在原木中可见”的下游，每件东西加起来都相对较小79秒。

超过260.2秒，它与政策本身的配置(警报评价期乘以评价期，加冷却)是分界线的，因为这一运行故意绕过了政策的检测逻辑，以隔离纯供给时间。 计划规模与320.2秒总和相对应，因为这种反应时间是任何生产规模外的一个真正部分，尽管这种测量排除了它。

步骤4：通过计划制定已部署的政策

计划是唯一从未触及AWS的命令。 它读了 qmax 和 ttotal 的文物，结合了两个操作员选择的输入，一个是 3000 ms p95 TTFAB 的 SLO，一个是 1.25x 的激增率( 舰队容忍 25% 的流量在一个规模外周期内增长而不突破 SLO)，并且做了以下的 ar

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Artificial Intelligence

Stop guessing auto scaling targets for TTS on Amazon SageMaker AI

A traffic spike hits your self-hosted text-to-speech (TTS) endpoint, and requests start queueing. Latency climbs past your service level objective (SLO) right when the most callers are listening. The auto scaling policy behind that endpoint was supposed to prevent this, but the target value it scales on was a guess, and guesses like that fail silently until real traffic proves them wrong.

Get that number wrong, and it’s paid for in one of two ways, and neither shows up as an alarm. Guess too conservative, and the fleet runs GPU instances that sit mostly idle. That cost lands on every invoice but never trips an incident report. Guess too aggressive, and the fleet carries a risk nobody sees. Each instance holds more concurrent requests than it can serve inside its latency promise. Nothing looks wrong until a real traffic spike pushes queueing time past the SLO. That happens right when the most callers are listening. That’s the hidden gap. A policy that looks fine on the console can already be failing the people it’s supposed to serve.

In this post, we deploy Kokoro-82M, an 82-million-parameter TTS model, as a custom container on an Amazon SageMaker AI real-time endpoint. AWS Cloud Development Kit (AWS CDK) stacks handle the deployment. We replace guesswork with two measurements taken against the live endpoint. Q_max is the highest concurrency one instance sustains before breaching the SLO, and T_total is how long a scale-out takes end to end. We derive the scale-out threshold, scale-in threshold, and both cooldowns from those two numbers by arithmetic. Every number in this post comes from an actual deployment and measurement run of the aws-samples repository referenced in this post.

Why default auto scaling metrics don’t work for TTS

Most public guidance on Amazon SageMaker AI auto scaling targets short synchronous request/response calls, or large language model (LLM) serving, where the natural metric is tokens generated per second. TTS synthesis fits neither pattern: a single request holds a connection open for several seconds with no token-generation loop to attach a metric to. Kokoro-82M is non-autoregressive, a StyleTTS2 derivative with an ISTFTNet decoder, trained adversarially rather than by autoregressive next-token prediction or iterative denoising. It produces a full utterance from one duration-aligned pass. There’s no generation loop to instrument with a per-token metric in the first place. What determines whether a TTS instance is overloaded is simpler than it sounds: how many requests are queued and executing on it right now.

The deployed policy scales on ConcurrentRequestsPerModel, a native metric Amazon SageMaker AI publishes itself rather than one this project’s own container code has to emit. Scaling on a metric Amazon SageMaker AI publishes removes an entire class of failure where a custom metric configuration looks correct on paper but nothing is actually publishing to it.

ConcurrentRequestsPerModel publishes a Maximum statistic over a 10-second window, a different quantity from the mean in-flight count a benchmarking client measures directly. The tooling accounts for this. At every rung of the concurrency ladder it measures both quantities against the same load and records the ratio between them. It then uses that ratio to convert the client-measured threshold into Amazon CloudWatch units before writing it into the deployable configuration. For the Kokoro-82M run in this post, the measured ratio came out to exactly 1.0 at every rung, so the client-measured and Amazon CloudWatch-reported thresholds match directly.

Solution overview

This solution deploys Kokoro-82M in a custom container on an Amazon SageMaker AI real-time endpoint, then replaces a guessed auto scaling target with one derived from two measurements taken against that live endpoint: Q_max, the concurrency ceiling one instance serves inside the SLO, and T_total, how long a real scale-out takes. A synthesis request flows through four layers: a client, the Amazon SageMaker AI endpoint, a custom container on a GPU instance behind that endpoint, and the Kokoro-82M model inside the container.

The container supports two wire transports: response streaming (InvokeEndpointWithResponseStream), which streams audio back as it’s synthesized so playback starts before the utterance finishes, and bidirectional streaming (InvokeEndpointWithBidirectionalStream), which holds one long-lived session open for multiple messages.

Running Kokoro-82M in a custom container, rather than behind a managed inference toolkit, is what makes the rest of this post possible. A container you control can enforce its own admission limits and expose the exact concurrency signal Amazon SageMaker AI already publishes, instead of retrofitting a borrowed metric onto a workload it wasn’t built for. The AWS CDK stacks that deploy this endpoint only attach auto scaling resources once a real, measured plan exists for the model, so a guessed configuration can never reach production silently. The rest of this post is that measurement.

Prerequisites

To follow this walkthrough, you need:

- An AWS account with permissions to create AWS Identity and Access Management (IAM) roles, Amazon Simple Storage Service (Amazon S3) buckets, Amazon Elastic Container Registry (Amazon ECR) repositories, and Amazon SageMaker AI resources.

- Note: the sample container pins an earlier PyTorch with a known advisory (PYSEC-2026-2286, fixed in torch 2.10.0). Review and upgrade the pin before any non-proof-of-value use.

- AWS CDK v2, installed and bootstrapped in your target account and region.

- Python 3.11 or later, uv, and Docker to build the container image.

- A ml.g5.xlarge service quota with at least one spare slot above your baseline before running the scale-out measurement in this post.

- The deployment code from the GitHub repository referenced in this post.

Walkthrough

By the end of this walkthrough, you will have a live Amazon SageMaker AI auto scaling policy for Kokoro-82M, derived entirely from two measurements taken against your own endpoint rather than from a guess.

Deploy the endpoint stacks

The endpoint deploys through three AWS CDK stacks, applied in order: SpeechFoundationStack creates the IAM roles and the Amazon S3 bucket that caches model weights, SpeechModelCacheStack syncs a model’s Hugging Face weights into that bucket once, and SpeechEndpointStack builds the container image, pushes it to Amazon ECR, and creates the Amazon SageMaker AI model, endpoint configuration, and endpoint.

The gate on the third stack keeps a default, unmeasured scaling configuration from reaching production. scaling_enabled is a static flag, true whenever max_instances is configured greater than min_instances, and it says nothing on its own about whether the numbers behind it were ever measured. The stack adds a second, independent check at synthesis time: does a real, feasible plan artifact exist for this model? Only when both conditions hold does the stack synthesize scaling resources: an EndpointAutoscaling construct that wires the scaling policy and alarms, and an EndpointObservability construct that adds the dashboards and alarms used to monitor it. A model with scaling_enabled=True but no measurement behind it gets the same no scaling resources output as one with scaling_enabled=False.

The project’s CLI wraps the three stacks behind a single command, applying them in dependency order:

SpeechModelCacheStack runs an AWS CodeBuild project that syncs the model’s Hugging Face weights into the cached Amazon S3 bucket once. A cold endpoint start doesn’t need to pull them from the internet. Gated by the preceding scaling_enabled check, SpeechEndpointStack builds the container image, pushes it to Amazon ECR, and creates the Amazon SageMaker AI model, endpoint configuration, and endpoint.

The methodology packages as a command-line tool chain, run in a fixed order:

Step 1: Check for drift

drift runs first, every time, before anything that costs money or holds an endpoint under load. It’s a read-only reconciliation between what’s actually deployed (scalable targets, scaling policies, alarm states) and what the config file says should be deployed. An orphaned policy from an earlier model, or a stopped or canceled qmax or ttotal run, would otherwise silently move capacity mid-measurement and corrupt the numbers being measured.

A clean result reports no mismatches and exits with a preflight-ok status, clearing the way for qmax. A stopped or canceled qmax or ttotal run is the most common source of a reported mismatch, because it can leave auto scaling frozen on a single instance. uv run tts-bench thaw --endpoint speech-kokoro-82m restores the deployed policy before you proceed, since qmax and ttotal both assume the deployed state matches the config file exactly.

Step 2: Measure Q_max, the concurrency ceiling

Q_max is the largest concurrency, requests queued plus requests executing on one instance, at which a request still reaches first audio byte inside the SLO at the 95th percentile. It’s measured with a closed-loop load ladder: hold a fixed concurrency N exactly, and walk N upward one rung at a time until the SLO breaks. A closed loop avoids coordinated omission, the flaw in fixed-arrival-rate tests where a slowing server makes the client send fewer requests instead of surfacing real queueing delay.

--dry-run prints the schedule and a wall-clock estimate with zero AWS calls, so you can budget the roughly 36-minute cost of a live run first. The live command freezes auto scaling and pins capacity to one instance for the duration, restored automatically on exit, so the number it produces is a genuine per-instance ceiling.

The load ladder recorded the following results:

Q_max = 50, and it’s a bracketed result: rung 60 actually crossed the SLO, which makes 50 a real ceiling rather than a lower bound. The same run measured the uncontended service time, the time one request takes with no queueing at all: mean 61.3 ms, p95 66.1 ms. That number drives a second, independent alarm described later in this post.

Step 3: Measure T_total, the real scale-out time

T_total is how long it takes, wall clock, from the moment more capacity is requested to the moment that capacity serves healthy traffic. It’s measured by forcing a scale-out directly, bypassing the deployed policy’s own detection logic, then timestamping every distinguishable stage.

ttotal requires the qmax artifact as input, since its recovery judgment reads off that ladder’s own measured rungs, and refuses to run against a ladder measured on a different wire transport.

The forced scale-out broke down into eight measured stages:

The dominant stage is Amazon Elastic Compute Cloud (Amazon EC2) instance provisioning plus the container image pull, at 181 seconds, 56 percent of the planned-against total. The highest-impact way to reduce T_total isn’t application code, it’s reducing image size or using warm capacity pools. Everything downstream of “instance visible in logs” is a comparatively small 79 seconds combined.

The 60-second addition on top of the measured 260.2 seconds isn’t measured in this run. It’s bounded from the policy’s own configuration (alarm evaluation period times evaluation periods, plus cooldown), because this run bypassed the policy’s detection logic on purpose to isolate pure provisioning time. plan sizes surge headroom against the full 320.2-second total, because that reaction time is a real part of any production scale-out even though this measurement excluded it.

Step 4: Derive the deployed policy with plan

plan is the only command that never touches AWS. It reads the qmax and ttotal artifacts, combines them with two operator-chosen inputs, an SLO of 3,000 ms p95 TTFAB and a surge ratio of 1.25x (the fleet tolerates traffic growing 25 percent within one scale-out cycle without breaching the SLO), and does the following ar

Auto-fetched and translated by AI Atlas. All rights belong to the original author.