AI agents使用Kokoro TTS在Amazon SageMaker AI上流传的低纬度演讲

人工智能

AI agents使用Kokoro TTS在Amazon SageMaker AI上流传的低纬度演讲

顾客向AI语音agent提问并等待。 agent已经有一个答案，当一个答案到来时，客户感到沮丧，困惑，不再关注。

文本转语音集成恰恰造成了这个差距。 一刑反应可能在600毫秒内产生，但全反应需要5秒，整合等待所有反应才开始合成。

这个显示的帖子如何缩小差距。 在Amazon SageMaker AI上，我们将每个句子流到一个主机Kokoro-82M 文本转语音(TTS)模型上，只要agent产生它，那么合成就从句子1开始，而不是完成的响应。 把音频带回浏览器也需要它自己的解决方案，所以我们还要既包括传输，又包括每个操作的成本。

解决方案概述

这个解决方案通过流式合成与生成一起，而不是等待它完成来弥补缺口。 由于agent产生每个句子，该句子流立即到一个主机的Kokoro-82M 文本转语音模型，因此在agent完成完全的响应前可以开始音频回放。 Kokoro-82M只持有8200万个参数，小到足以在单一的中层GPU实例上廉价运行。 这让处理真实交通的agent语音成本既低又低。

Kokoro-82M在SageMaker AI 实时端点上的定制容器中运行，运行在一个ml.g5.x大的GPU实例上。 该容器暴露了两种途径：一是POST/调用，它流出一个完整输入的响应；二是WS/调用双向流，它使一个连接在两个方向的许多信息之间保持开放。

以下图显示从客户端到模型的请求路径：

先决条件

在开始之前，你需要：

- AWS账户，允许创建 SageMaker AI端点；Agent Core runtime，具有Amazon Bedrock Agent Core的能力；AWS 身份和访问管理(IAM)角色；以及Amazon 弹性容器注册(Amazon ECR)寄存器，至少包括：Sagemaker: Create Endpoint, sagemaker:CreateModel, sagemaker:Invoke Endpoint With Bidifor Stream，以及Group:InvokeModel WithResponseStream 动作。

- 样品容器用已知的咨询(PYSEC-2026-2286，固定在火炬2.10.0中)贴上较老的PyTorch。 在任何无价值防伪用途之前审查并升级标针。

- AWS Cloud Development Kit(AWS CDK)v2，安装和靴子插入你的目标账户和AWS区域。

- 从一个宣布com.amazonaws.sagemaker.capabilities.bifo-streaming=真实的Docker标签的容器中部署的SageMaker AI 实时端点。

- Python 3.12或以后，uv，和Docker来构建容器图像(基础的HTTP/2客户端仅为Python 3.12和较新的预建轮船).

- AWS SDK for Python (Boto3)，支持引用 endpoint with bidirect stream.

- 熟悉WebSockets和Starlette或类似的ASGI框架。

- 来自GitHub寄存器的部署代码在此帖子中引用。

走过去

行走先部署端点和agent，然后通过客户端的修改工作，使递增流线工作结束。

部署终点和 agent

Kokoro-82M端点和agent运行时通过两个CLI工具进行部署，每个工具将自己的AWS CDK堆栈包裹：

语音信息部署 Kokoro-82M端点这包括建立基本IAM角色并将模型的权重同步到Amazon简单存储服务(Amazon S3)的基础和模型缓存堆栈。 端点堆栈然后构建容器图像，并创建SageMaker AI模型，端点配置，端点。 agent-infra 部署规定 Amazon Bedrock Agent Core 运行时托管 Strands agent，它呼唤你刚刚部署的终点，所以先部署终点。

第一步：处理 WebSocket 帧类型 SageMaker AI 实际发送

双向运输 SageMaker AI 将客户端的每条消息作为二进制的 WebSocket 框发送到您的容器。 写 Starlette WebSocket 处理器的常用方式是调用 receive text ()，它读取信件 ["text"] 和融资 KeyError 如果密钥不存在的话。 因为客户端发送的每一个信息都是作为二进制发送的，所以处理器在每次请求中都会失败。

失败通常被误诊。 一个将 str(e) 转发给客户端的通用例外处理器将 KeyError('text') 变成一个读取“ text” 的错误框架，它看起来像一个缺失的字段，而不是一个框架问题。 端点本身报告InService的全过程，因为它接受连接并响应每条消息，每次都有错误。

明确接受任意一种框架类型，而不是假设其中一种会显示：

在没有文本字段时，此解码字节为UTF-8，而不是假设一个框架类型。 通过发送二进制帧来测试它，因为这种失败模式会产生一个造型良好的错误帧和一个健康的终点状态，而不是明显的崩溃。

步骤2：在信件之间保持会话开放

在您的容器完成对一个消息的音频的合成后，本能是关闭会话，即HTTP循环结束的方式。 这是错误的。 在发送消息的音频和一个合成完整控制框架后，容器循环应该回到接收()时等待同一、仍然开放的会话上的下一个消息。 只有在客户端发送明确{'类型':"关闭"}消息时才关闭连接。

下图显示 1 会话顺序为两句，仅以结尾结束：

这样，您就可以将多个句子，一个接一个地发送到一个持久性的会话上，它们之间没有重联成本。 在一个答复后急切地关闭，将迫使每个句子重新连接起来，从而抵消大多数暂时性的好处。

第3步：以 agent 生成方式流放句子

随着会话行为到位，客户端对延迟问题的修补是一类将文本以两种形状到达的常态化的类：一个在前面已知的完整的字符串，或者一个活符流。 它以两种方式产生相同的输出：完整句子，每句完成后即可发送。

每个文本片段到达时调用一次种子()。 它返回任何新完成的句子，并保留线索文本。 调用 crosh () 一次，在输入结尾，释放任何部分文本，即使没有后面的提示。 这保持了一段流流，在一段时期到来前结束，不再撤销最后的句子。

这不是同时进行的综合。 每个双向处理器都是严格的顺序循环：接收消息，合成其音频，发送该音频，发送合成完整，然后接收下一个消息。 句子二的音频不会在句子一仍然传送时产生。 递增发送的收购更窄：第一句的合成开始于它存在的瞬间，而不是等待完整的响应。 对于一个在第一句和最后一句之间实时的多重判决反应，这就是整个暂时性的胜利。

我们通过一个现场终端验证了这一点发送了三次判决的意见所有三个块的到来都配有为每个块发送的文本的音频：

步骤 4：选择将音频带回客户端的传输方式

逐步将文本流到模型中，可以解决一半的问题。 另一半是把产生的音频发到通常是一个浏览器。 大部分agent和聊天前端已经通过Server-Sent Events(SSE)流出文本：单向，并得到了大多数浏览器的本土支持。 SSE是一种文本协议，不是用来搭载原始二进制有效载荷的。 在现有的SSE管道中添加语音的团队自然达到它已有的模式：基基64-编码音频，并在JSON事件中包裹。

Amazon Translatis证实编码质量并不不同

Base64将每个有效载荷都膨胀到接近预期的33%. 更微妙的成本是框架对齐。 SSE事件由空白行来划分，但内部的音频被划分为MP3帧。 这两种方案不一定与数据传输方式一致。 在一个测量的响应中，四个逻辑的MP3帧以6个送货块，两个结束中框到达。 一个消费者将每批交付当作一个完整的单元处理，在正常的健康交通中失败。 固定缓冲器仅在找到分隔符后才进入字节并作用：

今天用于模型造型腿的替代品完全跳过基座64和SSE框架。 它将原始二进制音频和JSON控制框架重叠在同一双向会话上，已经带有增量文本输入，并用一条规则将它们分解。

无法解码为 JSON 对象的块是音频，按收到时发送，在路径的任何地方没有编码步骤。 一个边缘大小写事项：原始的PCM音频可以合法地从字节0x7b开始，ASCII字符{，同一个字符打开一个JSON对象。 这个规则避免了将该音频错误分类，因为它要求整个块解析为有效的JSON，而不是它的第一个字节，一个PCM框架本质上永远不会做的事情。

注意：如果您的前端已经有一个正在工作的 SSE 文本管道，SSE 事件内的基础64 编码音频是实用的选择。 它花费了大约33%的额外有效载荷以及缓冲和重组步骤，但再利用你已经运行的基础设施。 若您正在构建新鲜的语音路径，或者说暂时性是优先，统一二进制和JSON方法既避免成本

第5步：将它放在一个真正的agent中

更充分，端到端的案件是一个活生生的，基因化的agent，一次形成其反应一个符号，在句子一存在之前无法知道句子四。 这个例子作为StraNDs agent运行，由Amazon Bedrock支持，部署到Amazon Bedrock Agent Core运行时，用第4步的图案调用Kokoro端点。 3条通信腿在这个设计中连接：浏览器-造型腿将Server-Sent Events带回呼叫者，agent的流线呼叫Amazon Bedrock进行文本生成，agent与Kokoro端点双向会话进行合成。

下图显示这三条交流腿，以及生成和合成之间的重叠之处：

请求的有效载荷搭载一个模式场，要么是"bidi"，要么是"batch"，发送到同一入口功能内的两个代码路径。 它们之间的基准测试完全改变一个变量：当合成开始。 在 run bidi 中，有两个任务同时运行。 一个排出贝德罗克的文本流并输入完成的句子到块。 另一个排水科科罗会议并释放音频块当他们到达。 两者都推入一个共享的队列，因此一个消费者以真实的到达顺序来输出事件：早期句子的音频可以到达呼叫者，而Bedrock仍然生成后一种。 run batch是故意相反的：它等待一个综合调用之前的完整响应，保持为一个公平的基线。

从agent容器返回呼叫器的腿向另一方向走去，确实使用了基数64，原因与第四步描述的不同。 Bedrock Agent CoreApp的切入点必须产生简单的，JSON-序列化的词典，它的SDK会自动转换成数据：{json}\n\n帧。 无法用原始的二进制框来形容。 这是JSON管理运行时间的结果，run bidi 和 run batch 都在这条腿上支付同样的成本，因此它们之间的测量差异是纯粹的时间：当第一个音频块被释放时。

测试和验证

我们通过发送 Sam 来隔离 TTS 运输本身对时间的贡献

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Artificial Intelligence

Stream low-latency speech from AI agents using Kokoro TTS on Amazon SageMaker AI

A customer asks an AI voice agent a question and waits. The agent already has an answer, it’s midway through generating a long, multi-paragraph response, but the customer hears nothing for five seconds. By the time an answer arrives, the customer is frustrated, confused, and no longer paying attention.

Text-to-speech integrations create exactly that gap. A one-sentence response might generate in 600 milliseconds, but the full response takes five seconds, and the integration waits for all of it before synthesis starts.

This post shows how to close that gap. We stream each sentence to a hosted Kokoro-82M text-to-speech (TTS) model on Amazon SageMaker AI as soon as an agent produces it, so synthesis begins on sentence one instead of the finished response. Getting that audio back to a browser needs its own solution too, so we also cover both transports for doing it and what each costs in practice.

Solution overview

This solution closes the gap by streaming synthesis alongside generation instead of waiting for it to finish. As the agent produces each sentence, that sentence streams immediately to a hosted Kokoro-82M text-to-speech model, so audio playback can start before the agent finishes speaking its full response. Kokoro-82M holds only 82 million parameters, small enough to run cheaply on a single mid-tier GPU instance. That keeps both latency and inference cost low for a voice agent handling real traffic.

Kokoro-82M runs in a custom container on a SageMaker AI real-time endpoint, on an ml.g5.xlarge GPU instance. The container exposes two routes: POST /invocations, which streams a response for one complete input, and WS /invocations-bidirectional-stream, which keeps one connection open across many messages in both directions.

The following diagram shows the request path from client to model:

Prerequisites

Before you begin, you need:

- An AWS account with permissions to create SageMaker AI endpoints; AgentCore runtime, a capability of Amazon Bedrock AgentCore; AWS Identity and Access Management (IAM) roles; and an Amazon Elastic Container Registry (Amazon ECR) repository, including at minimum the sagemaker:CreateEndpoint, sagemaker:CreateEndpointConfig, sagemaker:CreateModel, sagemaker:InvokeEndpointWithBidirectionalStream, and bedrock:InvokeModelWithResponseStream actions.

- The sample container pins an older PyTorch with a known advisory (PYSEC-2026-2286, fixed in torch 2.10.0). Review and upgrade the pin before any non-proof-of-value use.

- AWS Cloud Development Kit (AWS CDK) v2, installed and bootstrapped in your target account and AWS Region.

- A SageMaker AI real-time endpoint deployed from a container that declares the com.amazonaws.sagemaker.capabilities.bidirectional-streaming=true Docker label.

- Python 3.12 or later, uv, and Docker to build the container image (the underlying HTTP/2 client only ships pre-built wheels for Python 3.12 and newer).

- The AWS SDK for Python (Boto3) with support for invoke_endpoint_with_bidirectional_stream.

- Familiarity with WebSockets and Starlette or a comparable ASGI framework.

- The deployment code from the GitHub repository referenced in this post.

Walkthrough

The walkthrough first deploys the endpoint and agent, then works through the client-side changes that make incremental streaming work end to end.

Deploy the endpoint and agent

The Kokoro-82M endpoint and the agent runtime deploy through two CLI tools, each wrapping its own AWS CDK stacks:

speech-infra deploy provisions the Kokoro-82M endpoint. This includes the foundation and model-cache stacks that create the underlying IAM roles and sync the model’s weights into Amazon Simple Storage Service (Amazon S3). The endpoint stack then builds the container image and creates the SageMaker AI model, endpoint configuration, and endpoint. agent-infra deploy provisions the Amazon Bedrock AgentCore runtime hosting the Strands agent, which calls the endpoint you just deployed, so deploy the endpoint first.

Step 1: Handle the WebSocket frame type SageMaker AI actually sends

The bidirectional transport SageMaker AI uses delivers every piece of a client’s message to your container as a binary WebSocket frame. A common way to write a Starlette WebSocket handler is calling receive_text(), which reads message["text"] and raises KeyError if that key isn’t present. Because every message this client sends arrives as binary, that handler fails on every request.

The failure is commonly misdiagnosed. A generic exception handler that forwards str(e) to the client turns KeyError('text') into an error frame reading 'text', which looks like a missing field, not a framing issue. The endpoint itself reports InService throughout, since it accepts connections and responds to every message, with an error each time.

Accept either frame type explicitly instead of assuming one will show up:

This decodes bytes as UTF-8 when no text field is present, rather than assuming one frame type. Test it by sending a binary frame, because this failure mode produces a well-formed error frame and a healthy endpoint status, not an obvious crash.

Step 2: Keep the session open between messages

After your container finishes synthesizing one message’s audio, the instinct is to close the session, the way an HTTP cycle ends. That’s wrong here. After sending a message’s audio and a synthesis_complete control frame, the container loop should go back to waiting on receive() for the next message on the same, still-open session. Close the connection only when the client sends an explicit {"type": "close"} message.

The following diagram shows one session carrying two sentences in sequence, closing only at the end:

This way, you can send several sentences, one after another, on one persistent session, with no reconnection cost between them. Closing eagerly after one response would force a new connection per sentence, undoing most of the latency benefit.

Step 3: Stream sentences as the agent produces them

With the session behavior in place, the client-side fix for the latency problem is a class that normalizes text arriving in two shapes: a complete string known up front, or a live token stream. It produces the same output either way: complete sentences, ready to send as soon as each one is complete.

Call feed() once per fragment of text as it arrives. It returns any newly completed sentences, holding back trailing text. Call flush() once, at the end of input, to release whatever partial text remains even without trailing punctuation. This keeps a stream that ends before a period arrives from dropping a final sentence.

This isn’t concurrent synthesis. Every bidirectional handler is a strictly sequential loop: receive a message, synthesize its audio, send that audio, send synthesis_complete, then receive the next message. Sentence two’s audio isn’t generated while sentence one’s still transmits. What incremental sending buys is narrower: synthesis on sentence one begins the instant it exists, rather than waiting for the full response. For a multi-sentence response with real generation time between its first and last sentence, that’s the entire latency win.

We verified this against a live endpoint sending a three-sentence input over one session. All three chunks arrived with audio matching the text sent for each:

Step 4: Choose a transport for getting audio back to the client

Streaming text into the model incrementally solves half the problem. The other half is getting the resulting audio out to whatever renders it, commonly a browser. Most agent and chat frontends already stream text over Server-Sent Events (SSE): one-directional, and supported natively by most browsers. SSE is a text protocol, not built to carry a raw binary payload. A team adding voice to an existing SSE pipeline naturally reaches for the pattern it already has: base64-encode the audio and wrap it in a JSON event.

We measured the cost of that approach across 24 clips, confirmed with Amazon Transcribe that quality didn’t differ across encodings, and compared MP3 against AAC:

Base64 inflates every payload by close to the expected 33%. The subtler cost is frame alignment. SSE events are delimited by a blank line, but the audio inside is delimited into MP3 frames. The two schemes don’t necessarily align with how the transport chunks the data. In one measured response, four logical MP3 frames arrived as six delivery chunks, two ending mid-frame. A consumer that treats each delivery chunk as one complete unit fails on ordinary, healthy traffic. The fix buffers incoming bytes and acts only once a delimiter is found:

The alternative used today for the model-facing leg skips base64 and SSE framing entirely. It multiplexes raw binary audio and JSON control frames over the same bidirectional session already carrying incremental text input, and tells them apart with one rule.

A chunk that fails to decode as a JSON object is audio, forwarded as received, with no encoding step anywhere in the path. One edge case matters: raw PCM audio can legitimately begin with byte 0x7b, the ASCII character {, the same character that opens a JSON object. This rule avoids misclassifying that audio because it requires the entire chunk to parse as valid JSON, not its first byte alone, something a PCM frame will essentially never do.

Note: If your frontend already has a working SSE pipeline for text, base64-encoded audio inside an SSE event is the pragmatic choice. It costs roughly 33 percent extra payload and a buffer-and-reassemble step, but reuses infrastructure you already operate. If you’re building the voice path fresh, or latency is the priority, the unified binary and JSON approach avoids both costs and reuses the connection already carrying text.

Step 5: Put it together in a real agent

The fuller, end-to-end case is a live, generative agent composing its response one token at a time, with no way to know sentence four until sentence one exists. This example runs as a Strands agent, backed by Amazon Bedrock, deployed to Amazon Bedrock AgentCore runtime, calling the Kokoro endpoint with the pattern from Step 4. Three communication legs connect in this design: the browser-facing leg carrying Server-Sent Events back to the caller, the agent’s streaming call to Amazon Bedrock for text generation, and the agent’s bidirectional session with the Kokoro endpoint for synthesis.

The following diagram shows those three communication legs, and where the overlap between generation and synthesis happens:

The request payload carries a mode field, either "bidi" or "batch", dispatched to two code paths inside the same entrypoint function. A benchmark between them changes exactly one variable: when synthesis starts. In run_bidi, two tasks run concurrently. One drains the Bedrock text stream and feeds completed sentences into the chunker. The other drains the Kokoro session and emits audio chunks as they arrive. Both push into one shared queue, so a consumer yields events in true arrival order: audio for an early sentence can reach the caller while Bedrock still generates a later one. run_batch is the deliberate opposite: it waits for the full response before one synthesis call, kept as a fair baseline.

Going the other direction, the leg from the agent container back to the caller does use base64, for a different reason than Step 4 describes. BedrockAgentCoreApp’s entrypoint must yield plain, JSON-serializable dictionaries, which its SDK turns into data: {json}\n\n frames automatically. There’s no way to hand it a raw binary frame. That’s a consequence of the JSON-native managed runtime, not the choice Step 4 measured and moved away from. Both run_bidi and run_batch pay that identical cost on this leg, so the measured difference between them is purely timing: when the first audio chunk is emitted.

Testing and validation

We isolated the TTS transport’s own contribution to time to first audio by sending the sam

Auto-fetched and translated by AI Atlas. All rights belong to the original author.