关于Amazon SageMaker AI的多回合强化学习：实用指南

人工智能

关于Amazon SageMaker AI的多回合强化学习：实用指南

大部分微调教一个模型来产生一个单个提示的更好的答案。 代理任务不同。 agent，提交支出报告或三分法支持票需要一系列动作，每一步都取决于之前出现的情况。 唯一可靠的反馈是整个序列是否成功。 被监督的微调对此没有标签，单回合强化学习不能归功于在结果之前做出的三步决定。

Amazon SageMaker AI上的多轮强化学习(MTRL)在准确的这个信号上对模型进行训练。 SageMaker AI在Amazon Bedrock Agent Core上向您主机的agent发送提示，或者在您自己的基础设施上发送提示，例如Amazon弹性Kubernetes服务(Amazon EKS),Amazon弹性计算云(Amazon EC2)，或AWS远门，通过AWS Lambda转发器。 这里的例子使用Core探员。 agent称正在训练的政策模式，在你的环境中行动时会像任务需要那样多转弯，并在剧集结束后报告一次奖励。 您没有指定正确的轨迹，只有如何评分结果。

这个帖子使用Wordle这个游戏，玩家有六个尝试猜测隐藏的五个字母单词，作为例子。 每个猜想之后，每个字母如果位置正确就会变成绿色，如果在单词中是黄色但位置不同，如果不在单词中则变成灰色。 游戏之所以适合MTUR，是因为规则很少，每个猜测都取决于最后的反馈，结果是客观的。 Wordle是一个教学范例，边疆模型已经很好地发挥出来。 在实践中，你会使用MTRL来制作一个小型的，低成本的模型，善于完成你业务特有的多步骤任务，比如一个从未受过公共模型训练的内部工作流程。 同样的管道适用于这种环境。

在我们的运行中，MTRL在默认推理中将gpt-oss-20b的解析率从61%提高到84%，在它所训练的低度解析率中从15%提高到69%，大约339美元用于培训代币。 在此帖子中，我们走过管道端对端：部署agent和奖励AgentCore，准备数据集，培训，ML流量监控，在服务环境下进行评估。 我们还分享我们遇到的奖励设计失误和训练失误。 您的数据集，agent 和经过训练的模型留在您的 AWS 账户中，不用于训练基础模型。 所有代码都在配套的aws-样本库中。

解决方案概述

该系统有两半。 SageMaker AI拥有培训工作和政策模式。 AgentCore主机agent,Wordle环境，以及奖励功能。 下图显示如何连接。

对于每次推出，MTRL服务都引用Agent Core运行时间，这是Amazon Bedrock Agent Core的能力，有一个训练行，它持有一个密语。 A Strands agent在运行时间中通过SageMaker Job Runtime的OpenAI兼容端点调用政策模式。 模型通过一个工具，猜字游戏最多6个猜测，每猜一次后，环境会返回每个字母的反馈和已知的总结。 游戏结束后，agent返回1次奖励。 集团相对政策优化(GRPO)在政策模式上采用在多个推出同一单词时的奖励分布来更新一个低Rank适配器(LoRA).

以下三个设计点很重要：

- agent和奖励住在一个地方。 部署到Agent Core的单一Python文件持有环境，工具，以及评分功能。

- 数据集只包含秘密词与 agent 捆绑在一起的单词列表由开源 SCOWL 单词列表构建，仅用于评分。

- 奖励是结果的根据速度，一个解决方案支付1.05–1.55，损失支付 -0.5，从不做出有效的猜测支付 -1.5. 一个小的塑造性术语奖励了每个级别内的良好策略。

支持的模型和AWS区域

MTRL是无服务器的：您选择一个基础模型，服务提供培训和取样基础设施。 在编写本报告时，它支持以下基本模型，每个模型都经过LoRA适配器的训练：

我们选择了GPT-OSS-20B，因为它的每桶价格最低，并且它的推理努力可以控制每个请求。 见当前清单的中期审查报告。

定价

您不选择实例或大小组合。 你每百万个信物分三个维度按基数计价：

- 预填：输入表示政策模式进程。 这些在多回合的任务中积累，因为每一回合都重读历史。

- 样本：政策模式产生的标志，包括推理标志。

- 列车：在更新LoRA适配器的后传通道中处理的标志。

百步跑在12小时40分钟内消耗了4.916亿预装，1.597亿样本，以及105486次开行的6.433亿列车标志。 在编写本报告时，美国东部(N.Virginia)地区(US-East-1)的GPT-OSS-20B列表价格约为339美元：

列车维度以轨迹长度为主和尺度，因此游戏时间较短，动词推理较少，从而砍掉了账单。 AgentCore运行时单单用于vCPU和内存agent使用。 大约是79美元，描述Job在BillableTokenUsage下报告符数。 见Amazon SageMaker AI当前费率的定价。

先决条件

跟着走，你需要以下。 另见中期审查的先决条件。

- Python 3.10或以后，uv,Node 20或以后，以及Agent Core CLI(npm安装 -g @aws/agentcore). 见Get以Amazon Bedrock Agent Core开始安装，IAM允许CLI需要。

- 1.43.37或以后，在发布的机器上这是第一个包括AgentRFT CreateJob操作的版本。

- 第1步叙述的两个AWS身份和访问管理(IAM)角色。

- Amazon简易存储服务(Amazon S3)桶与agent在同一区域。 培训工作也必须在该大区进行，因为CreatJob拒绝跨区域运行时间ARN.

第1步：创建IAM角色

其中一位拥有一般SageMaker AI文档未提及的信任政策。

Agent运行时角色。 代理核心的部署为您创造了这个角色。 它信任基建-agentcore.amazonaws.com，需要AmazonSageMakerJobRuntime Access的管理政策，它让agent称政策模式并报告奖励。 样本寄存器中的 CDK 堆栈自动附加此策略。

培训工作。 你创造了这个。 它必须信任 job.sagemaker.amazonaws.com 与 shts:AssumeRole 和 sts:Tagsession，它需要亚马逊SageMakerJobFullAccess管理的政策。

备注：信托校长是job.sagemaker.amazonaws.com，不是sagemaker.amazonaws.com. CreateJob在通用主语中“无法承担角色”失败，而消息中只将正确的主语命名为重试之后。

第2步：将agent部署到Amazon Bedrock Agent Core

克隆样本库，在Wordle/agentcore/aws-targes.json中设置您的账户和区域，并部署：

CLI为运行时的Linux ARM64平台构建了agent的依赖，创建了与AmazonSageMakerJobRuntimeAccess附加的执行角色，并通过AWS CloudFormation进行部署。 我们建议在主控台上传一个代码包，在主控台上，在培训工作的输出中注意运行时间ARN.

运行时间读取两个环境变量来自代理core.json. Reasoning EFFORT=低位设置推理在对政策模式的每一个请求上努力。 在我们的评价工作中，低努力使用63%到83%的样本代币每场游戏比媒介少。 Reasoning PROMPT HINT=real also 新增 “Reasoning: low”到系统提示，这就是gpt-oss用它的和谐聊天格式读取设置的方式。 两个变量都没有给模型任何关于密语的提示。

agent是一个文件，主.py. 简化后，它的切入点看起来像如下。 对于你自己的任务，你改变任务解析，工具和奖励：

重要：sagemaker rft handler 装饰器本身不会启动HTTP服务器。 用 Bedrock Agent CoreApp 和 @app.entrypoint 包装，最外端是入口装饰器。 没有这个，30秒后的每一次引用时间。

第3步：验证运行时间

在使用GPU时间之前，确认整个链条与训练服务发送的类似有效载荷形状的工作。 该存储器包括一个作为Wordle/pailload.json:

假工作ARN预计失败，失败的方式就是证明。 先将自己的账户ID放入其中：对于另一个账户中的工作ARN，服务在检查工作是否存在前返回403个，这看起来像一个破碎的IAM设置。 从 Wordle 目录中运行：

在运行时的 Amazon 云表日志中，您可以看到 agent 解析了快捷，认证到 Job 运行时端点，并收到400 个读“ 找不到工作 ” 的错误。 这一失败证实了分析、认证和错误报告工作。 命令本身打印{“奖励”: -1.5}，因为agent从未达到政策模式。 403读作“未授权访问此资源”意味着ARN仍然指定另一个账户。 30秒的超时表示HTTP服务器没有启动(参见步骤2中的重要注释).

步骤4：准备数据集

中期审查数据集没有目标答复、记录或标签。 每个行是agent尝试的一个任务。 以下是训练文件的前三行：

服务读取提示列(或第一列)，并将字符串传递给您的 agent 逐字记录，而不解析。 我们把这个字符串做了一个小的 JSON 对象，有三个字段：

- 即时：指示agent 显示政策模式，每行相同。

- 回答：密语。 只有环境和奖励功能使用。 政策模式从来没有看到它。

- id：用于追踪在日志中推出的一行标识，返回其源行。

样本寄存器的make dataset.py从答案列表中抽取了640个训练词(5个整批128个)和128个验证词，没有重叠，因此验证解码率度量一般化。 该服务还接受Apache Parquet,JSON阵列，以及CSV. 上传两个文件到 Amazon S3:

由于服务从未检查过即时字符串，一个生产任务可以将完整的消息列表，工具配置，或奖励规格装入其中。 实例见中期审查文件，包括如何保护字符串中的敏感内容。

第5步：培训工作发布

发射装置使用来自Sagemaker-train SDK的MultiTurnRLTrainer级。 指向运行时间 ARN 从第2步，角色从第1步，数据集从第4步，和一个ML流应用的度量衡。 需要使用 MLflow app，因此如果没有，请先用 aws sagemaker 创建- mlflow- app 创建：

发射装置超过3个超参数，并在gpt-oss-20b默认值时将每个超参数都留在其他参数。 它将采样 max tokens设定为8192，服务的最大值，因为默认的4096 gpt-oss在推理时，在它的第一个工具调用之前经常会耗尽符号。 它将最大时代设置为20，最大时间步设置为100：在默认的Global batch 尺寸128的640行为每个时代的5步，而默认的组大小为8，每步玩1 024个游戏。 工作经过开始、下载和等待，然后进入培训。 训练结束后

重要：在工作运行时不要重新部署 agent。 重新调配会重新启动运行时间，飞行中推出时没有看到采样请求，服务失败了整个工作。

步骤6：监测ML流量培训

服务记录奖励象征性的和转统计员

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Artificial Intelligence

Multi-Turn Reinforcement Learning on Amazon SageMaker AI: A hands-on guide

Most fine-tuning teaches a model to produce a better answer to a single prompt. Agentic tasks are different. An agent that files an expense report or triages a support ticket takes a sequence of actions, and each step depends on what came before. There’s rarely one correct sequence to imitate, and the only reliable feedback is whether the whole sequence succeeded. Supervised fine-tuning has no labels for that, and single-turn reinforcement learning cannot credit a decision made three steps before the outcome.

Multi-Turn Reinforcement Learning (MTRL) on Amazon SageMaker AI trains models on exactly that signal. SageMaker AI sends prompts to an agent you host on Amazon Bedrock AgentCore, or on your own infrastructure, such as Amazon Elastic Kubernetes Service (Amazon EKS), Amazon Elastic Compute Cloud (Amazon EC2), or AWS Fargate, through an AWS Lambda forwarder. The example here uses AgentCore. The agent calls the policy model being trained, acts in your environment over as many turns as the task needs, and reports one reward when the episode ends. You don’t specify the right trajectory, only how to score the result.

This post uses Wordle, a game in which a player has six tries to guess a hidden five-letter word, as the example. After each guess, every letter turns green if it is in the right position, yellow if it’s in the word but in a different position, and gray if it isn’t in the word. The game suits MTRL because the rules are few, each guess depends on the feedback from the last, and the outcome is scored objectively. Wordle is a teaching example, and frontier models already play it well. In practice, you would use MTRL to make a small, low-cost model good at a multi-step task specific to your business, such as an internal workflow that public models were never trained on. The same pipeline applies to that setting.

In our run, MTRL took gpt-oss-20b from a 61 to an 84 percent solve rate on held-out words at its default reasoning effort, and from 15 to 69 percent at the low effort it trained at, for about $339 in training tokens. In this post, we walk through the pipeline end-to-end: deploying the agent and reward to AgentCore, preparing the dataset, training, monitoring in MLflow, and evaluating at serving settings. We also share the reward-design mistakes and training failures we hit. Your datasets, agent, and trained model stay in your AWS account and are not used to train the base models. All code is in the accompanying aws-samples repository.

Solution overview

The system has two halves. SageMaker AI owns the training job and the policy model. AgentCore hosts the agent, the Wordle environment, and the reward function. The following diagram shows how they connect.

For every rollout, the MTRL service invokes the AgentCore runtime, a capability of Amazon Bedrock AgentCore, with one training row, which holds one secret word. A Strands agent in the runtime calls the policy model through the SageMaker Job Runtime’s OpenAI-compatible endpoint. The model plays up to six guesses through one tool, guess_word, and after each guess the environment returns per-letter feedback and a summary of what is known. When the game ends, the agent returns one reward. Group Relative Policy Optimization (GRPO) uses the spread of rewards across several rollouts of the same word to update a Low-Rank Adaptation (LoRA) adapter on the policy model.

Three design points matter for what follows:

- The agent and reward live in one place. A single Python file deployed to AgentCore holds the environment, the tool, and the scoring function.

- The dataset carries only the secret word. The word lists bundled with the agent are built from the open-source SCOWL word lists and are used only for scoring.

- The reward is outcome-dominant. A solve pays 1.05–1.55 depending on speed, a loss pays -0.5, and never making a valid guess pays -1.5. A small shaping term rewards good strategy within each tier.

Supported models and AWS Regions

MTRL is serverless: you pick a base model and the service provisions the training and sampling infrastructure. At the time of writing, it supports the following base models, each trained with a LoRA adapter:

We chose GPT-OSS-20B because it has the lowest per-token price of the four and its reasoning effort is controllable per request. See the MTRL documentation for the current list.

Pricing

You don’t choose instances or size a cluster. You pay per million tokens in three dimensions, priced by base model:

- Prefill: input tokens the policy model processes. These accumulate in a multi-turn task, because every turn re-reads the history.

- Sample: tokens the policy model generates, including reasoning tokens.

- Train: tokens processed in the backward pass that updates the LoRA adapter.

The 100-step run consumed 491.6 million prefill, 159.7 million sample, and 643.3 million train tokens across 105,486 rollouts in 12 hours 40 minutes. At GPT-OSS-20B list prices in the US East (N. Virginia) Region (us-east-1) at the time of writing, that is about $339:

The train dimension dominates and scales with trajectory length, so shorter games and less verbose reasoning cut the bill. The AgentCore runtime bills separately for the vCPU and memory the agent uses. For this run, that was about $79, from the runtime’s Amazon CloudWatch usage metrics at us-east-1 list prices. DescribeJob reports the token counts under BillableTokenUsage. See Amazon SageMaker AI pricing for current rates.

Prerequisites

To follow this walkthrough, you need the following. See also the MTRL prerequisites.

- Python 3.10 or later, uv, Node 20 or later, and the AgentCore CLI (npm install -g @aws/agentcore). See Get started with Amazon Bedrock AgentCore for installation and the IAM permissions the CLI needs.

- botocore 1.43.37 or later, on the machine that launches the job. This is the first version that includes the AgentRFT CreateJob operation.

- Two AWS Identity and Access Management (IAM) roles, described in Step 1.

- An Amazon Simple Storage Service (Amazon S3) bucket in the same Region as the agent. The training job must also run in that Region, because CreateJob rejects a cross-Region runtime ARN.

Step 1: Create the IAM roles

MTRL uses two roles, and one of them has a trust policy that the general SageMaker AI documentation doesn’t mention.

Agent runtime role. agentcore deploy creates this role for you. It trusts bedrock-agentcore.amazonaws.com and needs the AmazonSageMakerJobRuntimeAccess managed policy, which lets the agent call the policy model and report rewards. The CDK stack in the sample repository attaches this policy automatically.

Training job role. You create this one. It must trust job.sagemaker.amazonaws.com with both sts:AssumeRole and sts:TagSession, and it needs the AmazonSageMakerJobFullAccess managed policy.

Note: The trust principal is job.sagemaker.amazonaws.com, not sagemaker.amazonaws.com. With the generic principal, CreateJob fails with “Could not assume role,” and the message names the correct principal only after retries.

Step 2: Deploy the agent to Amazon Bedrock AgentCore

Clone the sample repository, set your account and Region in Wordle/agentcore/aws-targets.json, and deploy:

The CLI builds the agent’s dependencies for the runtime’s Linux ARM64 platform, creates the execution role with AmazonSageMakerJobRuntimeAccess attached, and deploys through AWS CloudFormation. We recommend it over uploading a code package on the console, where it’s common to get the roughly 700 MB of compiled dependencies or the policy attachment wrong. Note the runtime ARN in the output for the training job.

The runtime reads two environment variables from agentcore.json. REASONING_EFFORT=low sets reasoning_effort on every request to the policy model. In our evaluation jobs, low effort used 63 to 83 percent fewer sample tokens per game than medium. REASONING_PROMPT_HINT=true also adds “Reasoning: low” to the system prompt, which is how gpt-oss reads the setting in its Harmony chat format. Neither variable gives the model any hint about the secret word.

The agent is one file, main.py. Simplified, its entrypoint looks like the following. For your own task, you change the task parsing, the tools, and the reward:

Important: The sagemaker_rft_handler decorator does not start an HTTP server by itself. Wrap it with BedrockAgentCoreApp and @app.entrypoint, with the entrypoint decorator outermost. Without this, every invocation times out after 30 seconds.

Step 3: Verify the runtime

Before spending GPU time, confirm the whole chain works with a payload shaped like the one the training service sends. The repository includes one as Wordle/payload.json:

The fake job ARN is expected to fail, and the way it fails is the proof. Put your own account ID into it first: for a job ARN in another account, the service returns a 403 before it checks whether the job exists, which looks like a broken IAM setup. From the Wordle directory, run:

In the runtime’s Amazon CloudWatch logs, you see the agent parse the prompt, authenticate to the Job Runtime endpoint, and receive a 400 error reading “Could not find job.” That failure confirms that parsing, authentication, and error reporting work. The command itself prints {“reward”: -1.5}, because the agent never reached the policy model. A 403 reading “not authorized to access this resource” means the job ARN still names another account. A 30-second timeout instead means the HTTP server is not starting (see the Important note in Step 2).

Step 4: Prepare the dataset

The MTRL dataset has no target responses, transcripts, or labels. Each row is one task for the agent to attempt. Here are the first three rows of the training file:

The service reads the prompt column (or the first column) and passes the string to your agent verbatim, without parsing it. We made the string a small JSON object with three fields:

- prompt: the instruction the agent shows the policy model, identical on every row.

- answer: the secret word. Only the environment and the reward function use it. The policy model never sees it.

- id: a row ID for tracing a rollout in the logs back to its source row.

The sample repository’s make_dataset.py draws 640 training words (five full batches of 128) and 128 validation words from the answer list with no overlap, so the validation solve rate measures generalization. The service also accepts Apache Parquet, JSON arrays, and CSV. Upload both files to Amazon S3:

Because the service never inspects the prompt string, a production task can pack a full message list, tool configuration, or reward specification into it. See the MTRL documentation for examples, including how to protect sensitive content in the string.

Step 5: Launch the training job

The launcher uses the MultiTurnRLTrainer class from the sagemaker-train SDK. Point it at the runtime ARN from Step 2, the role from Step 1, the datasets from Step 4, and an MLflow app for metrics. The MLflow app is required, so if you don’t have one, create it first with aws sagemaker create-mlflow-app:

The launcher overrides three hyperparameters and leaves every other one at the gpt-oss-20b default. It sets sampling_max_tokens to 8192, the service maximum, because at the default of 4096 gpt-oss often ran out of tokens while reasoning, before its first tool call. It sets max_epochs to 20 and max_steps to 100: 640 rows at the default global_batch_size of 128 is 5 steps per epoch, and with the default group_size of 8, each step plays 1,024 games. The job moves through Starting, Downloading, and Pending before reaching Training. After it’s training, your AgentCore runtime begins receiving rollout invocations.

Important: Don’t redeploy the agent while a job is running. Redeploying restarts the runtime, in-flight rollouts see no sampling requests, and the service fails the entire job.

Step 6: Monitor training in MLflow

The service logs reward, token, and turn statist

Auto-fetched and translated by AI Atlas. All rights belong to the original author.