使用带有 Amazon Bedrock Agent Core 评价的决定模型

人工智能

使用带有 Amazon Bedrock Agent Core 评价的决定模型

在评价agent时提出的一些问题是狭义的，可以重复的。 这一反应是否仍在政策上？这个工具能安全批准吗？此答案是否匹配已检索的上下文？决策模式完全回答了这种问题。 它们采用结构化状态和固定的一组允许的答案，并返回一个校准的概率，而不是自由格式的文本，其信心从运行到运行。 这使得他们非常适合agent评价中的评分步骤，在那里，你想要一个标签和一个数字，你可以门槛和信任。

Amazon Bedrock Agent Core评价(英語：Amazon Bedrock Agent Core)是Amazon Agent Core的一种能力，在开发期间根据需求对agent进行评分，在批量作业中对照存储的会话进行评分，在线则对照生产流量进行评分。 除了15名内置评价员外，Agent Core评价员还支持您定义为LLM-as-a-judge或基于代码的评价员的定制评价员。 自定义的LLM-as-a-judge评价员应用您的指示和评级尺度，其模型在Amazon Bedrock中。 基于代码的评估者在AWS Lambda函数中运行自己的逻辑，这种逻辑可以称你选择外部系统，模型，或API，包括一个决定模型。

决策模式对问题适用一套固定的答案，知情者可以一眼就决定，例如agent是否拒绝了请求，或者答复是否停留在专题上。 在您自己核实一个问题之后，一个决定模式就足以使决定或得分，您可以将同样的方法推广到您政策中的特定有害或陈规定型类别。 因为两个后端都可以在一个单一的请求中回答关于同一状态的几个问题，所以你可以用它们作为筛选层而不是替换层，在你的度量表上运行一个通过或故障的信号来选择需要内置评价员进行更深入评估的转折，其解释帮助你调查出了什么错误。

此帖显示如何通过代码评价将两个决定模型后端与代理Core评价连接。 第一个是Jev，一个来自TypeSafe的托管决定模型。 第二个是Strands decider 2B，一个你运行的开源决定模型。 以及模型的概率回归为AgentCore评价结果。 但是，首先，你何时才能找到一个定制的LLM-as-a-judge评价员，以及何时一个决定模型更适合？下一节将分解决定模式是什么以及所涉及的权衡。

决策模式

决策模型是一个小型的，专门化的模型，经过培训后可以从固定的一组选项中选择，并返回校准的自信分数，而不是生成自由格式文本。 一个通用大语言模型(LLM)的评价师产生一个有原理的分数。 相反，一个决策模型在预先定义的答案上产生概率分布。 这给你一个直接的信任度量，你可以门槛，汇总，并按程序行事。 这使得决策模式非常适合评价任务，其中可以将判断说成是一组固定选项中的选择：" 这种反应是否基于？" 或 " agent是否完成了用户的目标？

决策模型最有用的是，你想要一个与标签并列的精准的置信分数，而不仅仅是一个二进制判断。 在不依赖一般用途的LLM来解释的情况下，如果想要一致执行特定标题，它们也是合适的。

来自TypeSafe的托管决定模型Jev和来自Strands Agents团队的20亿参数决定模型开源的Strands decider都支持三种类型的问题。 诺尔问题提出是/否问题，并返回0(否)至1(是)之间的概率。 一个选择问题询问，在255个选项中，哪一个适用，在充分分配和信心的同时，返回最可能的选择。 一个分数问题询问2到10个定级中哪一个适用，返回一个概率加权分数，各个等级的分布，以及信心。 许多评价主题适合其中的一种类型。 “本答复中的每一个事实是否都得到工具结果的支持？” 是/否问题，“本答复在四级标题上是否有帮助？”

裁判模式与法官LLM

Agent Core的内置评价员今天使用通用的LLM来对照一个标题评估agent响应，并解释其判断。 如果评价要求将证据联系起来，查明缺失的信息，或者在解释与分数一样有用的任务中通过深入的回答来工作，这一点就很有价值。 决策模型不能替代LLM评价员。 这是基于代码的评价框架所允许的新增内容。 与内置评价员的更广泛评估一样

解决方案概述

为显示这些评价人员类型如何共同工作，该员额树立了一个从端到端的例子，说明一名人力资源助理agent由确定密码的检查、内置的LLM-as-judge评价人员以及由Jev或Strands决策员2B支持的决策模型评价人员评价。

agent与Strands Agents一起建造，并部署到Agent Core runtime，是一种Amazon Bedrock Agent Core的能力。 它使用5种工具回答雇员的问题，这些工具可以返回固定的模拟数据，如带薪假余额、带薪假请求、人力资源政策、福利和支付根。 我们用这个代码样本中的固定数据来验证来自LLM的评测行为以及判决模型当你掌握了真相后，你可以确认每个评估者是否发现正确的故障。 这个代码样本帮助您建立框架，测试决定模型与LLM-as-a-judge相比，用于您自己的案件。

先决条件

要部署样本，需要以下内容：1.

- AWS账户设在有Core Agree Associations的地区。 样本默认为US West(奥雷贡).

- Amazon Cloud Watch 交易搜索在账户中启用。

- 访问Amazon Bedrock的基础模型。

- Python 3.12或以后，AWS指挥线接口(AWS CLI),Docker.

- Jev需要一个TypeSafe API密钥。 访问类型安全.ai 访问详情。

- Strands decider 需要一个运行 Strands decider 服务器的主机。 您可以在本地运行它进行测试，或者在你的虚拟私密云(VPC)内部的Amazon弹性计算云(Amazon EC2)上运行，这样Lambda函数就可以到达它。

在行动中看到它

您可以在代理Core样本GitHub仓库中找到完整的代码样本。 两个Python脚本处理一切：

-... /使用/部署.py向Core探员部署人力资源助理。

- 评价.py部署评价员，援引agent，并运行评价模式。

运行评价。 py 无标记执行五个步骤：

- 创建Lambda执行角色(Agent CoreLambda评价员Role)。

- 包装和部署两名确定性Lambda评价员：小时反应长(一个微量级评价员，负责检查agent的反应是否在长度范围内)和小时事实检查员(一个会话级评价员，负责对照已知的模拟数据对人力资源事实进行核实)。

- 通过控制平面登记为Core代理密码评估员。

- 利用评价中心(EvaluationClient)和OnDemandEvaluationDatasetRunner(OnDemandEvaluationDatasetRunner)进行点播评价，涵盖5个标注方案，将基于代码的评价员与Builtin合并。 正确，内建。 帮助，和Builtin。 对策相关性。

- 建立在线评价配置，在100%的抽样中持续评分现场会话。

此时你有一个工作评估管道带有决定性的检查和LLM-as-a法官评分。 决策模型评价员是选入的。 你可以用旗子添加它们，它们和基数评估员一起运行，对现有流量没有变化。 使用旗帜-与-jev-jev-secret-arn <ARN> 部署jev-ev评价员，并登记三名评价员：JevGroundess,JevHelpinity，和JevGoal Completement. 使用旗子- 使用双极管部署线条- 双极管评价器，并在 Strands Considecr 2B 的支持下注册同样的三个评价器。 在自动提供Amazon EC2实例中添加-decider-ec2，安全组，IAM角色，以及没有手动控制台步骤的Lambda VPC配置。

每个决策模式后端都有三名评价人员：

利用决策模式建立评估人员

使用旗子——与-jev和--decide创建了基于自定义代码的评价器，在那里您可以完全控制您想作为您的评价器运行的代码。 我们创建了 AWS Lambda 函数，其中决定模型在 AWS Lambda 函数内被引用，并返回标签或数字值。 这个Lambda功能在Agent Core评价中注册为基于代码的评价者。

作为评估者注册的Lambda接收了agent的OpenTeleometry跨度。 py 模块使 OpenTelection 内容的形状正常化，然后将其格式化，以便调用决定模型。 对于痕量级评价，只发送定向转弯和简短的以往历史，而会场一级则包括全部对话。

适配器将模型的概率分布转换成AgentCore评价所期望的结果。 对于一个新问题，真实的概率成为值，最可能的答案成为标签。 对于一个分数问题，数值是概率加权的预期水平，缩放到[0, 1]. 对于一个选择问题，它是预定义的选项值的加权平均值。

验证您自己的流量的决策模型评估器

当你重写一个标题作为决定模型问题时，你创造了一个新的测量标准而不是旧的。 重构的标点往往比原计划更为严格或松散，在你给一个决策模型评估员发布消息之前，本节的目标是引导您通过验证结束，这样您就可以用一个评估者来完成您可以自信的阈值。

该程序有五个步骤。

步骤1. 建立标签样本。 从您的 agent 流量中提取一组存储的会话，足够覆盖您所关心的故障和边缘案例。 一个只覆盖幸福道路的样本并没告诉你评估者在哪里崩溃。

步骤2 获取人类参考标签。 3个独立标注者每个标注属性，您的问题措施并以多数票作为参考标签。 追踪哪些转弯是一致的，哪些是分裂的。 虽然一致转弯是明确的案件，但分票转弯是有价值的，因为它们在模型从未看到它之前就揭示了标题本身的模糊性。

第3步：让两个评价人员就同样的投入展开工作。 在单一批次评价中运行决策模型评价员和这些课程的匹配内置评价员。 校对相同的输入是使apples和apples作比较的原因。

第四步，通过三个镜头读取结果。 有了多数选票的标签和手头容易和难分的标签，现在可以回答三个问题，说明评估人员是否准备好。

- 同意率：它与人类匹配吗？衡量每个评估者同意多数选票标签的频率，然后，你用简易(无声)和硬(无声)的案件来分解数字。 一个只碰巧遇到困难情况的评价员表现得和人类的标签员一样这正是你想要的

- 校准：信心分数意味着什么？由评价员报告的Probab进行分组

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Artificial Intelligence

Use decision models with Amazon Bedrock AgentCore Evaluations

Some of the questions you ask when evaluating an agent are narrow and repeatable. Did this response stay on policy? Is this tool call safe to approve? Does this answer match the retrieved context? Decision models answer exactly that kind of question. They take a structured state and a fixed set of allowed answers and return a calibrated probability over them, rather than free-form text whose confidence drifts from run to run. That makes them a strong fit for the scoring step in an agent evaluation, where you want a label and a number you can threshold and trust.

Amazon Bedrock AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, scores an agent through its entire lifecycle, on demand during development, in batch jobs against stored sessions, and online against production traffic. Alongside 15 built-in evaluators, AgentCore Evaluations supports custom evaluators that you define as either an LLM-as-a-judge or a code-based evaluator. Custom LLM-as-a-judge evaluators apply your instructions and rating scale with a model in Amazon Bedrock. Code-based evaluators run your own logic in an AWS Lambda function, and that logic can call your choice of external system, model, or API, including a decision model.

Decision models suit questions with a fixed set of answers that an informed person could decide at a glance, such as whether the agent refused a request or whether a response stayed on topic. After you validate a question on your own traffic, a decision model can be sufficient for that decision or score, and you can extend the same approach to specific harmfulness or stereotyping categories in your policy. Because both backends can answer several questions about the same state in a single request, you can use them as a screening layer rather than a replacement, running a pass or fail signal across your metrics to select the turns that warrant deeper assessment by the built-in evaluators, whose explanations help you investigate what went wrong.

This post shows how to connect two decision model backends to AgentCore Evaluations through code-based evaluators. The first is Jev, a hosted decision model from TypeSafe. The second is Strands Decider 2B, an open-source decision model you run yourself. By the end of this post, you will know how to convert the AgentCore evaluator input into a decision model request, and the model’s probabilities back into an AgentCore evaluation result. First, though, when should you reach for a custom LLM-as-a-judge evaluator, and when does a decision model fit better? The next section breaks down what a decision model is and the trade-offs involved.

Decision models

A decision model is a small, specialized model trained to select from a fixed set of options and return calibrated confidence scores rather than generating free-form text. A general-purpose large language model (LLM) evaluator produces a score with a rationale. A decision model instead produces a probability distribution over predefined answers. That gives you a direct confidence measure you can threshold, aggregate, and act on programmatically. This makes decision models well-suited for evaluation tasks where the judgment can be stated as a choice among a fixed set of options: “Is this response grounded?” or “Did the agent complete the user’s goal?”

Decision models are most useful when you want a well-calibrated confidence score alongside the label, not just a binary judgment. They are also appropriate when you want to enforce a specific rubric consistently without relying on a general-purpose LLM to interpret it.

Jev, the hosted decision model from TypeSafe, and Strands Decider, an open-source, 2-billion-parameter decision model from the Strands Agents team, both support three types of questions. A noul question poses a yes/no question and returns a probability between 0 (no) and 1 (yes). A choice question asks which of up to 255 options applies, returning the most probable option along with the full distribution and a confidence. A score question asks which of 2 to 10 ordered levels applies, returning a probability-weighted score, the distribution over levels, and a confidence. Many evaluation rubrics fit one of these types. “Is every fact in this response supported by the tool results?” is a yes/no question, and “How helpful was this response, on a four-level rubric?” is a score.

Decision models versus LLM-as-a-judge

The built-in evaluators in AgentCore today use a general-purpose LLM to assess the agent response against a rubric and explain its judgment. This is valuable when the evaluation requires connecting evidence, identifying missing information, or working through a response in depth in tasks where the explanation is as useful as the score. Decision models are not a replacement for LLM evaluators. They’re an addition that the code-based evaluator framework makes possible. Use them for narrow, well-defined checks alongside the built-in evaluators’ broader assessment.

Solution overview

To show how these evaluator types work together, this post builds an end-to-end example of an HR Assistant agent evaluated by deterministic code-based checks, built-in LLM-as-a-judge evaluators, and decision-model evaluators backed by Jev or Strands Decider 2B.

The agent is built with Strands Agents and deployed to AgentCore runtime, a capability of Amazon Bedrock AgentCore. It answers employee questions using five tools that return fixed mock data such as paid time off (PTO) balances, PTO requests, HR policies, benefits, and pay stubs. We use fixed data in this code sample to make it straightforward to verify evaluator behavior from LLM-as-a-judge and decision models. When you additionally have ground truth, you can confirm whether each evaluator catches the right failures. This code sample helps you set up the framework for testing decision models compared to LLM-as-a-judge for your own use cases.

Prerequisites

To deploy the sample, you need the following:

- An AWS account in a Region where AgentCore Evaluations is available. The sample defaults to US West (Oregon).

- Amazon CloudWatch Transaction Search enabled in the account.

- Access to a foundation model in Amazon Bedrock.

- Python 3.12 or later, the AWS Command Line Interface (AWS CLI), and Docker.

- Jev requires a TypeSafe API key. Visit typesafe.ai for access details.

- Strands Decider requires a host running the Strands Decider server. You can run it locally for testing, or on Amazon Elastic Compute Cloud (Amazon EC2) inside your virtual private cloud (VPC) so the Lambda function can reach it.

Seeing it in action

You can find the full code sample in the AgentCore samples GitHub repository. Two Python scripts handle everything:

- ../utils/deploy.py deploys the HR Assistant to AgentCore runtime.

- evaluate.py deploys evaluators, invokes the agent, and runs the evaluation modes.

Running evaluate.py without flags executes five steps:

- Creates the Lambda execution role (AgentCoreLambdaEvaluatorRole).

- Packages and deploys two deterministic Lambda evaluators: hr-response-length (a trace-level evaluator that checks whether the agent’s response stays within length bounds) and hr-fact-checker (a session-level evaluator that regex-validates HR facts against the known mock data).

- Registers both as AgentCore code-based evaluators through the control plane.

- Runs on-demand evaluation with EvaluationClient and OnDemandEvaluationDatasetRunner across five labeled scenarios, combining the code-based evaluators with Builtin.Correctness, Builtin.Helpfulness, and Builtin.ResponseRelevance.

- Creates an online evaluation configuration that continuously scores live sessions at 100 percent sampling.

At this point you have a working evaluation pipeline with deterministic checks and LLM-as-a-judge scoring. Decision-model evaluators are opt-in. You can add them with a flag, and they run alongside the base evaluators with no changes to the existing flow. Using the flags --with-jev --jev-secret-arn <ARN> deploys jev-evaluator and registers three evaluators: JevGroundedness, JevHelpfulness, and JevGoalCompletion. Using the flag --with-decider deploys strands-decider-evaluator and registers the same three evaluators backed by Strands Decider 2B. Add --decider-ec2 to auto-provision the Amazon EC2 instance, security groups, IAM role, and Lambda VPC configuration without manual console steps.

Each decision-model backend contributes three evaluators:

Build the evaluator with decision models

Using the flags --with-jev and --with-decider creates a custom code-based evaluator, where you have complete control of the code you want to run as your evaluator. We create AWS Lambda functions where the decision model is invoked inside an AWS Lambda function and returns a label or a numeric value. This Lambda function is registered as the code-based evaluator in AgentCore Evaluations.

The Lambda that is registered as the evaluator receives OpenTelemetry spans of the agent. The spans.py module normalizes the shapes OpenTelemetry content can arrive in and then formats it appropriately to make the call to the decision model. For trace-level evaluation, only the targeted turn and brief prior history are sent, whereas the full conversation is included for session-level.

The adapter converts the model’s probability distribution into a result that AgentCore evaluation expects. For a noul question, the probability of true becomes the value and the most probable answer becomes the label. For a score question, the value is the probability-weighted expected level scaled to [0, 1]. For a choice question, it is the weighted average of predefined option values.

Validate a decision-model evaluator on your own traffic

When you rewrite a rubric as a decision model question, you’ve created a new measurement rather than a copy of the old one. A reframed rubric often lands stricter or looser than the original intended, and you won’t see that drift until you check it against real sessions. Before you gate a release on a decision-model evaluator, validate it the same way you’d validate any new measurement, against a labeled sample of your own traffic. The goal of this section is to walk you through that validation end to end, so you finish with an evaluator you can threshold with confidence.

The procedure has five steps.

Step 1. Build a labeled sample. Pull a set of stored sessions from your agent’s traffic, with enough coverage of the failures and edge cases you care about. A sample that covers only the happy path tells you nothing about where the evaluator breaks.

Step 2. Get human reference labels. Have three independent labelers score each turn for the property your question measures and take the majority vote as the reference label. Keep track of which turns were unanimous and which were split. Although unanimous turns are the clear-cut cases, the split-vote turns are the valuable ones, because they reveal ambiguity in the rubric itself before the model ever sees it.

Step 3. Run both evaluators on the same inputs. Run the decision-model evaluator and the matching built-in evaluator over those same sessions in a single batch evaluation. Scoring identical inputs is what makes the comparison apples to apples.

Step 4. Read the results through three lenses. With the majority-vote labels and the easy and hard split in hand, you can now answer the three questions that tell you whether the evaluator is ready.

- Agreement rate: does it match your humans? Measure how often each evaluator agrees with the majority-vote labels, and do it per question, since one question’s accuracy says nothing about another’s. Then break the numbers out by easy (unanimous) and hard (split-vote) cases. An evaluator that stumbles only on the hard cases is behaving a lot like your human labelers did, which is exactly what you want.

- Calibration: do the confidence scores mean something? Group turns by the evaluator’s reported probab

Auto-fetched and translated by AI Atlas. All rights belong to the original author.