将数据库分级转换为 RL 奖励：使用 ThinkingBox 进行后训练的 Agents
这是第一个博客文章的续篇，数据库已不同意。 思考盒式训练现在可以在GitHub上使用。
我们通过比较最终数据库状态和预期结果，建立了"思维Box"来评价agent的工作。 然后我们用同样的可执行信号作为奖励训练任务之外 ThinkingBox -Bench。 Qwen3.8-27B的平均成功率从51.70%上升到60.78%，而所有20次尝试中完成的正确任务从38次上升到107次，共50次GRPO步骤。
图1. ThinkingBox将终端数据库状态的可执行检查转换为针对ThinkingBox-Bench以外任务的agents训练后奖励。
这是Microsoft和Hugging Face的联合博客，特别感谢来自Microsoft的Mirco Milletari和Xiang Jiang，来自Hugging Face的Sergio Paniego和我们前实习生Zhuochun Li(匹兹堡大学)共同撰写/审查工作。
我们建造了"思维盒" 捕捉agents 说它们完成的时候没有。
它的507项任务中，每个在 ThinkingBox-Bench (our 基准测试 data set) 将agent 手拿一个真正的后端和一套工具，让它工作，然后检查数据库中真正改变的东西。 在第一篇文章中，我们发现一个模型关闭客户的票解决了，而它的背后的承运人例外仍然开放。 9个有胜任能力的工具呼叫，一个错误的数据库行。
然后我们注意到我们建造的东西。 对于大多数任务，可执行的检查可以从最终数据库状态直接返回1或0. 这不仅仅是一个年级生。 这可以是奖励功能。+
因此，我们从基准测试以外的 ThinkingBox-format 任务中构建了一个训练池，并将它们可执行的成果作为奖励。 捕捉是选择哪些任务适合agent训练：大多数任务或者已经解决，或者对前面的模型来说是无望的，两者都教不了什么。
困难与政策相关：对目前的模式来说，只有在该模式学习之前，一项任务才艰巨。
我们称之为离线政策方面的困难。 我们估计每项任务的成功概率，一次是在培训前采用基本政策，然后在整个运行过程中固定选定的培训人才库。
训练后发生了什么
我们利用离线政策关系困难来建造训练池，然后将Qwen3.8-27B与GRPO进行了50步的微调。 为了概括，我们评价了507 基准测试任务的每个任务20次。 通过@1 衡量所有10 140次尝试的平均成功率。 通过@ 20 度量至少一次解决的任务份额。 观察20/20测量了每次有记录的尝试所完成的任务比例。
表1. Qwen3.8-27B在离线选择的训练池上50步GRPO前后的成绩。
培训大大提高了可重复取得的成功，至少一次的覆盖率略有下降。 其余部分解释我们如何选择练习任务，建立奖励循环，并衡量这种权衡。
一项任务有什么改进？旅行者要求房间降级，但公司政策首先需要经理的批准。 训练前，agent在20次尝试中的15次尝试中更改了预订，并未经批准而发放退款。 在RL 微调之后，它在所有20个中都保留了预订和支付状态。
图2. 一项企业旅馆订房任务，每个模型20次尝试。 基座Qwen3.8-27B型号在15次尝试中未经管理者批准而更改预订。 RL微调模型保存了全部20个的预订和支付状态；所有预期记录都匹配到19个。
目录
赶时间？头条结果在以上。 实施在运行培训堆。
基准测试在60秒内发现了什么
在他们留下的数据库记录上思考Box-Bench的成绩agents. 其507项任务中的每一项都开始一个真正的后端，手握agent一套MCP工具以及一个模拟的有私人背景的LLM用户，最后会运行对终端数据库状态的可执行检查。 检查与后端需要正确的记录相对应，而不是特定的工具调用。 这促使模型发现可能的新解决方案，而不是“重新发现”人支持的解决方案，这虽然是正确的，但可能不够理想。
在第一篇文章中，我们运行了18个模型，每一个基准测试20倍，并报告一个漏洞，标题精度隐藏。 Kimi-K3至少一次解决了93.89%的任务，但在所有有记录的20次尝试中只完成了68次(13.41%)的任务。 Claude 电话 Opus 5颠倒了它：覆盖面缩小，更加一致。
更大的或更好的模型没有关闭。 Claude Opus 5.5在传球@1上得分比Claude Opus 5高0.66个百分点，然而在所有20次尝试上却通过了完全相同数量的任务，241次。 头条准确度的提高没有增加可靠性。
如果您选择了触及生产数据库记录的工作模型，不应单独阅读 pass@ 1。 观察20/20 显示任务在每次尝试上都成功，这更接近生产可靠性的要求。
可执行的终极状态判决也是培训奖励。 在解开训练圈之前，这里是密封的基准测试上的变化。
结果：可信赖性一栏移动
在训练Qwen3.8-27B后，我们评价了它从未训练过的507任务基准测试上选定的第50步检查站。 507项任务代表了分布外测试集：他们的域，数据库，以及大多数MCP工具在RL训练中都没有遇到。
表2. 我们第一个博客表2的选定模型和我们针对不同域和总体的微调模型，按域分列的思维Box-Bench通过@1 (%). 观测到的20/20数字超过整个基准测试，有507项任务。 对20次重复审判的所有507项任务对每个模式进行评价。 粗体标记纵队队长；下划线标记跑步者。
标题结果为表1中观测到的20/20行。 一个27B模型最初在每次尝试上完成38(7.5%) ThinkingBox-Bench，现在在每次尝试上完成107(21.1%)，几乎是3次。 相比之下，观测到的Kimi-K3的20/20为68,GPT-5.4为128,GPT-6 Astra为231,Claude Opus 5.5为241.
平均成功率和重复率产生不同的订单。 在表2中，我们注意到GPT-5.6 Sol在传球@1上的得分略高于经RL微调的Qwen3.8-27B(61.91%对60.78%)，但对所有20次录制的尝试都通过了较少的任务：82对107. Claude Sonnet 4.6在传球@1上得分较低，达到59.19%，然而每次都通过几乎相同数量的任务，达到102. 光靠衡量都无法掌握整个情况。
观察到的模式与磨损是一致的：可重复的成功上升，而至少一次的覆盖率略有下降。 这是与先前的工作有关的。 一种解释是概率分布向反复运行的轨迹移动，而一些低概率路径却消失了。
总的来说，这一模式并没有扩大。 它变得更加可重复。 下一个问题是这些收益落到哪里。
收益落地之处
图3. 域级基础分数，RLFT分数，百分点变化。 绿色变化单元格显示增益，红色单元格显示下降；蓝色阴影表示每个公尺内的得分水平。
Pass@1和观察到的20/20在所有五个领域都上升。 Pass@20有好有坏：在新银行和咨询方面有所增加，但在零售、旅行和汽车保险方面却有所下降。 汽车保险的宽度损失最大(-7.0分)，零售业的20/20收益最大(+24.5分)。
哪些可依赖性成本
首个使用简单数学定价的帖子一致性：将全部20跑运动的估计费用除以模型每次都通过的任务数量。
每个可靠任务的成本 = 507x20项运动的估计费用
对经过微调的检查站的量度符号用法进行同样的成本计算：
表3. 基础和RL微调的估计评价成本和可靠性Qwen3.8-27B。 每一次成功尝试的费用将估计的运动费用总额(507x20)除以通过尝试的次数。
按每个可靠任务8.00美元计算，微调模型接近GPT-5.4(6.80美元),GPT-6 Astra(7.45美元)和Claude Opus 5.5(7.80美元)在伴奏比较中。
单试视图
图4 每一次成功尝试通过的价格成本@1.
图4. 金箭追踪Qwen3.8-27B基价位置到RLFT(钻石)：通过@1从51.70%上升到60.78%，而每次成功尝试的估计费用从0.177下降到0.139。 环绕的圈子标志着帕雷托成本前沿。
搬迁后将微调后的检查站设置在靠近GPT-5.6 Sol的地方，不过GPT-5.6 Sol仍然比较便宜，在通行证上稍高一些@1. 我们纸上定价的4个型号已经超出了规划范围
美元价值是一种比较服务指数，不是发票或云帐。 为经过微调的检查站提供服务与基地模式相同。 计算包括模型推论费用，不包括管理培训数据集的培训和政策预测费用。
这些结果告诉我们有什么变化。 接下来的章节解释训练循环，以及为什么选择正确的任务最为重要。
培训循环
图5 六个阶段，现在的背景是：候选任务输入，前三个阶段一度下线，最后三个阶段重复训练中每个小组。 对训练有素的检查站进行比对 " 思考箱-奔驰 " 的评价。
选拔一度，离线，对照基础政策模式运行，即在此情况下为Qwen3.8 27B；培训后使用已整理的固定池。 图5中破碎的琥珀箭头标志着我们没有走的路：在训练运行中重新测试困难。
每一次推出都是完整的剧集，而不是文本的完成。 它获得一个新鲜的后端，一个孤立的MCP会话，以及GPT-5 Chat作为模拟用户。 大多数奖励来自决定性的终端状态检查；少数狭义要求可以引用配置的法官(GPT-5 Chat).
培训和评价必须同样解释Qwen的产出。 这包括推理提取，工具调用，参数打字，渲染，以及孤立的任务状态。 否则，奖励与评估的奖励标准不同。 共享解释码在教练/core/tb interpret.py中。
参考运行时，我们用符号-意为的GRPO在187项任务上运行了全参数RL微调Qwen3.8-27B. 每次更新共产生18组8集，即144集，共24个H100s. 训练与参数同步运行，优化器卸载；亚当W使用1e-6的学习率和零重量衰减。 我们训练了50步
零变种组在训练期间被替换为DAPO型。 系统错误被单独处理：后端超时不是不正确的agent结果，所以受影响的群体被重新填充，而不是作为失败得分。
这解释了保留的任务是如何训练的。 剩下的问题是哪些任务进入了培训池。
离线政策 - 相对困难
在深入选择训练任务之前
思维盒-奔驰任务没有被列入RL训练池。 基准测试专用于评价。
ThinkingBox-Bench相对于RL训练池来说是分布外的：它的域，数据库，政策和大多数工具在RL训练中都没有遇到。 这两套仍然共享 ThinkingBox 框架，模拟用户机械，可执行的终端状态分级，一些通用工具，以及类似的任务风格。 这些共同要素限制了转让索赔的范围。
为什么选题要归结到一个关于GRPO的算术事实。 如果全部8个推出失败，奖励矢量为[0,0,0,0]. 如果全部8项都成功，则为[1,1,1,1,1]. 无论在哪种情况下，在集团内部集中，都不会产生任务回报优势，因为没有任何轨迹比另一轨迹好。
[0,0,0,1,1,1,1]等混合组提供对比的GRPO可以学习。 这种区别
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox
This is the follow up to our first blog post, The Agent Said It Was Done. The Database Disagreed. ThinkingBox-Training is now available on GitHub.
We built ThinkingBox to evaluate an agent's work by comparing the final database state with the expected outcome. We then used the same kind of executable signal as a reward on training tasks outside ThinkingBox-Bench. For Qwen3.8-27B, average success rose from 51.70% to 60.78%, while tasks completed correctly on all twenty attempts rose from 38 to 107 with 50 GRPO steps.
Figure 1. ThinkingBox turns executable checks of terminal database state into rewards for post-training agents on tasks outside ThinkingBox-Bench.
This is a joint blog by Microsoft and Hugging Face, special thanks to Mirco Milletari and Xiang Jiang from Microsoft, Sergio Paniego from Hugging Face and our former intern Zhuochun Li (University of Pittsburgh) for co-authoring/reviewing efforts.
We built ThinkingBox to catch agents that say they are done when they are not.
Each of its 507 tasks in ThinkingBox-Bench (our benchmark dataset) hands an agent a real backend and a set of tools, lets it work, then checks what really changed in the database. In the first post, we caught a model closing a customer's ticket as resolved while the carrier exception behind it was still open. Nine competent-looking tool calls, one wrong database row.
Then we noticed what we had built. For most tasks, an executable check can return 1 or 0 directly from the final database state. That is not just a grader. This can be a reward function.+
So we constructed a training pool from ThinkingBox-format tasks outside the benchmark and used their executable outcomes as rewards. The catch was choosing which tasks are suitable for agent training: most tasks are either already solved or hopeless for the model in front of you, and both teach it nothing.
Difficulty is policy-relative: a task is hard for the current model only until that model learns it.
We call this offline policy-relative difficulty. We estimated each task's success probability once using the base policy before training and kept the selected training pool fixed throughout the run.
What happened after training
We used offline policy-relative difficulty to build the training pool, then fine-tuned Qwen3.8-27B with GRPO for 50 steps. To recap, we evaluate each of the 507 benchmark tasks 20 times. pass@1 measures average success across all 10,140 attempts. pass@20 measures the share of tasks solved at least once. Observed 20/20 measures the share of tasks that pass every recorded attempt.
Table 1. Qwen3.8-27B results before and after 50 steps of GRPO on the offline-selected training pool.
Training sharply increased repeatable success, with a small decline in at-least-once coverage. The rest of the post explains how we selected practice tasks, built the reward loop, and measured that tradeoff.
What does that improvement look like on one task? A traveler requests a room downgrade, but company policy requires manager approval first. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts. After RL fine-tuning, it preserved the booking and payment state in all 20.
Figure 2. One corporate hotel-booking task, 20 attempts per model. The base Qwen3.8-27B model changed the booking without required manager approval in 15 attempts. The RL fine-tuned model preserved booking and payment state in all 20; all expected records matched in 19.
Contents
In a hurry? The headline result is above. The implementation is in Run the training stack.
What the benchmark found, in sixty seconds
ThinkingBox-Bench grades agents on the database records they leave behind. Each of its 507 tasks starts a real backend, hands the agent a set of MCP tools and a simulated LLM user with private context, and at the end runs executable checks against the terminal database state. Checks are against the backend record that needs to be correct, not the specific tool calls. This incentivizes the model to discover possibly new solution paths rather than “rediscover” the human-backed solution, that might be suboptimal albeit correct.
In our first post we ran eighteen models through that benchmark twenty times each and reported a gap that headline accuracy hides. Kimi-K3 solved 93.89% of tasks at least once but completed only 68 of 507 tasks (13.41%) on all twenty recorded attempts. Claude Opus 5 inverted it: narrower coverage, far more consistency.
And bigger or better model did not close it. Claude Opus 5.5 scored 0.66 percentage points higher than Claude Opus 5 on pass@1, yet passed exactly the same number of tasks, 241, on all twenty attempts. That gain in headline accuracy bought zero additional dependability.
If you are choosing a model for work that touches production database records, pass@1 should not be read alone. Observed 20/20 shows which tasks succeed on every attempt, which is closer to what production reliability requires.
The executable terminal-state verdict was also the training reward. Before unpacking the training loop, here is what changed on the sealed benchmark.
Results: the dependability column moved
After training Qwen3.8-27B, we evaluated the selected step-50 checkpoint on the 507-task benchmark it had never trained on. The 507 tasks represent an out-of-distribution test set: their domains, databases, and most MCP tools were not encountered during RL training.
Table 2. ThinkingBox-Bench pass@1 (%) by domain, for selected models from our first blog's Table 2 and our fine-tuned model for different domains and overall. Observed 20/20 numbers are over the entire benchmark with 507 tasks. Each model is evaluated on all 507 tasks for 20 repeated trials. Bold marks the column leader; underline marks the runner-up.
The headline result is the observed-20/20 row in Table 1. A 27B model that initially completed 38 (7.5%) of ThinkingBox-Bench on every attempt now completes 107 (21.1%) on every attempt, almost thrice. For comparison, observed 20/20 for Kimi-K3 is 68, GPT-5.4 with 128, GPT-6 Astra has 231, and Claude Opus 5.5 leads with 241.
Average success and repeatability produce different orderings. In Table 2, we notice that GPT-5.6 Sol scored slightly higher on pass@1 than the RL fine-tuned Qwen3.8-27B (61.91% versus 60.78%) but passed fewer tasks on all 20 recorded attempts: 82 versus 107. Claude Sonnet 4.6 scored lower on pass@1, at 59.19%, yet passed nearly the same number of tasks every time, at 102. Neither metric alone captures the full picture.
The observed pattern is consistent with sharpening: repeatable success rose while at-least-once coverage fell slightly. This is in-line with prior work. One explanation is that probability distribution moved toward trajectories that work repeatedly while some low-probability paths disappeared.
Overall, the model did not become broader. It became more repeatable. The next question is where those gains landed.
Where the gains landed
Figure 3. Domain-level base scores, RLFT scores, and percentage-point changes. Green change cells show gains and red cells show declines; blue shading indicates the score level within each metric.
Pass@1 and observed 20/20 rose in all five domains. Pass@20 was mixed: it increased in neobank and consulting, but declined in retail, travel, and auto insurance. Auto insurance had the largest breadth loss (−7.0 points), while retail had the largest observed-20/20 gain (+24.5 points).
What dependability costs
The first post priced consistency using simple math: divide the estimated cost of the full 20-run campaign by the number of tasks the model passed every single time.
Cost per dependable task = estimated cost of the 507 × 20 campaign ÷ tasks passing 20/20
Applying the same cost math to the fine-tuned checkpoint's measured token usage:
Table 3. Estimated evaluation cost and dependability for base and RL fine-tuned Qwen3.8-27B. Cost per successful attempt divides total estimated campaign cost (507 x 20) by the number of passing attempts.
At $8.00 per dependable task, the fine-tuned model came close to GPT-5.4 ($6.80), GPT-6 Astra ($7.45), and Claude Opus 5.5 ($7.80) in the companion comparison.
The single-attempt view
Figure 4 prices cost per successful attempt against pass@1.
Figure 4. The golden arrow traces Qwen3.8-27B base pareto cost position to the RLFT (diamond): pass@1 rises from 51.70% to 60.78% while estimated cost per successful attempt falls from $0.177 to $0.139. Ringed circles mark the Pareto cost frontier.
The move places the fine-tuned checkpoint close to GPT-5.6 Sol, although GPT-5.6 Sol remains cheaper and slightly higher on pass@1. Four models priced in our paper fall outside the plotted range.
The dollar values are a comparative serving index, not invoice or cloud bills. Serving the fine-tuned checkpoint would be the same as the base model. The calculation includes model inference cost, and excludes training run and policy-probe cost for curating the training dataset.
These results tell us what moved. The next sections explain the training loop and why selecting the right tasks matters the most.
The training loop
Figure 5. The six stages, now in context: candidate tasks feed in, the first three run once offline, the last three repeat for every group in the training run. The trained checkpoint is evaluated against ThinkingBox-Bench.
Selection runs once, offline, against the base policy model, i.e in this case Qwen3.8 27B; training then uses that fixed pool that has been curated. The dashed amber arrow in Figure 5 marks the path we did not take: re-probing difficulty during the training run.
Every rollout is a complete episode, not a text completion. It gets a fresh backend, an isolated MCP session, and GPT-5 Chat as the simulated user. Most rewards come from deterministic terminal-state checks; a minority of narrow rubric requirements can invoke the configured judge (GPT-5 Chat).
Training and evaluation must interpret Qwen output identically. That includes reasoning extraction, tool calls, argument typing, rendering, and isolated task state. Otherwise the reward measures a different agent from the one evaluated. The shared interpretation code is in trainer/core/tb_interpret.py.
For the reference run, we ran full-parameter RL fine-tuned Qwen3.8-27B on 187 tasks using token-mean GRPO. Each update generated 18 groups of 8 rollouts, or 144 episodes, across 24 H100s. Training ran synchronously with parameter and optimizer offload; AdamW used a 1e-6 learning rate and zero weight decay. We ran the training for 50 steps.
Zero-variance groups were replaced DAPO-style during training. System errors were handled separately: a backend timeout is not an incorrect agent outcome, so affected groups were refilled rather than scored as failures.
That explains how retained tasks were trained. The remaining question is which tasks entered the training pool.
Offline policy-relative difficulty
Before diving deep into the selection of training tasks, a disclaimer:
No ThinkingBox-Bench task was included in the RL training pool. The benchmark was reserved for evaluation.
ThinkingBox-Bench is out-of-distribution relative to the RL training pool: its domains, databases, policies, and most tools were not encountered during RL training. The two sets still share the ThinkingBox framework, simulated-user machinery, executable terminal-state grading, a few common tools, and a similar task style. These shared elements limit the scope of the transfer claim.
Why selection matters comes down to one arithmetic fact about GRPO. If all eight rollouts fail, the reward vector is [0, 0, 0, 0, 0, 0, 0, 0]. If all eight succeed, it is [1, 1, 1, 1, 1, 1, 1, 1]. In either case, centering within the group produces no task-reward advantage because no trajectory did better than another.++
A mixed group such as [0, 0, 0, 1, 1, 1, 1, 1] supplies the contrast GRPO can learn from. That distinction
Auto-fetched and translated by AI Atlas. All rights belong to the original author.