看着
- 利用代理RL:Microsoft 研究亚洲推出是一种训练范式，部署中使用的同一种agent的拉带直接参与强化学习，消除了在训练框架内重新实施agent的需要。
- 轻量级按设计：Agent闪电v1.0在大约3500行代码中提供完整的agent RL控制平面。
- 土著Kubernetes支持：agents作为标准Kubernetes工作在自管集群、云Kubernetes或当地基础设施上运行，不依赖付费的商业沙盒服务。
- 数据效率高的培训配方：在SWE-bench Verifeed上从41.8%到56.4%的融资 Z2Z3.5-9B端对端编码agent输油管Qwen3.5-9B，增速14.6个百分点，仅使用基于开源数据集的约6000个培训样本。
AI agents已经从单一模型演变成复杂的全存储系统，由模型，工具，和执行环境所构建。 它们的能力越来越依赖于agent拉带，从模型之外协调它们。 强化学习(RL)是AI系统通过尝试和错误学习的一种方法，以对其行为的奖惩为指导。 RL可以使这些agents更好，但大多数agent RL系统需要开发人员在训练框架内重新执行agent. 费用很高，这意味着agent接受训练并不是部署的agent。
为了解决这个问题，Microsoft研究亚洲的研究人员拥有推出 Harnesed Agentic RL训练范式和开源一个完全重建的Agent闪电v1.0(在新标签中打开). 与原版相比，Agent闪电，v1.0更强调保持轻量级，与真正的吊带结合，以及一个完整，可复制的agent RL训练管道。
Agent闪电v1.0是围绕Harnessed Agentic RL重建的，主要改进有：
——轻量级：整个框架约3500行代码。 Agent闪电v1.0在一个足够小和清晰的代码库中执行一个完整的Harnessed Agentic RL系统，以理解，修改，扩展。
- 真正的agent牵引装置的训练：agents通过Agent闪电v1.0中的大语言模型(LLM)代号到达模型，使现有的牵引装置代码保持不变。
- 本地Kubernetes支持：agents作为Kubernetes工作直接运行，没有外部商业沙盒服务。 自我管理的集群和地方基础设施都能够支持大规模推出。
- 一个完整的编码agent培训实例：在SWE-bench验证的Qwen3.5-9B 融资 Pass@1上建造的端对端管道从41.8%提高到56.4%，绝对收益14.6个百分点，只使用约6000个培训样本。
传统代理RL的限度
传统的代理RL假设训练框架拥有与环境的互动循环。 在ReAct风格的循环中，模型生成一个动作，环境返回一个观测，观察被附加在上下文中，模型生成下一个动作，因此整个推出的地图都排在一个连续的符号轨道上。 早期的RL系统如verl,AReaL，和粘液都是这样建造的，这意味着训练agent需要在RL框架内重建其循环。
真正的牵引力已经超过了这种假设。 编码agents，如迷你-SWE-agent,OpenHands,OpenCode,Claude Code，和Codex各带来各自的上下文管理，工具协议，执行逻辑，和依赖性，通用的agent系统也是如此。 重建一个用于训练的费用昂贵，重建的agent可能不再以部署的agent那样的行为。
兹Agent 闪电走不同路线。 它将LLM代理置于agent和模型之间。 agent继续和以前一样运行：简单地指向之前在Agent闪电中称为模型API的终点，训练框架可以观察和记录其模型调用。 在v1.0中，研究人员进一步将这个范式正式定义为Harnessed Agentic RL：在部署时使用的agent系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系(图1)。
培训中四个真正的挑战
Harnessed Agentic RL和传统Agentic RL的一个核心区别是，环境交互循环由agent的拉带处理，而不是训练框架。 训练系统只能观察一系列的LLM请求和响应对，所以一个单一的推出可能会被分成一个可变数量的训练样本。 这是带来四大挑战：
- 重新整理和样本合并：Harneses将上下文保留为文本，但RL培训需要推出过程中所抽样的代号ID. 通过聊天模板和指示器再次传递文本可以转换符号边界，因此相邻的呼叫不能总是被合并为一个样本。
- 优点计算：复试、 次试剂和上下文总和可以将一个推开分成几个样品。 直接在样本层面计算基线和优势，会导致产生更多样本的推出被反复统计，这改变了推出层面的原始统计关系。
- 损失常态化：通过抽样计数来估计损失，使产生更多样本的推广工作获得更多的权重。 由于样本计数往往只是拉带行为的产物，损失正常化也必须避免被它扭曲。
- 训练后端调度：样本计数和长度仅在使用完后才能知道，而GPU计数和数据/频率并行配置通常固定。 后端必须将可变的工作量映射成固定资源。
视频系列
用3 500行代码建造完整的agent RL控制平面
在系统设计中，Agent闪电v1.0将简单化作为其第一原理。 整个框架共约3500行代码，有三个核心组成部分：API Gateway，推出控制器，以及自定义的训练员(图2).
API网关存储推出，模型，和事件，并作为OpenAI兼容的LLM代理。 它将每一个模式的呼叫与它的推出联系起来，并记录培训所需的提示、反应和日志概率。 推出控制器开始并管理agent执行，或者作为本地进程，或者作为标准Kubernetes任务，使agent执行与训练员分开。 定制的训练员以verl为基础，建立推广，等待他们完成，收集样品，并通过一个样品适配器组装最后的训练样品。 因此，对于一个现有的agent牵引装置，简单地将模型端点指向Agent闪电代理通常足以快速连接到RL训练。
异位调 RL
推出时间在agents之间差异很大。 同步RL在一个批次中等待最慢的agent，让GPU闲置，同时完全同步的RL 融资利用，但需要单独的GPU池来展开和训练。 作为回应，Agent闪电v1.0 推出 Collocted Async RL，它让推出和模型更新共享同一套GPU.
一旦系统收集到足够的推出，更新开始：API Gateway暂停接受新请求，等待已进行中的请求完成，更新完成后恢复推出。 整个州过渡过程透明到外部的agent牵引。 在实验中，这种方法在同步RL上实现了大约2x端到端的加速，同时使用GPU比常规同步RL更少(图3).
在 Kubernetes 上运行 agents
收集足够的展开意味着一次运行许多agents，消耗大量的CPU，内存，和计算资源。 其他的Harnessed Agentic RL框架经常托管那些在商业沙盒服务如Modal Sandbox或E2B上的agents，成本随规模而迅速攀升。 相反，Agent闪电v1.0将它们作为标准Kubernetes工作运行，重新使用现有的自管理集群，云Kubernetes，或本地基础设施(图4). 现有计算资源得到更高效的利用，大规模推广成本降低，整个管道保持开源并可以再生产。
6 000个培训样本，14.6分业绩收益
为了测试这一方法，研究人员在SWE-smith、迷你SWE-agent和Qwen3.5-9B上建造了一条完整的管道，涵盖数据清理、环境建设、悬赏保障和RL培训。 这套培训有大约6 000个样本，不需要大规模计算。 RL训练单融资 Qwen3.5-9B在SWE-bench验证中从41.8%到56.4%，增速14.6个百分点。
编码agent实验进一步证实了早先对两个挑战的分析：优势计算和损失正常化。 与样本级处理相比，推出级优势与推出级正常化相结合，实现了更高的验证奖励，并在培训期间使政策 en更加稳定(图5)。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
At a glance
- Harnessed Agentic RL: Microsoft Research Asia introduces a training paradigm in which the same agent harness used in deployment participates directly in reinforcement learning, removing the need to reimplement the agent inside the training framework.
- Lightweight by design: Agent Lightning v1.0 delivers a complete agent RL control plane in roughly 3,500 lines of code.
- Native Kubernetes support: agents run as standard Kubernetes jobs on self-managed clusters, cloud Kubernetes, or local infrastructure, with no dependency on paid commercial sandbox services.
- Data-efficient training recipe: an end-to-end coding agent pipeline raised Qwen3.5-9B from 41.8% to 56.4% Pass@1 on SWE-bench Verified, a 14.6 percentage point gain, using only about 6,000 training samples based on open sourced dataset.
AI agents have evolved from single models to complex full-stack systems built from models, tools, and execution environments. Their capabilities increasingly depend on the agent harness that coordinates them from outside the model. Reinforcement learning (RL) is an approach where AI systems learn through trial and error, guided by rewards and penalties for their actions. RL can make those agents better, but most agent RL systems require developers to reimplement the agent inside the training framework. That is costly, and it means the agent being trained is not quite the agent that gets deployed.
To address this, researchers at Microsoft Research Asia have introduced the Harnessed Agentic RL training paradigm and open-sourced a fully rebuilt Agent Lightning v1.0 (opens in new tab). Compared with the original, Agent Lightning, v1.0 puts more emphasis on staying lightweight, on integrating with real harnesses, and on a complete, reproducible agent RL training pipeline.
Agent Lightning v1.0 was rebuilt around Harnessed Agentic RL, with key improvements:
- Lightweight: the entire framework is about 3,500 lines of code. Agent Lightning v1.0 implements a complete Harnessed Agentic RL system in a codebase that is small and clear enough to understand, modify, and extend.
- Training on a real agent harness: agents reach the model through the large language model (LLM) proxy in Agent Lightning v1.0, leaving existing harness code unchanged.
- Native Kubernetes support: agents run directly as Kubernetes jobs, without external commercial sandbox services. Self-managed clusters and local infrastructure alike can support rollouts at scale.
- A complete coding agent training example: an end-to-end pipeline built on Qwen3.5-9B raised Pass@1 on SWE-bench Verified from 41.8% to 56.4%, an absolute gain of 14.6 percentage points, using only about 6,000 training samples.
The limits of traditional agentic RL
Traditional agentic RL assumes the training framework owns the interaction loop with the environment. In a ReAct-style loop, the model generates an action, the environment returns an observation, the observation is appended to the context, and the model generates the next action, so the whole rollout maps onto one continuous token trajectory. Early RL systems such as verl, AReaL, and slime were built this way, which meant training an agent required rebuilding its loop inside the RL framework.
Real harnesses have outgrown that assumption. Coding agents such as mini-SWE-agent, OpenHands, OpenCode, Claude Code, and Codex each bring their own context management, tool protocols, execution logic, and dependencies, as do general-purpose agent systems. Rebuilding one for training is expensive, and the rebuilt agent may no longer behave in the same way as the deployed agent.
Agent Lightning takes a different route. It places an LLM proxy between the agent and the model. The agent continues to run as before: simply point the endpoint that previously called the model API at Agent Lightning, and the training framework can observe and record its model calls. In v1.0, the researchers go further and formally define this paradigm as Harnessed Agentic RL: whichever agent harness is used in deployment is the harness that takes part directly in reinforcement learning during training (Figure 1).
Four challenges in training with real harnesses
A core difference between Harnessed Agentic RL and traditional agentic RL is that the environment interaction loop is handled by the agent harness rather than the training framework. The training system can only observe a series of LLM request and response pairs, so a single rollout may be split into a variable number of training samples. This brings four key challenges:
- Retokenization and sample merging: Harnesses keep context as text, but RL training needs the token IDs sampled during the rollout. Passing text through the chat template and tokenizer again can shift token boundaries, so adjacent calls cannot always be merged into one sample.
- Advantage calculation: Retokenization, subagents and context summarization can split one rollout into several samples. Computing baselines and advantages directly at the sample level causes rollouts that produce more samples to be counted repeatedly, which alters the original statistical relationships at the rollout level.
- Loss normalization: Averaging loss by sample count gives more weight to rollouts that produce more samples. Since sample count is often just a product of harness behavior, loss normalization must also avoid being distorted by it.
- Training backend scheduling: Sample count and length are known only after the harness finishes, while GPU counts and data/tensor parallel configurations are usually fixed. The backend has to map a variable workload into fixed resources.
video series
Building a complete agent RL control plane in 3,500 lines of code
In system design, Agent Lightning v1.0 treats simplicity as its first principle. The entire framework is about 3,500 lines of code, with three core components: the API Gateway, the Rollout Controller, and the Customized Trainer (Figure 2).
The API gateway stores rollouts, models, and events, and serves as an OpenAI-compatible LLM proxy. It links every model call from the harness to its rollout and records the prompts, responses, and log probabilities that training needs. The rollout controller starts and manages agent execution, either as local processes or as standard Kubernetes jobs, keeping agent execution separate from the trainer. The customized trainer, built on verl, creates rollouts, waits for them to finish, collects samples, and assembles the final training samples through a sample adapter. As a result, for an existing agent harness, simply pointing the model endpoint at the Agent Lightning proxy is usually enough to connect quickly to RL training.
Collocated async RL
Rollout times vary widely across agents. Synchronous RL waits for the slowest agent in a batch and leaves GPUs idle, while fully asynchronous RL raises utilization but needs separate GPU pools for rollout and training. In response, Agent Lightning v1.0 introduces Collocated Async RL, which lets rollout and model updates share the same set of GPUs.
Once the system has collected enough rollouts, the update begins: the API Gateway pauses accepting new requests and waits for requests already in progress to finish, and rollout resumes after the update completes. The entire state transition is transparent to the external agent harness. In experiments, this approach achieved about a 2x end-to-end speedup over synchronous RL while using fewer GPUs than conventional asynchronous RL (Figure 3).
Running agents on Kubernetes
Collecting enough rollouts means running many agents at once, which consumes substantial CPU, memory, and compute resources. Other Harnessed Agentic RL frameworks often host those agents on commercial sandbox services such as Modal Sandbox or E2B, where cost climbs quickly with scale. Instead, Agent Lightning v1.0 runs them as standard Kubernetes jobs, reusing existing self-managed clusters, cloud Kubernetes, or local infrastructure (Figure 4). Existing compute resources are used more efficiently, large rollouts cost less, and the whole pipeline stays open source and reproducible.
6,000 training samples, a 14.6-point performance gain
To test the approach, researchers built a full pipeline on SWE-smith, mini-SWE-agent, and Qwen3.5-9B, covering data cleaning, environment construction, reward-hacking safeguards, and RL training. The training set holds about 6,000 samples and needs no large-scale compute. RL training alone raised Qwen3.5-9B from 41.8% to 56.4% on SWE-bench Verified, a gain of 14.6 percentage points.
The coding agent experiments further confirm the earlier analysis of two challenges: advantage calculation and loss normalization. Compared with sample-level handling, rollout-level advantage combined with rollout-level normalization achieves a higher validation reward and keeps policy entropy more stable during training (Figure 5).
Auto-fetched and translated by AI Atlas. All rights belong to the original author.