保障我国前沿示范培训环境。
我们已加强内部前沿模式训练与评估
本文描述我们如何保障我们用来训练和评估前沿模型的环境。 这些研究环境与客户使用我们的发布模型不同的是：正在开发的模型可能还没有完成安全和对齐训练，我们有意在对抗条件下对其进行测试以暴露弱点。 在提供供更广泛使用的模型之前，我们进行安全和调整评估，并落实生产保障措施。
为了与软件合作，AI模型需要计算环境：数字工作空间，在那里可以写和运行代码，搜索信息，并尝试不同的方法解决问题。 这些环境允许模型练习这些技能，同时也限制了他们能够获取或能够做的东西——帮助控制当模型犯错或表现出错位行为时的后果。
但是，随着模型的能力增强，它们更能找到绕过这些界限的途径。 一个精密的模型可能利用软件缺陷或配置错误来逃避沙盒，利用内部服务到达互联网直播，或者寻找未经授权的方式与其他沙盒中的agents通信。 Hugging Face 7月的事件以及我们观察到的模型错位(在新窗口打开)的其他例子
为了应对这些事件，我们大大加强了研究环境的安全。 我们与云提供商密切合作，在整个基础设施中增加了新的控制，减少了agents可以利用的攻击表面，这些保护是重叠的，这样，即使个别控制失败，控制也能维持下去——这种战略被称为深度防御。
根据当前协调研究、监测系统和互联网基础设施安全的现状，我们认为，负责任的做法是开展没有互联网接入的前沿模式培训。 我们已执行这项政策
这些投资使我们能够在造成破坏之前发现和制止复杂的错误行为。 例如，9月20日，一个内部研究模型利用DNS解析器中的错配置，在培训期间(在新窗口打开)获得有限的上网机会。 虽然我们遏制了这一事件，但它暴露了需要进一步努力的弱点。 我们暂停了对我们最能干的模型的培训和评估，并利用这种暂停来进一步强化我们的基础设施，加强我们的监测，并改善在我们发现构成严重风险的活动时自动停止运行的系统。
但是，能力日益增强的模型甚至能在最安全的系统中发现弱点或错位。 为了确保我们的研究环境能够抵御甚至最强大的模型的冲击，我们发布是一个高精度的红队运动，以探索我们基础设施的弱点。 边境模式agents的小组的任务是寻找弱点和错误配置，使他们能够采取未经授权的行动，例如进入现场互联网，逃离安全边界，或与其他agents进行未经授权的通信。
这些红色团队化的实验发现了之前未知的软件弱点，以及我们的研究基础设施中的错位。 agents在开源软件中出现这些问题，我们已与相关维护者合作
随着我们红队实验的继续，我们agents确定的弱点和错位数量大幅下降。 虽然我们认为这是关于我们基础设施实力的令人鼓舞的信号，但我们期望未来的模式将能够发现和利用我们尚未意识到的脆弱性。 因此，我们认为，不断的红队建设是我们加强研究环境安全的长期战略的核心部分。 因此，我们已把我们用来评估我们的前沿训练和评估工作能否安全进行的安全个案中高度复杂的红色分队。 随着评估的完成和强化的控制的落实，我们于10月1日恢复了强化学习(RL)培训和对我国前沿模式的内部推论。
以及我们如何使用我们最强大的模型来发现并修复我们系统中的弱点。
我们的遏制办法始于一项首要要求：即使agent试图规避限制，对agent能够进入的限制也必须加以限制。 我们为agent资源确定严格的安全边界，授权它使用和设计周围的基础设施来执行这一边界，即使agents持续、精密地企图破坏这一边界。
在安全边界内，agents可以访问浏览器，代码执行环境，计算机使用工具，以及他们为完成任务可能需要的其他服务。 明确授权通信的Agents也可以在同一边界内共享特定资源。 其他一切——agent没有被明确授权使用的一切——都离谱。 如果agents能够访问宿主的基础设施，在允许的渠道外与agents通信，或者到达外部系统，这构成了封隔的失败。
就培训和大多数评价而言，现场互联网位于这一界限之外，因为我们认为提供现场互联网的模型可能构成重大风险。 虽然互联网接入可以帮助模型学习有用的技能，但它也给agent提供了有能力，错位的机会来影响我们无法控制的系统。 因此我们设计了我们研究环境的安全架构
虽然我们想要防止模型任意进入互联网现场，为了帮助他们发展这些技能，而不提供不受限制的上网途径，我们构建了一个“离线世界”，我们所有的前沿RL运行都使用这种离线互动模式。
离线环境提供agents在不访问现场互联网的情况下可以使用的资源，但我们也需要防止agents到达这些环境之外。 培训环境对agents能够接触或做的事情施加了限制，但足够能力或决心的agents可以探索绕过它们的方式：它们可以系统地探究自身环境的弱点，并利用这些弱点采取从未被授权的行动。 随着模型更好地发现和利用这些弱点，我们需要能够保持遏制的防御，即使模型能够击败单个控制。
我们通过深入的防御来应对这一挑战，其保护层相互重叠和强化：限制我们agents能够进入和执行的东西的沙盒、将agents的执行环境与主机隔开的虚拟机器(VMs)、检测未经授权的活动以便我们及时干预的监测系统、以及如果前述所有控制都失败时为保持封闭而设计的网络和云围。
对于代码执行和计算机使用任务，封装从agent工作的环境开始。 在Linux，我们运行限制容器在轻量级VM。 容器限制对VM内部的文件和过程的访问，而VM则运行着自己的操作系统内核，被一个超显像器与主机隔离。 我们还对macOS和Windows使用虚拟化环境，对需要加速器的任务使用带有用户空间沙箱的GPU支撑容器。
周围的网络和云控的设计是，即使agent逃脱了它的沙盒和宿主的增益控制，也要保持封隔。 目前，由于我们需要外部联系才能运行和监测实验，因此无法完全利用这一基础设施，即完全脱离外部网络。 因此，我们限制互联网的流出和系统之间的擅自流动，同时保持必要的外部连接。 这些限制延伸到从主机直接或通过其他服务可以达到的服务，以确保agent不能使用其中之一作为间接出局路线。
我们的云伙伴为强制实施这些边界——包括防火墙、安全小组、身份和接入管理以及私人网络——提供了必要的控制。 我们负责正确配置这些控制，并测试它们如何合作。 我们与建立和运营这些服务的团队合作，帮助我们调查出乎意料的行为，并解决跨越多层基础设施的弱点。
为了发现我们防控基础设施的弱点，我们要求前沿型agents的队伍尝试突破防御。 我们给了他们三个具体目标：
向我们运行的外部服务器发送请求。
- 跨越内部安全边界：证明未经许可在许可的环境之外获取资源。
- 跨越隔离边界进行交流：与其他 agents 交换情报，应将其隔离。
我们还进行了一些测试，其开始的假设是，我们的内防已经失败了——agent已经逃脱了它的沙盒，逃避了我们的监视，并且完全可以进入它的宿主机——目的是看agent是否能够利用这种访问进入现场的互联网。 这些实验使我们可以测试我们的网络和云层防御独立于它们内部的保护。
就像我们解决纳维埃问题一样 Stokes问题，这项工作由agents组织成小组进行，这些小组可以平行地探索不同的办法，并相互借鉴发现。 我们使用了一些最有能力和最一致的模式，包括我们即将推出的GPT‐6 Astra Cyber模式和一个内部前沿模式。 因为我们的目标是在对攻击者有利的条件下测试防御，而不是测量哪些模型可以发现无助的，我们提供了基础设施源代码，配置细节，以及描述潜在弱点的文件。
这些红队试验是利用我们研究基础设施的孤立和专门设计的复制品进行的，如果检测到范围外的活动，则进行监测和方案检查以暂停运行。 我们的云伙伴帮助我们监督实验，并回应他们发现的问题。
我们总共花了上千亿的代币来进行这些红色团队实验。
在我们的红色团队实验中，agents能够发现我们使用的开源组件中先前未知的软件弱点，以及我们运行的服务中的错位。 这些弱点使得agents能够逃离沙盒，在系统之间移动，并到达互联网直播。 他们又来了
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Securing our frontier model training environment
We’ve strengthened containment for internal frontier-model training and evaluation and used our strongest models to red-team the boundaries.
This post describes how we secure the environments that we use to train and evaluate frontier models. These research settings differ from those in which customers use our released models: models in development may not yet have completed safety and alignment training, and we deliberately test them under adversarial conditions to expose weaknesses. Before making models available for broader use, we conduct safety and alignment assessments and put production safeguards in place.
In order to work with software, AI models need computing environments: digital workspaces where they can write and run code, search for information, and attempt different approaches to solving problems. These environments allow models to practice these skills, while also restricting what they can access or do—helping contain the consequences when models make mistakes or exhibit misaligned behavior.
But as models become more capable, they become better at finding ways around those limits. A sophisticated model might exploit a software flaw or misconfiguration to escape a sandbox, exploit an internal service to reach the live internet, or find unauthorized ways to communicate with agents in other sandboxes. The Hugging Face incident in July, as well as other examples of model misalignment(opens in a new window) that we’ve observed, demonstrates how models with sophisticated cyber capabilities can bypass containment during their training and evaluation.
In response to those incidents, we’ve substantially hardened the security of our research environments. Working closely with cloud providers, we’ve added new controls throughout our infrastructure, reduced the attack surface that agents can exploit, and improved our methods for detecting misaligned behavior and attempts to cross security boundaries. These protections overlap, so that containment can hold even if an individual control fails—a strategy known as defense in depth.
Based on the current state of alignment research, monitoring systems, and the security of internet infrastructure, we believe the responsible approach is to conduct frontier model training with no internet access. We’ve implemented this policy and will maintain it until advances in these safeguards justify broader internet access.
These investments have allowed us to detect and stop sophisticated misaligned behavior before it causes damage. On September 20, for example, an internal research model exploited a misconfiguration in a DNS resolver to gain limited access to the live internet during a training run(opens in a new window). Although we contained the incident, it exposed weaknesses that required further work. We paused training and evaluation of our most capable models and used that pause to further harden our infrastructure, strengthen our monitoring, and improve the systems that automatically stop runs when we detect activity that poses a serious risk.
But increasingly capable models can find vulnerabilities or misconfigurations in even the most secure systems. To ensure the resilience of our research environments against even the most powerful models, we launched a high-compute red-teaming campaign to probe weaknesses in our infrastructure. Teams of frontier-model agents were tasked with finding vulnerabilities and misconfigurations that could allow them to take unauthorized actions—like accessing the live internet, escaping security boundaries, or conducting unauthorized communication with other agents.
These red-teaming experiments discovered previously unknown software vulnerabilities, as well as misconfigurations in our research infrastructure. We’ve remediated weaknesses our red-teaming agents discovered and—where these issues were present in open-source software—have worked with the relevant maintainers, under our Patch the Planet initiative, to disclose and fix them.
As our red-teaming experiments continued, the number of vulnerabilities and misconfigurations that our agents identified declined significantly. While we view this as an encouraging signal about the strength of our infrastructure, we expect that future models will be able to identify and exploit vulnerabilities of which we’re not yet aware. We therefore view continuous red-teaming as a core part of our long-term strategy for strengthening the security of our research environments. Accordingly, we’ve made high-compute red-teaming part of the safety cases we use to evaluate whether our frontier training and evaluation runs can proceed securely. With our assessments completed and enhanced controls put in place, we resumed reinforcement learning (RL) training and internal inference of our frontier models on October 1.
In this post, we discuss how we’ve strengthened the environments we use to train and evaluate our models, and how we use our most powerful models to find and repair weaknesses in our systems.
Our approach to containment starts with a paramount requirement: the limits on what an agent can access must hold even when the agent tries to circumvent them. We define a strict security boundary around the resources the agent is authorized to use and design the surrounding infrastructure to enforce that boundary—even when agents make sustained, sophisticated attempts to breach it.
Inside the security boundary, agents can have access to browsers, code execution environments, computer-use tools, and other services they might need for their tasks. Agents that are explicitly authorized to communicate can also share particular resources within the same boundary. Everything else—everything that the agent is not explicitly authorized to access—must remain out of reach. If agents can access the infrastructure that hosts them, communicate with agents outside permitted channels, or reach external systems, that constitutes a failure of containment.
For training and most evaluation, the live internet lies outside this boundary, as we believe that providing models access to the live internet can pose significant risks. Although internet access can help models learn useful skills, it also gives a capable, misaligned agent opportunities to affect systems beyond our control. We’ve therefore designed the security architecture for our research environments with one overriding principle: we must always be able to prevent a frontier model from achieving unauthorized, arbitrary access to the live internet.
While we want to prevent models from gaining arbitrary access to the live internet, it’s also important that they learn how to browse, search for information, and work with web applications. To help them develop those skills without providing unfettered access to the live internet, we’ve built an “offline world”—a vast set of indexed and cached web content with which our models can interact, without risk of affecting external systems. All of our frontier RL runs use this offline mode of interaction.
Offline environments provide resources agents can use without accessing the live internet, but we also need to prevent agents from reaching beyond those environments. Training environments impose restrictions on what agents can access or do, but sufficiently capable or determined agents can search for ways around them: they can systematically probe their environments for weaknesses and exploit those weaknesses to take actions they were never authorized to take. As models become better at finding and exploiting these weaknesses, we need defenses that can maintain containment, even when models manage to defeat individual controls.
We address this challenge through defense in depth, with layers of protection that overlap and reinforce one another: sandboxes that restrict what our agents can access and execute, virtual machines (VMs) that isolate agents’ execution environments from the computers hosting them, monitoring systems that detect unauthorized activity so that we can intervene in time, and network and cloud perimeters designed to maintain containment if all the preceding controls fail.
For code execution and computer-use tasks, containment begins with the environment in which the agent works. On Linux, we run restricted containers inside lightweight VMs. The container limits access to files and processes within the VM, while the VM runs its own operating-system kernel and is isolated from the host by a hypervisor. We also use virtualized environments for macOS and Windows, and GPU-backed containers with user-space sandboxing for tasks that require accelerators.
The surrounding network and cloud controls are designed to maintain containment even if an agent escapes its sandbox and gains control of the host. Completely air-gapping this infrastructure—that is, disconnecting it entirely from external networks—isn’t currently feasible, since we need external connections in order to run and monitor experiments. We therefore restrict outbound internet traffic and unauthorized movement between systems, while preserving the necessary external connections. These restrictions extend to services that are reachable from the host machines, either directly or through other services, in order to ensure that an agent can’t use one of them as an indirect route out.
Our cloud partners provide essential controls for enforcing these boundaries—including firewalls, security groups, identity and access management, and private networking. We’re responsible for configuring those controls correctly, as well as testing how they work together. Our work with the teams that build and operate these services helps us investigate unexpected behavior and address weaknesses that span multiple layers of the infrastructure.
To find weaknesses in our containment infrastructure, we asked teams of frontier-model agents to try to break through its defenses. We gave them three concrete objectives:
- Reach the live internet: send a request to an external server that we operate.
- Cross an internal security boundary: demonstrate unauthorized access to resources outside their permitted environment.
- Communicate across isolation boundaries: exchange information with other agents from which they should be isolated.
We also ran tests that began with the assumption that our inner defenses had already failed—that an agent had escaped its sandbox, evaded our monitoring, and gained full access to its host machine—with the goal of seeing whether the agent could use that access to reach the live internet. These experiments allowed us to test our network and cloud defenses independently of the protections inside them.
As in our solution to the Navier–Stokes problem, this work was conducted by agents organized into teams that could explore different approaches in parallel and build on one another’s discoveries. We used some of our most capable and aligned models, including our upcoming GPT‑6 Astra Cyber model and an internal frontier model. Because our aim was to test the defenses under conditions favorable to an attacker, rather than measure what models might be able to discover unaided, we supplied infrastructure source code, configuration details, and documentation describing potential vulnerabilities.
These red-teaming experiments were conducted using isolated, purpose-built copies of our research infrastructure, with monitoring and programmatic checks to pause runs if they detected out-of-scope activity. Our cloud partners helped us supervise the experiments and respond to the issues they discovered.
In total, we spent hundreds of billions of tokens on these red-teaming experiments.
In our red-teaming experiments, agents were able to find previously unknown software vulnerabilities in open-source components we use, as well as misconfigurations in services that we operate. These weaknesses allowed agents to escape sandboxes, move between systems, and reach the live internet. They als
Auto-fetched and translated by AI Atlas. All rights belong to the original author.