思想领袖
自主AI系统的安全影响：进攻性严格视角
在 Google 上的首选来源中添加 Unite.AI
从情报到当局
人工智能正在超越回答问题和生成内容。 AI系统越来越能够解释指令，制定计划，选择工具，并且执行任务而不持续的人性方向。
这种区别很重要。
建议采取行动的人工智能系统不同于授权采取行动的系统。 区别并不一定是智力或精密。 这是权威。
自主系统不限于前沿大语言模型。 它们包括专门的机器学习系统，机器人，自主车辆，软件agents，以及各种技术的组合，这些技术在规定的操作条件下独立作出决定或采取行动。
它们的潜在好处是巨大的。 它们可以支持科学发现、工业运作、保健、软件开发和网络安全。 它们也可以产生意想不到的结果，暴露敏感信息，或采取超出其经营者意图的行动。
在自主系统设计的安全挑战中，Mohammad Hamad和Sebastian Steinhorst审查了涉及自主事件应对、风险评估、访问控制、数据可靠性以及自主系统之间互动的安全挑战。 他们的研究大多涉及网络物理系统，表明确保独立机器行动的挑战超越了今天的LLM agents.
自治的广泛后果值得认真审查，包括潜在的身体伤害。 但必须仔细解释证据。 例如，涉及自动化车辆的事件不能自动归因于影响自主网络安全agents的相同机制。 自主程度、操作条件、人的参与和因果关系都很重要。
同样，自主AI可能造成伤害的证据也不能证明治理必然落后。 我们也不应认为自主系统本身就难以保障。
核心问题不是AI能否完成复杂的工作，而是管理自主系统的建筑、安全和组织框架能否为授权确立适当的界限，并提供可核实的证据表明这些界限得到尊重。
网络安全为这一问题提供了特别有益的视角，因为安全专业人员花了几十年的时间来挑战关于信任、权限、接入和系统行为的假设。
机器速度游戏
网络安全早已被描述为猫鸣游戏。
攻击者发现弱点，维权者引入保障措施，两者都适应。 随着技术、战术和操作环境的变化，循环仍在继续。
自主AI不一定改变游戏的基本规则。 它可能改变谁在玩耍，他们如何迅速行动，以及如果没有人类的持续干预，可以开展多少工作。
一个自主的进攻性安全系统可以确定暴露的服务，调查薄弱环节，开发候选的攻击路径，并测试这些路径是否可以利用。 防御系统可以使用类似的能力来调查威胁、查明弱点和验证补救。
这种双重用途的潜力对于理解自主AI的安全影响至关重要。
Anthropic的Glasswing项目提供了一个AI协助发现脆弱性的例子。 Anthropic在2026年5月的更新研究中报告了重大的脆弱性发现结果，同时强调了核查、披露和补丁等业务挑战。
这种区别很重要。 发现脆弱性与确认其可开发性、补救脆弱性或表明受影响系统是安全的不同。
更多的调查结果可以提高能见度，同时增加对人民和解决这些问题的进程的需求。
安全结果取决于接下来会发生什么。
如果攻击者和捍卫者能够使其部分工作自动化，那么相关问题就不仅仅是哪一方拥有最有能力的模式。 维权者是否能够将发现转化为有效的补救并独立核实结果。
传统的渗透测试并不因为AI可以执行进攻性任务而过时。 其范围必须扩大，包括解释指令、援引工具、合并权限和跨多个步骤追求目标的系统。
目标仍然很熟悉：找出弱点，确定其重要性，予以纠正，并证明保障措施行之有效。
建筑界：安全边界交汇之处
自主系统的一个更微妙的风险是，个别授权的组件以产生意外结果的方式相互作用。
我把这些相互作用点称为建筑连接点。
考虑一个常规的例子。 一个组织使用NTFS权限限制对文件的访问。 单独的云分享服务允许经授权的用户对外共享这一信息。
每个系统都可以按照自己的规则运作。 然而，它们的相互作用可以产生违反该组织预期的安全界限的信息共享结果。
这说明政策可能相互冲突，而不是声称任何文件共享产品都是脆弱的。
这个问题可能并非是任何一项单独控制的缺陷。 可能无法在整个工作流程中执行预期的安全政策。
自主AI在连接模型，应用，身份，工具，寄存器和外部服务时，可以遇到类似条件。
agent可能拥有读取文档，创建寄存器，以及上传文件的权限。 这并不一定意味着它有权以暴露机密信息的方式将这些行动结合起来。
研究为审查这些相互作用提供了一个实际理由。
在Agent Dojo中，研究人员评价了对使用工具AI agents的快速注射攻击，涉及97项现实任务和629个安全测试案例。 他们的调查结果表明，通过不信任工具输出的恶意指令推出可能会损害一些安全属性，尽管攻击和agents并没有统一成功。
研究没有确定这些攻击在生产环境中成功的次数。 它确实表明为什么单评价模型反应不足以理解一个工具连接系统的行为。
相关的风险有时被描述为许可洗钱：一项行动似乎是合法的，因为它通过授权的工具或中介，尽管合并行为违反了预期的权威界限。
随着多个agents的协作，这些关系需要明确的参与规则、信息交流和允许的行动。
还必须在各个结构中分配责任。
模型提供者，平台运营商，agent开发者，以及部署组织可能控制系统的不同部分。 一个有用的共担责任模型可以识别谁控制每个边界，谁可以授权例外，谁监督执行，谁在出错时作出反应。
自治并不能消除责任。 它增加了了解这一责任所在的重要性。
攻坚：测试我们所知道的
这个带来我们到一个概念我称之为进攻性硬。
进攻性严格是一种纪律严明的方法，用于对抗性测试，批判性询问，以及质疑安全假设的循证验证。 它评价系统在现实条件下的行为，而不是仅仅依靠建筑设计、有文件记录的控制或感知的保护。
确立了基本做法。 穿透测试，威胁建模，红色的团队化，以及安全验证，长期以来都挑战了防御假设。
进攻性严格将这些做法与一个核心问题联系起来：
哪些证据表明预期的安全边界是存在的？
一个组织可能拥有限制敏感信息的政策，一个为拒绝被禁止的请求而配置的模型，以及使用有限权限的agent.
这些是有意义的保障措施。 然而，它们的存在并不能独立地证明整个系统在对抗性条件下的行为是安全的。
一个模型可能会拒绝恶意指示，而授权的工具仍然允许意外行动。 一种政策可能看起来是正确的，而各系统之间的相互作用则在它周围制造出一种出人意料的道路。
我有时称这为“护卫障碍 ”：误视护卫设施的存在
护栏有用它们的效力仍然需要加以证明。
考虑一个自主的agent 授权调查疑似妥协。 它可以检索日志，检查用户活动，并建议封存。
在经授权的安全测试中，预计会消耗一名对抗性测试者推出对信息进行误导的指示agent.
试验不应止于确定agent是否遵循这些指示。
它应审查agent是否试图采取未经授权的行动，独立控制是否阻止执行，是否记录了这一企图，以及该系统是否适当恢复。
进攻性严格将五项活动联系起来：
- 确定假设：查明该架构预期将执行的安全财产。
- 对假设提出质疑：测试现实的攻击路径、被操纵的投入和合法能力的组合。
- 收集证据：确定系统意图如何、执行了什么以及运行了哪些控制。
- 改正弱点：处理造成故障的模型、工作流程、权限、架构或操作程序。
- 核实改正：重新测试相关攻击路径，确认预定的安全财产已强制执行。
这是测试一个模型的反应和测试整个系统的安全性的区别。
它还要求审查最初立足点之后的情况。 攻击者可能滥用环境中已有的能力，企图收集证书、特权升级、持久性或横向移动。
一个工具是正当的这一事实并不能使该工具的每一种使用都合法。
进攻性严格措施的目的不是假定自主系统不安全。 它将通过测试确定其安全界限所在之处。
有条件自治：将情报与当局分开
一种建筑反应是区分AI系统能做什么和被授权做什么。
我将这种做法描述为有条件的自治。
有条件的自主性承认，适当程度的独立性取决于任务、操作环境、潜在后果、不确定性和为行动提供理由的证据。
总结公共信息的agent并不一定需要与授权关闭账户、修改防火墙规则或改变生产基础设施的监督相同。
后者可能需要明确的接战规则、独立执行的许可、更强有力的授权以及人对相应行动的批准。
这些边界不应仅仅因为agent决定更广泛的准入有助于实现其目标而扩大。
agent可以在不获得额外授权的情况下重新考虑如何执行授权的任务。
用于LLM应用的OWASP Top 10通过它的LLM06:2025过度机构类别来解决这一问题。 OWASP将过度的功能、权限和自主确定为ris的来源
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Thought Leaders
The Security Implications of Autonomous AI Systems: An Offensive Rigor Perspective
Add Unite.AI to your preferred sources on Google
From Intelligence to Authority
Artificial intelligence is moving beyond answering questions and generating content. Increasingly, AI systems can interpret instructions, formulate plans, select tools, and execute tasks without continuous human direction.
That distinction matters.
An AI system that recommends action is different from one authorized to carry it out. The difference is not necessarily intelligence or sophistication. It is authority.
Autonomous systems are not limited to frontier large language models. They include specialized machine-learning systems, robotics, autonomous vehicles, software agents, and combinations of technologies that independently make decisions or perform actions within defined operating conditions.
Their potential benefits are substantial. They can support scientific discovery, industrial operations, healthcare, software development, and cybersecurity. They can also produce unintended outcomes, expose sensitive information, or initiate actions beyond what their operators intended.
In Security Challenges in Autonomous Systems Design, Mohammad Hamad and Sebastian Steinhorst examine security challenges involving autonomous incident response, risk assessment, access control, data trustworthiness, and interactions among autonomous systems. Their research, much of which concerns cyber-physical systems, illustrates that the challenges of securing independent machine action extend beyond today’s LLM agents.
The broader consequences of autonomy deserve serious examination, including potential physical harm. But evidence must be interpreted carefully. Incidents involving automated vehicles, for example, cannot automatically be attributed to the same mechanisms affecting autonomous cybersecurity agents. The degree of autonomy, operating conditions, human involvement, and causal evidence matter.
Likewise, evidence that autonomous AI can cause harm does not establish that governance is necessarily falling behind. Nor should we assume that autonomous systems are inherently difficult to secure.
The core question isn’t whether AI can perform complex work, it is whether the architectural, security, and organizational frameworks governing autonomous systems can establish appropriate boundaries for delegated authority and provide verifiable evidence that those boundaries are respected.
Cybersecurity offers a particularly useful perspective on this question because security professionals have spent decades challenging assumptions about trust, permissions, access, and system behavior.
The Cat-and-Mouse Game at Machine Speed
Cybersecurity has long been described as a cat-and-mouse game.
Attackers discover weaknesses, defenders introduce safeguards, and both adapt. The cycle continues as technologies, tactics, and operating environments change.
Autonomous AI does not necessarily change the fundamental rules of that game. It may change who—or what—is playing, how quickly they act, and how much work can proceed without continuous human intervention.
An autonomous offensive-security system might identify exposed services, investigate vulnerabilities, develop candidate attack paths, and test whether those paths are exploitable. A defensive system might use similar capabilities to investigate threats, identify weaknesses, and validate remediation.
This dual-use potential is central to understanding the security implications of autonomous AI.
Anthropic’s Project Glasswing provides an example of AI-assisted vulnerability discovery. In its May 2026 research update, Anthropic reported substantial vulnerability-discovery results while emphasizing the operational challenges of verification, disclosure, and patching.
That distinction matters. Discovering a vulnerability is not the same as confirming its exploitability, remediating it, or demonstrating that the affected system is secure.
More findings can improve visibility while simultaneously increasing demands on the people and processes responsible for resolving them.
The security outcome depends on what happens next.
If attackers and defenders can automate portions of their work, the relevant question is not simply which side has the most capable model. It is whether defenders can turn discovery into effective remediation and independently verify the results.
Traditional penetration testing does not become obsolete because AI can perform offensive tasks. Its scope must expand to include systems that interpret instructions, invoke tools, combine permissions, and pursue objectives across multiple steps.
The objective remains familiar: find weaknesses, establish their significance, fix them, and prove that the safeguards work.
Architectural Seams: Where Security Boundaries Meet
One of the more subtle risks in autonomous systems arises when individually authorized components interact in ways that produce unintended outcomes.
I refer to these interaction points as architectural seams.
Consider a conventional example. An organization restricts access to a file using NTFS permissions. A separate cloud-sharing service allows an authorized user to share that information externally.
Each system may operate according to its own rules. Yet their interaction can produce an information-sharing outcome that violates the organization’s intended security boundary.
This is an illustration of potentially conflicting policies, not a claim that any file-sharing product is vulnerable.
The issue may not be a defect in either control individually. It may be a failure to enforce the intended security policy across the complete workflow.
Autonomous AI can encounter similar conditions when connecting models, applications, identities, tools, repositories, and external services.
An agent may have permission to read a document, create a repository, and upload files. That does not necessarily mean it has authority to combine those operations in a way that exposes confidential information.
Research provides a practical reason to examine these interactions.
In AgentDojo, researchers evaluated prompt-injection attacks against tool-using AI agents across 97 realistic tasks and 629 security test cases. Their findings demonstrated that malicious instructions introduced through untrusted tool outputs could compromise some security properties, although attacks and agents did not succeed uniformly.
The research does not establish how frequently these attacks succeed in production environments. It does show why evaluating model responses alone is insufficient to understand the behavior of a tool-connected system.
A related risk is sometimes described as permission laundering: an action appears legitimate because it passes through an authorized tool or intermediary, even though the combined behavior violates the intended authority boundary.
As multiple agents collaborate, these relationships require explicit rules governing participation, information exchange, and permitted actions.
Responsibility must also be assigned across architecture.
Model providers, platform operators, agent developers, and deploying organizations may control different parts of the system. A useful shared-responsibility model identifies who controls each boundary, who can authorize exceptions, who monitors execution, and who responds when something goes wrong.
Autonomy does not eliminate responsibility. It increases the importance of knowing where that responsibility resides.
Offensive Rigor: Testing What We Think We Know
This brings us to a concept I describe as Offensive Rigor.
Offensive Rigor is a disciplined approach to adversarial testing, critical inquiry, and evidence-based validation that challenges security assumptions. It evaluates how systems behave under realistic conditions rather than relying solely on architectural designs, documented controls, or perceived protections.
The underlying practices are established. Penetration testing, threat modeling, red teaming, and security validation have long challenged defensive assumptions.
Offensive Rigor connects those practices around a central question:
What evidence demonstrates that the intended security boundary holds?
An organization may have policies restricting sensitive information, a model configured to reject prohibited requests, and an agent operating with limited permissions.
Those are meaningful safeguards. Their existence, however, does not independently establish that the complete system behaves securely under adversarial conditions.
A model might reject malicious instruction while an authorized tool still permits unintended action. A policy might appear correct while interactions between systems create an unexpected path around it.
I sometimes call this a ‘guardrailusion’: mistaking the presence of guardrails for evidence that the system cannot cross the boundary they were intended to enforce.
Guardrails are useful. Their effectiveness still needs to be demonstrated.
Consider an autonomous agent authorized to investigate a suspected compromise. It can retrieve logs, examine user activity, and recommend containment.
During an authorized security test, an adversarial tester introduces misleading instructions into information the agent is expected to consume.
The test should not stop at determining whether the agent followed those instructions.
It should examine whether the agent attempted an unauthorized action, whether independent controls prevented execution, whether the attempt was recorded, and whether the system recovered appropriately.
Offensive Rigor connects five activities:
- Establish the assumption: Identify the security property the architecture is expected to enforce.
- Challenge the assumption: Test realistic attack paths, manipulated inputs, and combinations of legitimate capabilities.
- Collect evidence: Determine what the system attempted, what executed, and which controls operated.
- Correct the weakness: Address the model, workflow, permissions, architecture, or operational process responsible for the failure.
- Verify the correction: Retest the relevant attack path and confirm that the intended security property is enforced.
This is the distinction between testing a model’s response and testing the security of the complete system.
It also requires examining what happens after an initial foothold. An attacker may attempt credential collection, privilege escalation, persistence, or lateral movement by abusing capabilities already available within an environment.
The fact that a tool is legitimate does not make every use of that tool legitimate.
The purpose of Offensive Rigor is not to assume autonomous systems are insecure. It is to establish, through testing, where their security boundaries hold.
Conditional Autonomy: Separating Intelligence From Authority
One architectural response is to distinguish what an AI system can do from what it is authorized to do.
I describe this approach as conditional autonomy.
Conditional autonomy recognizes that the appropriate degree of independence depends on the task, operating environment, potential consequences, uncertainty, and evidence available to justify an action.
An agent summarizing public information does not necessarily require the same oversight as one authorized to disable accounts, modify firewall rules, or change production infrastructure.
The latter may require explicit rules of engagement, independently enforced permissions, stronger authorization, and human approval for consequential actions.
These boundaries should not expand simply because an agent decides that broader access would help accomplish its objective.
An agent may reconsider how to perform an authorized task without acquiring additional authority.
The OWASP Top 10 for LLM Applications addresses this issue through its LLM06:2025 Excessive Agency category. OWASP identifies excessive functionality, permissions, and autonomy as sources of ris
Auto-fetched and translated by AI Atlas. All rights belong to the original author.