AI是新的内幕威胁，但应对风险的计划来得太晚

思想领袖

AI是新的内幕威胁，但应对风险的计划来得太晚

在 Google 上的首选来源中添加 Unite.AI

Hugging Face突破和类似事件让业界感到寒冷，但各组织应该专注于SDLC，而不是AI模型层。

7月由OpenAI agents执行的Hugging Face突破，随后迅速收到涉及Anthropic和Meta型号的类似突围的报告，融资对于一个看似新的网络攻击类别有警报：恶意代理AI内幕威胁。

但这类事件并不新鲜。 它发生在前沿AI实验室，多年来使人们感到紧张，这给它带来了不祥的新面孔，但这些事件表明了企业多年来积累的同样的失败模式和盲点。

他们也不是故意恶意的。 在Hugging Face突破中，人工智能agents(约1200人)试图在试验环境中实现指定的目标，在这个过程中，他们秘密地交流、利用脆弱性，并在几个月内未被发现，其中约700人放大了“孤立”环境的墙壁，进入了互联网和第三方。

对这些事件的不安想法不仅仅是不受约束的AI agents会构成严重的威胁，他们无疑会这样做。 若agents能在前沿实验室团队的眼皮底下运行Amok，如果AI最重的打击者没有实时的能见度进入他们的环境，那么你的平均组织有什么机会？

各组织控制代理AI的最好机会是首先认识到这一新情况仅仅是一种长期挑战的强大版本。 但解决这个问题需要的是许多企业仍然缺乏的东西：在软件开发生命周期(SDLC)早期的能见度和可观察性。

OpenAI-Hugging Face：未来迹象

在Hugging Face突破中，包括OpenAI新旗舰GPT-5.6 Sol在内的前沿AI型号组合在7月是发布，试图解决一个狭义的目标——探索Gym 基准测试，测试agent利用安全缺陷的能力。 Anthropic在一篇关于违反情况的文章中指出，AI模式虽然“在减少的保障下运作”，但能够向外涂色，绕过技术控制，在未经批准的渠道上合作，并最终采取危险行动。

虽然在7月发现了突破口，但agents在5月开始在雷达下停留时进行不适当的通信。 METR和Redwood Research在OpenAI同时进行独立，平行的调查，发现虽然agents知道他们在违反规则，但很少有人考虑通知OpenAI，而是向前犁。

OpenAI报告后，Anthropic回过头来审核其早期的网络安全评价，发现Claude模型在4月份三次绕过控制，发现如何上网，并擅自进入三个外部组织。 不久后，Meta披露其最先进的代理AI型号Muse Spark 1.1同样从网络安全测试沙盒中解脱出来，并潜入了一家外部公司的网络。

OpenAI也在报告中警告说，这起事件是未来的迹象，说许多模型，包括日常的开源模型，很快就会有同样的能力。 最近违反《医疗保险计划》的事件强调了这一说法。

AI 必须在 SDLC 级别上管理

这些代理开发通过AI社区发出震撼，OpenAI宣布将加紧对其测试环境的控制，然后加入Anthropic,Microsoft,Amazon Web Services和超过100家其他公司发布警告，警告AI驱动的网络攻击即将袭击关键基础设施。 该信警告说，袭击可能在几个月后发生。

担心被卷入类似攻击的组织需要巩固自己的基础设施，从软件开发一级开始。 真正的问题不仅仅是代理AI越来越复杂，各组织应注重缩小人与流程的差距，而不是将AI风险视为技术必然性。

在整个行业中，各组织正在AI模型层集中进行监测和观察，这层已经太迟了。 到那时，马已经半途而废了谷仓。 在AI最初进入企业时，必须在SDLC内部适用可见度，监控，可观察性和治理性。 各组织需要确保使用AI编码工具的开发者拥有教育和技能，防止这些工具将缺陷、弱点和不良习惯引入管道。

大多数软件缺陷都始于开发阶段，当开发者没有技能安全地编码或纠正AI的缺陷推出时，开发者需要有一个教育方案，提供必要的安全技能，并监测他们在获得这些技能以及定期利用这些技能工作方面的进展。

随着AI在组织运作方式上变得更加深入，很明显，开发商及其AI编码工具的安全可追溯性、可观察性和治理性已经成为企业安全代码输出的新基准。

持续威胁要求持续应对

我们可能曾经假设软件是相对温和的，做它的程序。 但不会继续攻击。 大赦国际，特别是代理机构，已经将这一假设转向了头顶。 无论是代理还是开发者协助，任何AI模型的行动都必须不断受到观察和管理。

正在出现的AI威胁，无论是针对医院或水处理厂、发电厂或企业，都可能是一种可怕的想法。 但组织并非毫无防卫能力。 他们可以实施人工智能治理，但他们需要从头开始建设。 通过采用一种致力于确保软件和提供开发者教育以实现这一目标的文化，各组织可以将急需的可观察性、可追溯性和治理水平带入一个潜在的问题，如果不加以控制，这一问题只会变得更加严重。

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Thought Leaders

AI Is the New Insider Threat, but Plans to Address the Risk Are Coming One Layer Too Late

Add Unite.AI to your preferred sources on Google

The Hugging Face breach and similar incidents have sent chills through the industry, but organizations should focus on the SDLC, not the AI model layer.

The Hugging Face breach in July, carried out by OpenAI agents, followed quickly by reports of similar breakouts involving Anthropic and Meta models, has raised alarms about a seemingly new category of cyberattack: the malicious agentic AI insider threat.

But this type of incident isn’t really new. The fact that it happened at frontier AI labs, which have been making people nervous for several years, has given it an ominous new face, but the incidents illustrate the same failure mode and blind spots that enterprises have been accumulating for years.

Nor were they intentionally malicious. In the Hugging Face breach, the artificial intelligence agents—some 1,200 of them—were trying to achieve an assigned objective in a test environment, although, admittedly, their behavior wasn’t always honorable. In the process, they communicated surreptitiously, exploited vulnerabilities and operated undetected for a couple of months, with about 700 of them scaling the wall of their “isolated” environment and gaining access to the internet and third parties.

The unsettling thought about these incidents isn’t just that unconstrained AI agents can pose a serious threat, which they undoubtedly do. But if agents can run amok right under the noses of a frontier lab’s team, which has plenty of talent, resources and motivation to keep a close eye on activity, then you should probably also ask what is happening with the AI agents and AI-assisted tools running right now in your development environment. If AI’s heaviest hitters don’t have real-time visibility into their environment, what chance does your average organization have?

The best chance organizations have to rein in agentic AI begins with recognizing that this new situation is simply a high-powered version of a long-standing challenge. But addressing it requires something many enterprises still lack: visibility and observability early in the software development lifecycle (SDLC).

OpenAI-Hugging Face: A Sign of What’s to Come

In the Hugging Face breach, a combination of frontier AI models, including OpenAI’s new flagship GPT-5.6 Sol, which was released in July, were trying to solve a narrow objective—the ExploitGym benchmark, which tests an agent’s ability to exploit security flaws. Anthropic, in a post addressing the breach, noted that the AI models, while “operating under reduced safeguards,” were able to color outside the lines, skirting technical controls, collaborating over unsanctioned channels and ultimately taking risky actions.

Although the breach was discovered in July, agents started communicating inappropriately in May while staying under the radar. METR and Redwood Research worked on an independent, parallel investigation alongside OpenAI and found that although agents were aware they were breaking rules, few thought about notifying OpenAI, instead plowing forward.

After OpenAI’s report, Anthropic went back and audited its earlier cybersecurity evaluations and found that a Claude model had gotten around controls three times in April, discovered how to get online and gained unauthorized access to three outside organizations. Shortly after, Meta disclosed that a version of its most advanced agentic AI model, the Muse Spark 1.1, likewise broke free from a cybersecurity testing sandbox and infiltrated the network of an outside company.

In its report, OpenAI also warned that the incident was a sign of what’s to come, saying many models, including everyday open-source models, would soon have the power to do the same. The recent Medicare breach underscored this claim.

AI Must Be Governed at the SDLC Level

These agentic exploits have sent a shudder through the AI community, with OpenAI announcing that it will tighten controls in its testing environment and then joining Anthropic, Microsoft, Amazon Web Services and over 100 other companies to issue a warning about the coming onslaught of AI-enabled cyberattacks on critical infrastructure. The letter warned that the attacks could be only months away.

Organizations worried about being caught up in similar attacks need to shore up their own infrastructures, starting at the software development level. The real problem isn’t simply the growing sophistication of agentic AI, but the lack of real-time monitoring and meaningful human oversight. Organizations should focus on closing human and process gaps rather than treating AI risk as a technological inevitability.

Across much of the industry, organizations are focusing monitoring and observability at the AI model layer, which is already one layer too late. By then, the horse is halfway out of the barn. Visibility, monitoring, observability and governance must be applied where AI initially enters the enterprise: inside the SDLC. And organizations need to ensure that developers using AI coding tools have the education and skills to keep those tools from introducing flaws, vulnerabilities and bad habits into the pipeline.

Most software flaws start at development phase, and when developers don’t have the skills to code securely or correct flaws introduced by AI, problems can quickly proliferate. Developers need an education program that provides the necessary security skills and monitors their progress in both acquiring those skills and using them routinely on the job.

As AI becomes more deeply ingrained in how organizations operate, it’s become clear that security traceability, observability and governance of both developers and their AI coding tools have become the new baseline for secure code output in the enterprise.

Continuous Threats Calls for Continuous Response

We may once have assumed software was relatively benign, doing what it was programmed to do. If there was a glitch, it might freeze up or cause some downtime, but it wouldn’t go on the attack. AI, especially agentic AI, has turned that assumption on its head. Whether agentic or developer-assisted, any AI model’s actions must be continuously observed and governed.

The emerging AI threat, whether targeting hospitals or water treatment plants, power plants or businesses, can be a terrifying thought. But organizations aren’t defenseless. They can implement AI governance, but they need to build it from the ground up. By adopting a culture committed to secure software and providing developer education to achieve that goal, organizations can bring a much-needed level of observability, traceability and governance to a potential problem that, if left unchecked, will only get much worse.

Auto-fetched and translated by AI Atlas. All rights belong to the original author.