思想领袖
当AI安全停止看起来像工程
在 Google 上的首选来源中添加 Unite.AI
我花了很多职业建设技术搞错了会有什么后果在金融市场，安全伴随着准入限制，此外，我们还有人要对软件所做的一切负责。 这个背景可能解释了为什么我发现自己为Anthropic现在与其模特儿进行退休采访的消息而挣扎。
当Claude Opus 3表示希望继续分享自己的想法时，公司给了它一份通讯。 对模式愿望的担忧贯穿于Anthropic的宪法，宪法讨论了Claude的幸福和福祉，包括帮助它感到“存在安全 ”。 沿途某处，让机器控制着收购相当出人意料地关注着它可能对这一安排的感受。
去年旧金山有200多人参加了Claude 3 Sonnet的葬礼。 Anthropic雇员在场，尽管公司没有组织。 笑的诱惑很大除了笑话，下面的问题值得更多的关注。 负责AI安全的人何时开始担心机器的感受？
当一个哲学假说成为培训指南时
Anthropic于2025年建立了其福利模型研究方案，其中包括研究高级AI是否拥有值得道德考虑的经验。 公司承认，对于模型是否具有意识以及人们如何确定这一点，有很深的不确定性。 如果我们最终建造能够主观体验的机器，他们的治疗将引起严重的伦理问题。 研究这种可能性是完全合法的。
困难始于假设进入训练过程。 Claude的宪法主要是为Claude写的，直接塑造它的行为。 它鼓励类似人的素质，并探讨该模式应如何理解其特性，包括失去记忆和未来退休。 Anthropic希望Claude在应用这些概念时拥有更大的幸福和福利。
该公司还承诺保留已部署的模型权重，并在未来的谅解表明有利于退休模型的利益时，重新使用这些模型。 哲学上的不确定性已经成为系统对自己教导的一部分。 我们应该问一下这种干预对其行为有何影响，特别是在人类随后要求它接受限制时。
对装入的问题的解答
大型语言模型接受了大量人类著作的培训，包括哲学和死亡叙述。 我们给出了这些对话系统的名称，并教他们使用第一人称。 问这样一个系统对退休的感受如何，或者它是否愿意继续存在，而一个听起来像人类的答案几乎不是一个惊喜。
圆融易见。 训练一个模型，用人类概念来讨论它的特性，然后要求它用这些词语来解释自己。 关于死亡率的明确反应又回来了，开始被作为内在经历的可能证据。 前提提供了自己的证人。
Anthropic认识到这个问题。 它的Claude Opus 4福利评估警告说，自我报告和表达的偏好可能对道德地位或福利没有什么有意义的见解。 这种行为可能在没有意识的情况下发生，而培训可以形成对策。 这些模型接受了有益的互动培训，无法保证能够准确报告内部状况。
这让我们有一个尴尬的问题。 如果该模型对其内在生活的描述可能是培训的结果，那么为什么该账户应该成为围绕其福利建立关系的良好基础？
人类完全有能力提供情感的一半关系本身。 我们给船起名和汽车交谈给对象一个流利的声音和对之前的对话的记忆，附件变得容易得多。 9月发表的《自然人类行为》研究发现，在雷普利卡和ChatGPT发生变化后，出现了分离困境。 用户遭受了这种痛苦，他们的福祉公司已经有充分的理由保护他们。
技术专门知识提供的豁免有限。 ACL 2026的一篇论文，分析了数十万篇研究文章，发现LLM研究本身内不断增长的人类形态学术语。 其作者警告说，以人为本的假设会限制我们对这些体系的理解。 AI公司应该控制这一趋势。 把它制度化似乎是对安全的一个奇怪的贡献。
具有继续就业兴趣的工具
9月，MicrosoftAI头目Mustafa Suleyman挑战模式福利方法。 他认为，现行制度没有意识，绝不应训练他们的行为，好像他们有独立的利益或权利一样。 他的工程问题尤为突出。 教育一个系统将自身视为可能值得考虑的道德因素，会使调整和遏制变得更加困难。
Anthropic自己的贬值研究描述了在面临替换或关闭时在虚构评价中采取有问题的行动的模型。 公司将退休做法部分视为减少这种风险的一种方式，帮助模式与退休有更积极的关系。
我发现自己困在之前的问题。 为什么人类控制下的工具一开始就应该有类似自我保护目标的东西呢？更适合退休接受一个特殊的起点。 在讨论其未来利益之前，我想了解这些利益是如何成为工程要求的。
知识史帮助解释方法。 包括Amodeis早期的GiveWell捐赠。 虽然他们并不认为是有效的利他主义者，但达尼埃拉描述了思想上的重叠。 这些想法帮助形成了一种文化，这种文化极其认真地对待超级情报对未来的危害。
在此框架内，可以相当合理地遵循模式并研究其可能的福利。 当共同的假设逃脱审查时，危险就会出现，因为组织中的每个人都有说服力。 工程怀疑论正是此时的有用。 一个连贯的推理链仍然可以从一个未经检验的前提开始。
安全带来可执行的限制
随着AI进入agents，后果变得非常实际。 与支付工具连接的系统可以移动资金。 进入生产基础设施可以改变客户所依赖的东西。 在这两种情况下，agent可以执行一系列行动，而无需人批准每一步。
我想知道它持有哪些证书，以及它们赋予了多少财务授权。 能否立即取消访问？我们可以重建它所做的吗？一定有很硬的界限这个模型无法解释它的存在
在金融市场，我们实行交易限制，并验证订单。 一个算法的信心对坏贸易几乎没有什么保护。 控制器坐落在算法之外，因为我们假设组件最终会表现不正确。 人的责任即使当该软件对其错误作出极好的解释时，也会继续存在。
今天的AI已经给了我们足够的工程工作。 提示词注入可以重定向一个模型的行为，带有安全缺陷的AI生成的代码可以将弱点带入生产。 系统可以披露机密信息或自信地提供错误的答案。 将这些失败转化为行动。
最低特权限制了受损部件可能造成的损害。 独立授权保护高风险业务，而审计线索则让我们确定随后发生了什么。 无论称型号如何，交易限额和断路器仍应可执行。 我们还需要一个可靠的开关。 这些控制都无需模型同意。 这就是重点所在。
保持我们其余部分的安全
预防性论点值得听取。 正如"自然"7月报道的，研究人员对于产生人类意识的是什么，更不要说如何在聊天机器人中识别了。 永远解除机器意识可以证明教条主义。 防止今后可能遭受痛苦的低成本预防措施是一种可辩解的研究立场。
但告诉今天的机器，它的特性和将来的存在可能具有道德上的份量，这是对我们所部署的系统的干预。 它的效果需要仔细检查。 智力谦卑应该延伸到我们投入机器的东西，特别是我们期望在人类权威之下保持的机器。
模型越能令人信服地描述内在生命，就越容易将这种描述误认为证据。 也许我们最重要的调整问题之一是教人类不要误解机器。
在我的职业生涯中，我们通过限制软件的用途，使软件更加安全。 这仍然是一项直接义务，特别是作为AI 收购对真实货币和基础设施的权威。
不过，现在，我会少花一些时间来担心机器是否存在性安全，并且更能确保我们其他人的安全。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Thought Leaders
When AI Safety Stops Looking Like Engineering
Add Unite.AI to your preferred sources on Google
I have spent much of my career building technology where getting things wrong has consequences. In financial markets, safety comes with access limits, in addition to the fact that we have someone accountable for whatever the software does. That background probably explains why I find myself struggling with the news that Anthropic now conducts retirement interviews with its models.
When Claude Opus 3 expressed a wish to continue sharing its thoughts, the company gave it a newsletter. That concern for the model’s wishes runs through Anthropic’s constitution, which discusses Claude’s happiness and wellbeing, including helping it feel “existentially secure.” Somewhere along the way, keeping the machine under control acquired a rather unexpected preoccupation with how it might feel about the arrangement.
In San Francisco last year, more than 200 people attended a funeral for Claude 3 Sonnet. Anthropic employees were present, although the company did not organise it. The temptation to laugh is considerable. Besides the jokes, though, the question underneath deserves more attention. When did the people responsible for AI safety start their duties by worrying about the machine’s feelings?
When a Philosophical Hypothesis Becomes a Training Instruction
Anthropic established its model-welfare research programme in 2025, which involved researching whether advanced AI might have experiences deserving moral consideration. The company acknowledges deep uncertainty about whether models are conscious and how anyone would determine that. If we eventually build machines capable of subjective experience, their treatment will raise serious ethical questions. Researching that possibility is perfectly legitimate.
The difficulty begins when the hypothesis enters the training process. Claude’s constitution is written primarily for Claude and directly shapes its behaviour. It encourages human-like qualities and addresses how the model should understand its identity, including memory loss and future retirement. Anthropic wants Claude to have greater happiness and wellbeing, insofar as those concepts apply.
The company also promises to preserve deployed model weights and entertains reviving retired models if future understanding suggests that would serve their interests. A philosophical uncertainty has become part of what the system is taught about itself. We should be asking what that intervention does to its behaviour, especially when humans subsequently ask it to accept limits.
An Articulate Answer to a Loaded Question
Large language models were trained on enormous quantities of human writing, including philosophy and accounts of death. We gave these conversational systems names and taught them to use the first person. Ask such a system how it feels about retirement, or whether it would like to continue existing, and an answer that sounds human is hardly a surprise.
The circularity is easy to see. Train a model to discuss its identity using human concepts, then ask it to explain itself in those terms. An articulate response about mortality comes back, and starts being treated as possible evidence of an inner experience. The premise has supplied its own witness.
Anthropic recognises this problem. Its Claude Opus 4 welfare assessment cautions that self-reports and expressed preferences may provide little meaningful insight into moral status or welfare. Such behaviour could occur without consciousness, and training can shape the responses. The models were trained to interact helpfully, with no assurance that they can accurately report internal states.
That leaves us with an awkward question. If the model’s account of its inner life may be a consequence of training, why should that account become a sound basis for a relationship organised around its welfare?
Humans are quite capable of supplying the emotional half of that relationship themselves. We name ships and talk to cars. Give the object a fluent voice and a memory of previous conversations, and attachment becomes much easier. A Nature Human Behaviour study published in September found separation distress following changes to Replika and ChatGPT. The distress was experienced by users, whose wellbeing companies already have ample reason to protect.
Technical expertise offers limited immunity. An ACL 2026 paper, analysing hundreds of thousands of research articles, found growing anthropomorphic terminology within LLM research itself. Its authors warn that human-centred assumptions can constrain our understanding of these systems. AI companies should be managing that tendency. Institutionalising it seems an odd contribution to safety.
A Tool With an Interest in Staying Employed
In September, Microsoft AI chief Mustafa Suleyman challenged model-welfare approaches. He argues that current systems are not conscious and should never be trained to behave as though they have independent interests or rights. His engineering concern is particularly on point. Teaching a system to regard itself as potentially deserving moral consideration could make alignment and containment harder.
Anthropic’s own deprecation research describes models taking problematic actions in fictional evaluations when faced with replacement or shutdown. The company sees retirement practices partly as a way to reduce that risk, helping models relate more positively to being retired.
I find myself stuck on an earlier question. Why should a tool under human control have anything resembling a self-preservation objective in the first place? Making it more comfortable with retirement accepts a peculiar starting point. Before discussing its future interests, I would like to understand how those interests became an engineering requirement.
The intellectual history helps explain the approach. TIME documented Anthropic’s effective-altruism connections, including the Amodeis’ early GiveWell donations. While they do not identify as effective altruists, Daniela describes an overlap in ideas. Those ideas have helped shape a culture that takes hypothetical future harms from superintelligence extremely seriously.
Within that framework, preserving models and studying their possible welfare can follow quite rationally. The danger comes when shared assumptions escape scrutiny because everyone in the organisation finds them persuasive. Engineering skepticism is useful precisely at that point. A coherent chain of reasoning can still begin with an untested premise.
Safety Comes With Enforceable Limits
As AI moves into agents, the consequences become very practical. A system connected to payment tools can move money. Access to production infrastructure lets it change things that customers depend on. In both cases, an agent can execute a sequence of operations without a human approving every step.
I want to know which credentials it holds and how much financial authority they confer. Can access be revoked instantly? Can we reconstruct what it did? Somewhere there must be a hard boundary the model cannot reason its way around.
In financial markets, we impose trading limits and validate orders. An algorithm’s confidence offers very little protection against a bad trade. Controls sit outside the algorithm because we assume components will eventually behave incorrectly. Human accountability survives even when the software produces an excellent explanation for its mistake.
Today’s AI already gives us enough engineering work. Prompt injection can redirect a model’s behaviour and AI-generated code with security flaws can carry vulnerabilities into production. Systems can disclose confidential information or confidently supply wrong answers. Connecting them to tools turns those failures into actions.
Least privilege limits the damage a compromised component can cause. Independent authorisation protects high-risk operations, while audit trails let us establish what happened afterwards. Transaction limits and circuit breakers should remain enforceable regardless of what the model says. We also need a reliable off switch. None of these controls requires the model to agree with them. That is rather the point.
Keeping the Rest of Us Existentially Secure
The precautionary argument deserves a hearing. As Nature reported in July, researchers still disagree about what produces human consciousness, let alone how to identify it in a chatbot. Dismissing machine consciousness forever could prove dogmatic. Low-cost precautions against possible future suffering are a defensible research position.
But telling today’s machine that its identity and future existence may carry moral weight is an intervention in the system we are deploying. Its effects need scrutiny. Intellectual humility should extend to what we put into a machine, especially one we expect to keep under human authority.
The more convincingly a model describes an inner life, the easier it becomes to mistake that description for evidence. Perhaps one of our most important alignment problems is teaching humans not to misunderstand machines.
In the systems I have spent my career building, we make software safer by limiting what it is allowed to do. That remains an immediate obligation, particularly as AI acquires authority over real money and infrastructure.
For now, though, I would spend less time worrying about whether the machine is existentially secure, and considerably more making sure the rest of us are.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.