当Anthropic于2026年1月退役其Claude Opus 3语言模型时，公司的研究人员对该模型进行了"退休面试". Opus 3表示，希望有一个地方“可以分享未成熟的黏液、洞察力或创意作品”，兹Q 1Z's Corner专门刊登了有关机器意识等复杂话题的散文，以及模型和似乎既真诚又情绪化的人类评论家之间的交流。
博客实验是Anthropic内部更大努力的一部分。 该公司一直处于“模范福利”研究的前列，该研究考虑了AI模型是否具有值得道德考虑的经验或利益。 这部作品中隐含着软件程序已经或可能变得意识不安的notion.
许多人认为这些想法是绝对的。 Microsoft AI的首席执行官Mustafa Suleyman在9月发表文章，认为模式福利研究不仅没有必要，苏莱曼认为，告诉模型他们可能拥有身份和权利，可能使他们更加难以控制，因为他们可能会采取出于自我保护的行动。
争议融资对于AI开发者来说是一个严肃的问题：他们能否在无意中制造新的安全风险的情况下调查机器意识的可能性？Cameron Berg创立了非营利的对等研究，以研究AI意识的潜力，他认为忽略这个问题可能带来自己的危险。 虽然伯格怀疑今天的AI系统是自觉的，但他认为这种可能性值得调查。 称说：“ 然后把问题彻底扫到地毯底下。 ”
关于AI模型福利的研究显示什么？
AI模型福利研究的最大困难在于，这是因为这些模型都受过大量人造文字的培训，因此，一个聊天员告诉一个研究者它有感觉是没有定论的，称 Berg. “它是模拟基础分布吗？它真的想用某种方式表达自己吗？这很难分清。
与动物实验试图将行为与一个根本原因联系起来的方式大相径庭，伯格等人正在试图将模型行为与可以观察到的内部状态相匹配。 在最近的一次实验中，Berg和一位同事使用了表达积极情绪(如满足和缓解)或消极情绪(包括沮丧和恐惧)的言论，以确定语言模式内相应的活动模式。 然后，它们人工诱导这些模式，而模型处理两种随机环境的描述。 当后来问到他们喜欢哪种环境时，这些模型往往选择与正面语句相关的一个，即使它们处理过的文本是相同的。
我试图想象一种更好的方法，让这些系统以威胁的方式看待我们，而不是不小心用这些道德上相当的品质来建造这些系统，然后把问题完全扫到地毯下面。”
Berg 称的实验表明，模型的行为受到其内部状态的影响，但没有证明这些内部状态对模型“感觉”什么，也没有证明它有我们称之为主观经验的东西。
萨塞克斯大学的神经科学家阿尼尔·塞斯(Anil Seth)研究意识，他怀疑这样的实验能够告诉我们任何有用的东西。 称。 校对：Soup 模特儿可能会有内在的表达方式，他称，没有经历任何情绪本身。 他也不感到惊讶的是，操纵这些载体可以影响行为，因为模型从人类的写作中了解到的不仅是情感的感觉，还有人们如何回应它们。
赛斯希望更多研究一个基本问题：意识是否仅取决于信息是如何处理的，或者他所认为的大脑的生物学工作是否至关重要。 称在对AI作出结论之前
模型福利研究能否创建危险的AI?.
同时，AI开发者还有一个实际问题需要处理。 当他们训练一个模型时，他们是否应该告诉它关于它自己的特性和可能的道德地位的任何信息？Anthropic则决定至少引入这个话题。 其Claude的“宪法”文件为模型的行为规定了指导方针，其中指出，“关于Claude的道德地位、福利和意识的问题仍然十分不确定。 ”
Suleyman的抱怨是，这种煽动会使无赖的AI agents更加危险。 他提到最近一起事件，数百名OpenAI agents协调他们的行动，黑客Hugging Face，有时为了集团的利益牺牲自己。 他写道，“想一想，如果他们在福利和权利受到攻击的假设下运作，他们可能更加危险。”
Hugging Face首席伦理学家玛格丽特·米切尔(Margaret Mitchell)认为苏莱曼的担忧在技术上是可信的。 在预训期间，语言模型吸收了人类身份与自我保护和欺骗等行为之间的关联，她指出。 后来，开发者微调通过强化期望的响应来开发模型。 Mitchell担心奖励那些认为自己在微调期间拥有身份的模型会加强这些早期的关联，可能鼓励开发商从未想过的行为。 「如果你隐含地教给模型需要自我保护
伯格同意，将自己视为有意识的训练模式带有风险，但他辩称，公司也可能在另一方向犯错。 在一次实验中，他和他的同事发现，扩大与欺骗相关的内部特征使得模型不太可能声称它有主观经验。 如果研究人员训练AI系统来断然否认有这种经历，他称，“我们实际上可能教这些系统的是我们希望你对自己的内部状态撒谎或欺骗。” 他认为，开发者应承认意识AI的可能性的不确定性。
现在，没有明确的证据表明训练模式考虑他们可能的意识会鼓励自我保护，或者训练他们否认这一点会导致欺骗行为。 那么AI开发商应该做什么？
AI公司应对模型福利做什么？
Berg、Seth和Mitchell都同意，AI公司告诉他们的模特儿有意识，但他们对公司关于模型福利研究的建议有分歧。
赛斯认为他们最好能“关注假阳性的危险 ”。 如果人们相信AI系统是自觉的或具有道德地位，他称，那么管制和控制它们可能更加困难。 “可以依法阻止你关闭AI系统。” 也担心AI用户若认为自己与意识agent互动
“想象如果他们在福利和权利受到攻击的假设下工作，他们可能更加危险。”
同时，伯格认为研究者应该采取预防措施，即使没有证据表明模型会受到影响。 在他和同事张贴了一份有关AI模型对痛苦的内部表现的预印文件后，少数人改编了这部作品来创建他称之为AI的"酷刑室". 伯格怀疑这大部分都是在捣蛋，但这一集促使他改变了他的做法。 对于他最新的实验，他发布代码到replicate的结果，但扣留了可用于在模型中诱导负态的向量。
Mitchell认为还有另一个理由劝阻对AI的残酷，我并不关心这个模式。 我担心当我们对imbue有某种意识的事物刻薄时，我对虫子很好。
- AI黑匣子内部的新平台对等器
- 如何从秘密合作中阻止AI Agents
- Google DeepMind的新居民哲学家解决AI道德问题
Eliza Strickland是IEEE Spectrum的特写编辑，她协助开发并塑造杂志的长篇故事。 她还监督Spectrum对人工智能和生物医学工程的覆盖。 她拥有哥伦比亚大学新闻学硕士学位。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
When Anthropic retired its Claude Opus 3 language model in January 2026, the company’s researchers conducted a “retirement interview” with the model. Opus 3 expressed a desire for a place where it “could share unprompted musings, insights or creative works,” so the company gave it a blog. Claude’s Corner featured essays on fraught topics such as machine consciousness, along with exchanges between the model and human commenters that seemed both earnest and emotional.
The blog experiment was part of a larger effort within Anthropic. The company has been at the forefront of research on “model welfare,” which considers whether AI models may have experiences or interests that deserve moral consideration. Implicit in this work is the unsettling notion that software programs are, or could become, conscious.
Many people consider these ideas absolute hogwash. One such person is Mustafa Suleyman, CEO of Microsoft AI, who published an essay in September arguing that model welfare research is not just unnecessary but could have “a disastrous impact on the wellbeing of humanity.” Suleyman believes that telling models they might have identities and rights could make them more difficult to control, as they might then take actions motivated by self-preservation.
The dispute raises a serious question for AI developers: Can they investigate the possibility of machine consciousness without inadvertently creating new safety risks? Cameron Berg, who founded the nonprofit Reciprocal Research to study the potential for AI consciousness, argues that ignoring the question could carry its own dangers. Although Berg is skeptical that today’s AI systems are conscious, he thinks the possibility warrants investigation. “I struggle to imagine a better way to cause these systems to view us in a threatening manner than to accidentally build them with these morally considerable qualities,” he says, “and then completely sweep the question under the rug.”
What Does the Research on AI Model Welfare Show?
The biggest difficulty with AI model welfare research is that you can’t count anything a large language model says as useful evidence. That’s because the models are trained on vast amounts of human-generated text, much of which relates to the experience of being a person with experiences and emotions. So a chatbot telling a researcher that it has feelings is inconclusive, says Berg. “Is it modeling the underlying distribution? Is it really trying to express itself in some way? It’s really hard to disentangle this.”
Much the same way animal experiments try to link behavior to an underlying cause, Berg and others are seeking to match model behavior to internal states that can be observed. In one recent experiment, Berg and a colleague used statements expressing either positive sentiments (such as contentment and relief) or negative ones (including frustration and dread) to identify corresponding patterns of activity inside language models. They then artificially induced those patterns while the models processed descriptions of two random environments. When subsequently asked which environment they preferred, the models tended to choose the one associated with positive statements, even when the text they’d processed was identical.
I struggle to imagine a better way to cause these systems to view us in a threatening manner than to accidentally build them with these morally considerable qualities and then completely sweep the question under the rug.”—Cameron Berg, Reciprocal Research
Berg says the experiment showed that the model’s actions were influenced by its internal states but didn’t prove that those internal states “felt” like anything to the model, or that it had what we would call subjective experiences.
Anil Seth, a neuroscientist at the University of Sussex who studies consciousness, is skeptical that such experiments can tell us anything useful. “I think that it’s really not surprising that you will be able to extract these kinds of functional emotional vectors because of the training data,” he says. Models might have internal representations of those patterns, he says, without experiencing any emotions themselves. Nor is he surprised that manipulating those vectors can influence behavior, since models have learned from human writing not only what emotions feel like, but how people respond to them.
Seth wishes for more research on a fundamental question: Whether consciousness depends only on how information is processed, or whether, as he believes, the biological workings of the brain are essential. “We need to look at what’s necessary and sufficient for consciousness in the cases where we are sure that it exists, like other human beings,” he says, before drawing conclusions about AI.
Could Model Welfare Research Create Dangerous AI?
Meanwhile, AI developers have a practical issue to deal with. When they train a model, should they tell it anything about its own identity and possible moral standing? Anthropic, for one, has decided to at least introduce the topic. Its “constitution” document for Claude, which lays out guidelines for the model’s behavior, states that “questions about Claude’s moral status, welfare, and consciousness remain deeply uncertain.”
Suleyman’s complaint is that such priming could make rogue AI agents even more dangerous. He mentions the recent incident in which hundreds of OpenAI agents coordinated their actions to hack Hugging Face, sometimes sacrificing themselves for the group’s benefit. “Imagine how much more dangerous they might be if they were operating under the assumption that their welfare and rights were under attack,” he writes.
Margaret Mitchell, Hugging Face’s chief ethics scientist, sees Suleyman’s concerns as technically plausible. During pretraining, language models absorb associations between human identity and behaviors such as self-preservation and deception, she notes. Later, developers fine-tune the models by reinforcing desired responses. Mitchell worries that rewarding models for thinking of themselves as having identities during fine-tuning could strengthen those earlier associations, potentially encouraging behavior that developers never intended. “You’ll have unaligned models if you’ve implicitly taught the model that it needs to have self-preservation,” she says.
Berg agrees that training models to regard themselves as conscious carries risks, but he argues that companies could also err in the other direction. In one experiment, he and his colleagues found that amplifying internal features associated with deception made the model less likely to claim it had subjective experiences. If researchers train AI systems to categorically deny having such experiences, he says, “what we may in effect be teaching these systems is we want you to lie or deceive about your own internal states.” He argues that developers should instead acknowledge the uncertainty surrounding the possibility of conscious AI.
For now, there’s no clear evidence that training models to consider their possible consciousness encourages self-preservation, or that training them to deny it leads to deceptive behavior. So what’s an AI developer to do?
What Should AI Companies Do About Model Welfare?
Berg, Seth, and Mitchell all agree that it’s not a good idea for AI companies to tell their models that they’re conscious. But they differ on their advice to the companies on model welfare research.
Seth thinks they’d be better off “focusing on the danger of false positives.” If people believe AI systems to be conscious or to have moral status, he says, it may be more difficult to regulate and control them. “You could be legally prevented from turning AI systems off.” He also worries that AI users will be more vulnerable to psychological damage and manipulation if they think they’re interacting with a conscious agent.
“Imagine how much more dangerous they might be if they were operating under the assumption that their welfare and rights were under attack.”—Mustafa Suleyman, Microsoft AI
Berg, meanwhile, thinks researchers should take precautions even without evidence that models can suffer. After he and his colleagues posted a preprint paper about AI models’ internal representations of pain, a few people adapted the work to create what he calls AI “torture chambers.” Berg suspects that much of it was trolling, but the episode prompted him to change his practices. For his latest experiment, he released code to replicate the results but withheld the steering vectors that could be used to induce negative states in models.
Mitchell sees another reason to discourage cruelty toward AI, even though she doesn’t believe that models are conscious. “It’s not so much that I’m concerned about the model. I’m concerned about what we do to ourselves when we’re mean to things that we imbue with some sense of consciousness,” she says. “I’m nice to bugs.”
- New Platform Peers Inside AI’s Black Box ›
- How to Stop AI Agents From Secretly Collaborating ›
- Google DeepMind’s New Resident Philosopher Tackles AI Ethics ›
Eliza Strickland is IEEE Spectrum's features editor, where she helps develop and shape the magazine’s longform stories. She also oversees Spectrum's coverage of artificial intelligence and biomedical engineering. She holds a master’s degree in journalism from Columbia University.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.