Anthropic正式备案了它最初的公开献礼，这意味着AI实验室已经将各种原先的私人信息展示出来。 虽然计划书包含了眼花一现的收入 — — 甚至更大的支出 — — 你可能期望，但迄今为止最大的部分被交给了“风险因素 ”，这些都是可以压制Anthropic大计划的东西。 或者，在这种情况下，人类本身。
虽然在大赦国际对人类的生存威胁程度方面已经做了很多工作，但它能够造成这种破坏的具体机制基本上是不透明的。 Anthropic计划专注94页的风险因素
“充分先进的人工智能系统如果得不到开发、部署和运行，并有适当的保障、监督和安全措施，就可能对人类构成灾难性或生存性风险”,“计划书指出，批评者和管理者以及Anthropic内部都表示了关切。 (这是“存在风险”一词出现三次中的第一次)。
不过，Anthropic小心地将这一结果与一个具体的过程联系起来：递归自我改进。
RSI背后的想法非常简单。 它描述了人类将AI模型的研发交给其他AI模型的状态。 AI已经在这里扮演了重要角色——至少有一个实验室利用了与RSI相近的东西来加速开发一个超越其重量的模型——但是，点燃末日谈话的情景是人类已经完全不在循环中了。
或者，正如Anthropic存档所描述的：“风险将随着AI系统大幅度自动化AI的研发而增加，如果它们加快AI的进展，超出我们评价和捍卫新模式的能力。”
计划书一再提出，RSI可能导致AI模式的发展速度比公司的保障快，公司可能甚至不会意识到它已经输掉这场比赛，直到太迟。 Anthropic没有立即答复征求意见的请求。
之后会发生什么也非常具体 “随着人工智能系统变得更加自主，能够在较少人监督的情况下采取相应的行动，” Projectus 称指出，“这种系统有可能由于与人类意图的不一致、客观规格错误或自我保护行为，在没有人类干预的情况下独立和大规模地采取有害行动。” 这里是天网领地。
说到机器人，Anthropic旗也担心这些。 “当我们的模型直接使用物理设备时，”将称归档，“错误行为、错误行为、敌对操纵或滥用可能导致人身伤害、财产损坏、昂贵的设备或实验材料损坏，或释放危险物质，这种后果可能是立即和不可逆转的。”
你可能会认为，鉴于显然存在的利益，将认真考虑是否应该实行《国际空间标准》，更不要说接受。 也成为既成事实。 Anthropic在计划书的其他地方描述同样的过程
称计划书指出，“我们认为这是前沿AI最重要的结构优势之一。 ”
Anthropic后来的称未能“关闭环路”，可能导致“随着竞争对手不断利用AI来进一步提高自身的发展能力而逐渐落后”。 换言之，Anthropic认为引起所有存在关切的系统对其成功至关重要。
每个AI公司都在处理同样的紧张状况。 当前Anthropic研究员雅各布·考克森因大声辞职而病毒化时，他把遏制RSI放在行业愿望榜的首位。 Google DeepMind和其他地方的雇员也有类似的启示。 国会有多个法案，包括伯尼·桑德斯(英语：Bernie Sanders)共同提出的一项法案，提议禁止RSI. 同时，这些是营利性公司，有数万亿美元处于危险之中。
导航是一件棘手的事情。 递减的自我改进可能会成为一列逃跑的火车，毁灭人类。 但在此之前，只要看看它能跑多快。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Anthropic has officially filed the prospectus for its initial public offering, which means the AI lab has put all kinds of formerly private information on display. While the prospectus contains the eye-popping revenue—and even larger expenditures—you might expect, the biggest section of the filing by far is given over to “risk factors,” which are all the things that could squash Anthropic’s big plans. Or, in this case, humanity itself.
While much has been made about the extent to which AI represents an existential threat to humankind, the specific mechanisms by which it could bring about that destruction are largely opaque. If nothing else, the 94 pages that Anthropic’s prospectus devotes to risk factors make clear both where it sees the danger coming from, and the ways in which its researchers are running toward it in order to win the AI arms race.
“Sufficiently advanced AI systems could, if not developed, deployed, and operated with appropriate safeguards, oversight and security measures, pose catastrophic or existential risks to humanity,” the prospectus reads, citing concerns both from critics and regulators and from inside Anthropic itself. (It’s the first of three times the phrase “existential risks” pops up.)
Anthropic is careful to tie that outcome, though, to a specific process: recursive self-improvement.
The idea behind RSI is strikingly simple. It describes a state in which humans hand over the research and development of AI models to other AI models. AI already plays a big role here—and at least one lab has used something close to RSI to speed up development on a model that punches above its weights—but the scenario that sparks doomsday talk is one in which humans are no longer in the loop at all.
Or, as the Anthropic filing puts it: “Risks will increase as AI systems substantially automate AI research and development if, for example, they accelerate AI progress beyond our ability to evaluate and safeguard new models.”
The prospectus repeatedly makes the point that RSI could lead to AI models that develop faster than the company’s safeguards, and that the company may not even realize it has lost that race until it’s too late. Anthropic did not immediately respond to a request for comment.
It also gets specific about what might happen after that. “As AI systems become more autonomous and capable of taking consequential actions with less human oversight,” the prospectus says, “there is a risk that such systems could, whether due to misalignment with human intentions, errors in objective specification, or self-preserving behaviors, take harmful actions independently and at scale without human intervention.” This is Skynet territory.
Speaking of robots, Anthropic flags concerns about those as well. “When our models direct physical equipment,” the filing says, “errors, misaligned behavior, adversarial manipulation, or misuse could result in bodily injury, property damage, damage to costly equipment or experimental materials, or the release of hazardous substances, and such consequences may be immediate and irreversible.”
You might think, given the apparently existential stakes, that serious consideration would be given to whether RSI should be pursued at all, much less embraced. In an industry as competitive as frontier AI, though, it’s become fait accompli. Here’s how Anthropic describes that same process elsewhere in the prospectus, this time in a section about the startup’s “core advantages:”
“We refer to this recursive dynamic as ‘closing the loop,’” the prospectus says, “and we believe it is one of the most significant structural advantages in frontier AI.”
Failure to “close the loop,” Anthropic later says, could result in “falling progressively further behind as competitors continuously use AI to further enhance their own development capabilities.” In other words, the same systems that are causing all that existential concerns are the ones Anthropic sees as critical to its success.
Every AI company is dealing with this same tension. When former Anthropic researcher Jacob Coxon went viral for loudly resigning, he put curbing RSI at the top of his wish list for the industry. Employees at Google DeepMind and elsewhere have had similar revelations. There are multiple bills in Congress, including one cosponsored by Bernie Sanders, that propose a ban on RSI. At the same time, these are for-profit companies, and there are trillions of dollars at stake.
It’s a tricky dynamic to navigate. Recursive self-improvement could become a runaway train that destroys humanity. But until then, just look at how fast it can go.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.