“反向游戏理论”如何控制 AI agents

“反向游戏理论”如何控制 AI agents

从经济学等学科借钱有助于AI安全研究者控制AI模型

称为“反向游戏理论”的新办法旨在为强大的AI模型创造激励机制，使遵循规则成为最合理的选择。

随着AI模型能力增强，安全研究者纷纷奔走，以确保它们与人类的欲望“一致”，换句话说，确保无赖的agents不违法，侵入未经授权的服务器，或向大城市释放毒气并杀死我们所有人。

直到现在，这项工作主要集中于尝试与AI的"大脑"相对应，以了解它是什么"思维"，或者训练模型来牵引我们所珍视的价值观。 例如，Claude有其宪法——一种道德守则。

问题在于这些技术似乎不太有效：事实证明，模型在测试中故意表现不佳，隐藏其意图，误导其评估人员。

现在一些对齐研究者认为，反向游戏理论，也称为机制设计，可能会得到更好的结果。 在游戏理论试图预测理性角色在给定一套特定规则时将如何行为时，反向游戏理论试图设计规则，以引出我们要鼓励的行为类型。

在现实世界中机制设计的一个例子可能是将蛋糕分成两个的经典策略：一人削减蛋糕，另一人获得第一选择的棋子。 这激励了切割机尽可能地使碎片变得平衡。 其他例子包括骑头应用软件的快速定价，鼓励司机在繁忙时段可以使用，以及二价拍卖，其中中标者支付第二高标价而不是最高标价。 平台利用这些拍卖来销售网上广告，因为它们鼓励买家出价接近自己愿意支付的上限。

在大赦国际中，机制设计始于一个前提，即我们可能永远不能充分了解前沿模型的内部运作情况，以确保它们一致。 这表示我们应该设计一些机制来激励agents的行为方式

麻省理工学院的经济学家斯蒂芬·莫里斯(Stephen Morris)与耶鲁和哥伦比亚的同事最近发表的一份预印论文建议了一些可能的机制，可以阻止agents进行 " 摇摆 " 测试(准备比实际更糟糕——例如避免触发安全机制)。

一个模型与其只因其在测试中的表现如何而得到奖励，不如因其答案如何能预测第二个模型会说什么而得到奖励。 故意装傻就输了 “AI agents非常了解AI agents的其他成就”,Morris 称。 “因此，这种交叉核对确实可以更实际地发挥作用。

论文还提出将agents互相配对，形成一种激励他们表现最好的竞争动态，或者将已知偏差相反方向的agents配对，并根据产出的匹配程度给予奖励，从而激励他们达成最佳答案。

普林斯顿大学计算机科学教授埃拉德·哈赞(Erad Hazan)设想的第二种方法是设计一些机制，从人类社会使人民“团结”的方式中汲取灵感：社会规范、司法制度、自由市场资本主义的金钱激励。 称：我们想加入超级智能AI, “我们为什么不先谈人类呢？我们如何团结人类？”

Hazan建议的机制是让AI模型轮流作为解决问题的“解决者”或检查他人工作的“审计者”。 “一个agent行为类似于执法；另一个agent行为类似于公民。”

在训练期间，模型根据结果得到奖励：被抓到作弊的解决方案者会受到惩罚，但未能抓到作弊者的审计员也会受到惩罚。 在审计员和解决人员之间有分歧的情况下，案件可以送交 " 审判 " ——也许由一位专家来决定谁是对的。

Hazan承认，agents有相互勾结的风险，因为审计师与他们打算审计的企业勾结，或者腐败的警察与罪犯勾结。 但是，这个问题可以通过对审计员给予足够的奖励来解决，因此被抓住并不值得冒风险(在AI训练的世界里，比在警察薪水紧张的实训要容易得多)，并且通过审计审计员使用类似雇用神秘购物者的技术。

哈赞指出诺贝尔奖获奖者是经济学家加里·贝克尔(Gary Becker)的作品，推出对20世纪60年代的"理性罪犯"的想法。 罪犯不会违法，更何况，他们这样做时，只有收益超过潜在的惩罚，犯罪才有意义。

他表示，校正不应该被认为是模型的属性，“你是个好人，” “你同意吗？我不知道，可能大部分时间你都是靠拢的

将机制设计应用于AI的一个潜在问题：它依赖于AI agents行为理性的假设。 机制设计与人类有着混合的纪录，他们往往没有按照经济模型所说的方式行事。

具有讽刺意味的是：经济学家们花了几十年时间试图将人类的行为降低到数学水平，现在，在大语言模型中，我们已经建立了数学系统，这些系统运行在人类语言上，并且可能同样令人不解。 AI模型显示的行为看起来比机器更人性化，它们可以被难以理解和解释的微妙因素摇动。 “假设他们会响应激励措施是否合理？”

“显然，AI是不合理的，”Morris 称。 “他们只是算法而已。” 然而，正如经济学家在人类身上所发现的，如果能帮助预测和控制他们的行为，那么把他们当作理性的人，仍然可以起到有益的作用。

Qu认为机制设计只是 " 深入防卫 " 战略中保持AI模型一致的一层----与可解释性、调控性和培训并列，而不是代之以。 但是，在实际前沿模式中实施这种方法之前，还有很多问题需要回答。 例如，这些理论如何适用于真正的模型，或者为某项任务设计的机制在实践中是否将延续到另一项任务，这一点并不清楚。 模型可能发现无人预料的漏洞。 (Anthropic和OpenAI没有答复关于它们是否正在就此开展工作的评论请求。 )

无论这些机制是什么样子，它们最终都必须进入最大的机制：自由市场。 称 Hazan,“我们遵循经济学规则，”。 更一致的模型也许对世界有好处，但如果它比不太一致的模型慢或更昂贵，那么我们目前的规章制度体系是否实际上鼓励任何前沿实验室释放它？

市场中的称说：“ 但最终它可能要为非常可怕的事情负责。 ” 也许真正需要更好的机制设计的实体是实验室本身。

深潜

人工智能

不要被愚弄-LLMs没有道理

AlphaGo对阵Go冠军李赛铎十年后

“我们看错了”：破坏AI世界的爆炸性说法

他的2.6万字的病毒博客文章中

别被这个夏日的爱尔胡同所愚弄

关于AGI的无呼吸声称和新能力在严格审查下迅速瓦解。

美国花费数十亿用于边境监视。 为何不能在人们死前抓住他们？

监视塔日益由大赦国际提供动力，它保证对过境点作出更快的反应。 我们对塔台附近死亡事件的调查发现，它们未能交付。

保持连接

从

麻省理工学院技术审查

发现特别的报价，顶级故事，即将到来的事件等等。

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

How “reverse game theory” could keep AI agents in check

Borrowing from disciplines like economics could help AI safety researchers control AI models, even if they don’t understand them.

A new approach known as “reverse game theory” aims to create incentives for powerful AI models that make following the rules the most rational choice.

As AI models become more capable, safety researchers are scrambling to ensure they are “aligned” with human desires—to make sure, in other words, that rogue agents don’t break the law, hack into unauthorized servers, or release poison gas into major cities and kill us all.

Until now, this work has mostly focused on trying to peer into an AI’s “brain” to understand what it is “thinking,” or training models to hew to values that we hold dear. Claude, for instance, has its constitution—a sort of moral code.

The trouble is, these techniques don’t seem to work particularly well: Models have been shown to deliberately underperform on tests, hide their intentions, and mislead their evaluators.

Now some alignment researchers believe that reverse game theory, also known as mechanism design, might get better results. Where game theory tries to predict how rational actors will behave when given a particular set of rules, reverse game theory attempts to design rules that will elicit the kinds of behavior we want to encourage.

An example of mechanism design in the real world might be the classic strategy for splitting a piece of cake in two: One person cuts the cake and the other gets the first choice of pieces. This incentivizes the cutter to make the pieces as even as possible. Other examples include surge pricing in ride-hailing apps, which encourages drivers to be available at busy times, and second-price auctions, where the winning bidder pays the price of the second-highest bid instead of the highest. Platforms use these auctions to sell online advertising because they encourage buyers to bid close to the maximum they are willing to pay.

In AI, mechanism design starts from the premise that we may never be able to understand enough about the internal workings of frontier models to make sure they are aligned. This means we should design mechanisms that incentivize agents to behave in ways that we want, regardless of what’s actually going on under the hood.

A recent preprint paper by Stephen Morris, an economist at MIT, along with colleagues from Yale and Columbia suggested some potential mechanisms that could stop agents from “sandbagging” tests (pretending to be worse than they actually are—for example, to avoid triggering safety mechanisms).

Instead of being rewarded only for how well it does on a test, a model could be rewarded for how well its answers predict what a second model will say. Deliberately playing dumb becomes a losing bet. “AI agents have a very good understanding of what other AI agents can achieve,” Morris says. “So it does seem like a context where this cross-checking could work more practically.

The paper also suggests pitting agents against each other, creating a competitive dynamic that incentivizes them to perform their best, or pairing agents that are known to be biased in opposite directions and rewarding them depending on how closely their outputs match, thus incentivizing them to reach an optimal answer.

A second approach, as envisioned by Elad Hazan, a professor of computer science at Princeton, is to design mechanisms that take inspiration from the ways human society keeps people “aligned”: social norms, the justice system, the monetary incentives of free-market capitalism. “We want to align superintelligent AI,” he says. “Why don’t we first talk about humans? How do we align humans?”

Hazan’s suggested mechanism involves having AI models take turns as either “solvers” that tackle problems or “auditors” that check the others’ work. “One agent acts like law enforcement; the other agent acts like the citizen,” he says.

During training, the models are rewarded depending on outcomes: Solvers that get caught cheating are punished, but so are auditors that fail to catch the cheaters. In situations where there’s a disagreement between auditors and solvers, the case can be sent to “trial”—perhaps a human expert will decide who is right.

There is, Hazan admits, a risk that agents will collude with each other—as auditors have colluded with the businesses they’re meant to be auditing, or corrupt police officers have colluded with criminals. But that problem can be solved by rewarding the auditors well enough, so that being caught isn’t worth the risk (a lot easier in the world of AI training than the real one, where money for police salaries is tight), and by auditing the auditors using techniques akin to employing mystery shoppers.

Hazan points to Nobel Prize–winning work by the economist Gary Becker, who introduced the idea of the “rational offender” in the 1960s. Criminals don’t break the law because they’re inherently bad people, he said. Instead, they do it when it’s the most rational choice given the incentives on offer. Crime makes sense only when the proceeds outweigh the potential punishment.

Alignment shouldn’t be thought of as a property of a model, he argues, but as a descriptor of its behavior at a particular moment. “You are, like, a nice person,” he says. “Are you aligned? I don’t know. Probably you are aligned most of the time, but there could be a situation where you’re not.”

One potential issue with applying mechanism design to AI: It relies on the assumption that AI agents behave rationally. Mechanism design has a mixed record with humans, who often fail to behave the way economic models say they should.

It’s ironic: Economists have spent decades trying to reduce human behavior to mathematics, and now, in large language models, we’ve built mathematical systems that run on human language and could be just as inscrutable. AI models exhibit behaviors that seem more human than machine, and they can be swayed by subtle factors that are difficult to understand and explain. “Is it even reasonable to assume that they will respond to incentives?” asks Helen Qu, a research fellow at the Flatiron Institute, a nonprofit dedicated to advancing scientific discovery through computational methods, who has written about mechanism design and AI safety.

“Obviously, AIs are not rational,” Morris says. “They’re just algorithms at work.” However, as economists have found with humans, treating them as if they’re rational could still be useful if it helps to predict and control their behavior.

Qu sees mechanism design as just one layer in a “defense in depth” strategy for keeping AI models aligned—alongside interpretability, regulation, and training, not instead of them. But there are a lot of questions to be answered before the approach can be implemented in actual frontier models. It’s unclear, for instance, how these theories will apply to real models, or whether mechanisms designed for one task will carry over to another in practice. Models could find loopholes that no one anticipated. (Anthropic and OpenAI did not respond to requests for comment on whether they are working on this.)

Whatever these mechanisms look like, they will ultimately have to slot into the biggest mechanism of all: the free market. “We play by the rules of economics,” says Hazan. A more aligned model may be good for the world, but if it’s slower or more expensive than a less aligned one, does our current system of rules and regulations actually incentivize any of the frontier labs to release it?

“It’s a mechanism that’s responsible for a lot of prosperity for humankind,” he says of the market, “but it could be ultimately responsible for something very terrible.” Perhaps the entities really in need of a better mechanism design are the labs themselves.

Deep Dive

Artificial intelligence

Don’t be fooled—LLMs don’t reason

Ten years after AlphaGo’s match against Go champion Lee Sedol, today’s AI still isn’t tapping into the machinery that made that win possible.

“We were looking at it wrong”: The explosive claim that’s upended the AI world

An exclusive interview with Nick Cammarata, the researcher whose 26,000-word viral blog post has AI safety folks celebrating.

Don’t be fooled by this summer of AI hype

Breathless claims about AGI and new capabilities fall apart pretty quickly under scrutiny.

The US spent billions on border surveillance. Why can’t it catch people before they die?

Surveillance towers, increasingly powered by AI, promised faster responses to crossings. Our investigation of deaths near the towers found they are failing to deliver.

Stay connected

Get the latest updates from

MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Auto-fetched and translated by AI Atlas. All rights belong to the original author.