语文模式应停止之处：设计用于惩罚错误的薪金、录取和其他领域的AI

思想领袖

语文模式应停止之处：设计用于惩罚错误的薪金、录取和其他领域的AI

在 Google 上的首选来源中添加 Unite.AI

我花了几年时间将AI建设成一个错误答案附带价格的产品：一个澳大利亚市场的AI动力工资合规平台，其中某人的银行账户错误获得，以及一个大学招生产品，一个差劲的匹配可以塑造学生未来的四年。 决定建筑结构的问题在两者中都是一样的。 语言模型在决定主义接管之前应该走多远？

美国五分之一的薪工单包含一个错误，在诉讼进入画面前，平均成本各291美元。 《欧盟AI法》将AI用于高风险类别的录取决定和就业管理。

让规则拥有数字

决定性规则拥有任何带有美元数字或法律后果的东西，模型围绕这些规则来解释和解释，仅此而已。

在发薪平台上，我们的目标是在发薪规则和AI之间划清界限。 薪金专家首先以结构化的格式界定薪金规则。 该系统然后处理这些规则并产生由此可能产生的各种可能情况。 这一整套情景成为模型真相的来源。 模型不能创建新的规则，改变现有的规则，或决定使用哪些来源。 它只能检索相关规则设置，查看速率表，并引用它所使用的情景。 规则创建，情景生成和源选择完全停留在AI层之外。

录取产品遵循相同的逻辑：地点要求，预算限制，机构地位，资格门槛等作为规则适用，模式才能影响排名。 如果一个学生称他们必须上加州的大学，州外的学校不能凌驾于这个之上，因为模型认为这是一个很强的适合。 该模型处理较软的偏好和解释，而确定性的逻辑则是在大约正确而不能接受的地方的护栏。

检索模型的底部，最多点

获得补偿的一代将答案与组织本身的政策挂钩，而不是与培训中吸收的模式挂钩。 它的售价也过高。 当斯坦福的RegLab测试基于RAG的商用法律研究工具时，他们仍然在17%到33%的时间里产生幻觉。 检索缩小了模型可以表达的内容，模型仍然可以弄错了。

我直接学的在发薪平台上，系统一度发现一个案件的正确就业奖，模型仍然对加班规则的解释不正确。 正确的来源已经找到，但答案是错误的，因此，之后，我们变得更加严格对于模型可以自行决定什么。 重要的发薪规则进入了明确的业务逻辑，每个答案都必须得到源材料的支持，我们加入了围绕这个案例构建的失败测试。 获取正确的信息只有在同时控制模型如何使用时才会有所帮助。

使每个答案都能追踪

在生成回复之前，系统会准确记录使用哪个规则书和版本，以及模型和其他请求的细节。 说大赦国际的答复是基于公司政策的说法很难核实。 说它是使用特定协议的第7版生成的，该版本已被保留，可以与其他版本相比，是更强有力的证据。 我们还追踪每个模型和工具的呼叫：用户问什么，什么被检索，什么工具运行，什么模型返回，以及沿途的任何错误。 当一个支助问题出现时，我们可以确切地看到一个错误的答案来自何处。 2024年加拿大航空公司的裁决，法庭让航空公司享受其聊天机发明的折扣，显示是替代成本。

人类后果在哪里

NIST的Generative AI Profile将系统单独做什么和需要一个人做什么作为治理决定之间的界线。 当系统不确定时，当信息冲突时，或者当决定可能对用户产生有意义的影响时，我们把案件转到人身上。 在入学时，非同寻常的学生情况，或无法明确满足学生困难要求的推荐，都会被贴上标签，供辅导员审查。 这一步骤吸引了表面上看似合理但错过了学生形象限制的建议，每个建议都反馈到系统中。 两种产品都无法自行完成。 在依赖建议之前，需要咨询员的投入，而且发薪系统产生的每笔工资在发薪之前都要标出供批准。 人文审查并不是针对每一项请求的。 目标是在系统有充分理由不确定时，自信地将例行案件自动化，并让一个人参与。

决定模特儿今天能拥有什么

我对任何工作流程都有一个实际测试。 如果一个错误的输出可以在到达一个人之前被捕获和取消，模型就可以拥有这个步骤。 如果根据规则可以检查答案，则由规则拥有，模型解释。 如果一个错误的产出在任何人看到它之前就达到工资检查或申请决定，该步骤还没有准备好。

语言模型是我们在一本规则书面前最好的口译员。 在惩罚错误的领域，解释是应该停止的地方。

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Thought Leaders

Where the Language Model Should Stop: Designing AI for Payroll, Admissions, and Other Domains That Punish Mistakes

Add Unite.AI to your preferred sources on Google

I have spent the past several years building AI into products where a wrong answer has a price attached: an AI-powered payroll compliance platform for the Australian market, where an error lands in someone’s bank account, and a college admissions product, where a poor match can shape a student’s next four years. The question that decided the architecture was the same in both. How far should the language model be allowed to go before something deterministic takes over?

One in five U.S. payrolls contains an error, at an average cost of $291 each before litigation enters the picture. The EU AI Act puts AI used for admissions decisions and employment management in its high-risk categories.

Let the Rules Own the Numbers

Deterministic rules own anything with a dollar figure or a legal consequence attached, and the model interprets and explains around those rules, and nothing more.

On the payroll platform, our goal was a clear boundary between the payroll rules and the AI. Payroll experts first define pay rules in a structured format. The system then processes those rules and generates every possible scenario that can result from them. That complete set of scenarios becomes the model’s source of truth. The model cannot create new rules, change existing ones, or decide which sources to use. It can only retrieve the relevant rule set, look up rate tables, and cite the scenario it used. Rule creation, scenario generation, and source selection stay outside the AI layer entirely.

The admissions product follows the same logic: location requirements, budget limits, institution status, and eligibility thresholds are applied as rules before the model can influence a ranking. If a student says they must attend college in California, an out-of-state school cannot override that because the model thought it was otherwise a strong fit. The model handles softer preferences and explanations, and deterministic logic is the guardrail wherever approximately right is not acceptable.

Retrieval Grounds the Model, Up to a Point

Retrieval-augmented generation keeps answers tied to the organization’s own policies rather than whatever the model absorbed in training. It is also oversold. When Stanford’s RegLab tested commercial legal research tools built on RAG, they still hallucinated between 17% and 33% of the time. Retrieval narrows what the model can say, and the model can still get it wrong.

I learned that directly. On the payroll platform, the system once found the correct employment award for a case, and the model still interpreted the overtime rules incorrectly. The right source had been retrieved, yet the answer was wrong, so after that we became much stricter about what the model could decide on its own. Important payroll rules moved into explicit business logic, every answer had to be supported by source material, and we added tests built around the case that had failed. Access to the right information only helps if you also control how the model uses it.

Make Every Answer Traceable

Before a response is generated, the system records exactly which rulebook and version were used, along with the model and other request details. Saying that an AI response was based on company policy is difficult to verify. Saying that it was generated using version 7 of a specific agreement, which has been retained and can be compared with other versions, is much stronger evidence. We also trace every model and tool call: what the user asked, what was retrieved, which tools ran, what the model returned, and any errors along the way. When a support issue comes in, we can see exactly where an incorrect answer came from. The 2024 Air Canada ruling, where a tribunal made the airline honor a discount its chatbot had invented, shows what the alternative costs.

Put Humans Where the Consequences Are

NIST’s Generative AI Profile treats the line between what a system does alone and what requires a person as a governance decision. We route a case to a human when the system is uncertain, when information conflicts, or when the decision could meaningfully affect the user. In admissions, an unusual student profile, or a recommendation that does not clearly satisfy a student’s hard requirements, gets flagged for counselor review. That step has caught recommendations that looked reasonable on the surface but missed a constraint in the student’s profile, and each one fed back into the system. Neither product finalizes anything on its own. Counselor input is required before a recommendation is relied on, and every pay stub the payroll system generates is flagged for approval before it goes out. Human review is not meant for every request. The goal is to automate routine cases confidently and involve a person when the system has a good reason to be uncertain.

Deciding What the Model Can Own Today

There is a practical test I apply to any workflow. If a wrong output can be caught and undone before it reaches a person, the model can own that step. If the answer can be checked against a rule, the rule owns it and the model explains it. If a wrong output would reach a paycheck or an application decision before anyone sees it, that step is not ready.

Language models are the best interpreters we have ever put in front of a rulebook. In domains that punish mistakes, interpretation is where they should stop.

Auto-fetched and translated by AI Atlas. All rights belong to the original author.