在 InfoQ 近期举办的一场题为《生产环境中的 AI：哪些会失效、哪些能正常运行，以及由谁审批上线？》的网络研讨会上，一个由资深工程实践者组成的小组深入剖析了部署智能体系统过程中遇到的各类现实难题。本次研讨会由 Hien Luu 主持，嘉宾包括来自 ThoughtWorks 的 Zichuan Xiong 和 Prem Chandrasekaran、隐私领域专家 Katharine Jarmul 以及金融 AI 从业者 Lan Chu。
Luu 在开场时指出了三个核心驱动因素：AI 的失效模式与传统软件不同，它会生成看似合理但不准确的结果，而不是直接崩溃；编码智能体生成生产代码的速度已经超过了人类的代码评审速度；敏感数据有了新的泄露途径，包括提示词、上下文窗口以及检索到的文档。
ThoughtWorks 托管服务 AI 运维负责人 Xiong 强调了 LLM 流水线中普遍存在的“语义漂移”问题。他描述了一个文本转 SQL 的翻译智能体如何频繁地混淆特定领域的概念，例如将销售领域和客服支持领域的“会员”视为同一概念，从而在下游业务分析中产生不易察觉的错误，而这些错误却能绕过常规的链路追踪检测。
同样，Chu 分享了一个由金融合规系统中的“嵌入漂移”引发的严重故障。在部署了一个基于向量相似度阈值来拦截违规投资建议的智能体后，底层嵌入模型的一次后台更新改变了向量距离分布。这导致线上流量中的误报和漏报率急剧飙升，而与此同时，所有基础设施健康检查却完全显示为绿色（正常）。
此外，Chandrasekaran 指出 RAG 流水线可能会在无意中检索到敏感文档，而 Jarmul 则提到模型本质上会记忆训练数据，可能被用来去匿名化信息或泄露 API 密钥。Chandrasekaran 还提到了 GitSpawn 漏洞，说明恶意仓库配置能够在执行常规命令时触发未授权代码运行。
随着编码助手生成的生产代码数量日益庞大，传统的人工代码评审流程正面临难以持续的瓶颈。ThoughtWorks 技术与商业服务负责人 Chandrasekaran 强调，逐行的人工代码评审无法跟上高速发展的智能体产出。工程团队必须在 CI/CD 流水线中实现战术层面的风格和语法检查自动化，人工审查只聚焦高层架构、业务意图以及非功能性需求。
Xiong 补充说，团队必须将工作“左移”，在拉取请求生成之前就在 harness 层面评估意图。Chandrasekaran 还强调了变异测试的再度兴起，认为这是一项关键实践。通过引入刻意的代码变异，团队可以验证自动化测试套件是否真正校验了底层逻辑，而不只是单纯满足表面的代码覆盖率指标。
在评估智能体系统时，Chu 强调，评估体系应当作为长期持续运转的基础设施，而不是发布前一成不变的检查清单。有效的评估需要在 span 级别进行持续的在线追踪、日志过滤，并使用 LLM 对故障模式进行分类，从而使团队能够将生产环境中的边缘案例反馈到离线基准测试套件中。
为了应对隐私和安全风险，Jarmul 警告不要依赖提示词工程来进行系统防御。她主张进行系统性的红队测试、持续的威胁建模、为自主智能体定义严格的功能范围，并提升可观测能力，以便处理更大的事件量。为了限制自主智能体行为的影响范围，Chandrasekaran 建议将每个 AI 智能体都视为一名新入职的工程师来管理。
企业应当为智能体分配独立的非人类身份，实施分级访问控制，区分只读操作与可回滚的写入操作，对第三方 MCP 依赖项进行版本管理，并且把 API 密钥严格隔离在沙箱内，脱离智能体运行环境。归根结底，工程、安全、隐私团队负责验证安全防护机制，但部署审批仍是一项战略性的业务决策。
查看英文原文：https://www.infoq.com/news/2026/10/ai-systems-production/
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
At a recent web-based seminar entitled " AI in the production environment: what is going to fail, what is going to work properly, and by whom? ", a team of senior engineering practitioners conducted an in-depth analysis of the practical challenges encountered in deploying the AI agent system. The seminar was moderated by Hien Luu and brought together Zichuan Xiong and Prem Chandrasekaran, privacy expert Katharine Jarmul and financial AI practitioner Lan Chu.
In opening the session, Luu identified three core drivers: AI's failure model, unlike traditional software, would produce seemingly unreasonable but inaccurate results rather than a direct collapse; code code AI agent generation had outpaced human coding; and sensitive data had new leaks, including prompt, context window and retrieved documents.
ThinktWorks hosting AI traffic manager Xiong highlighted the problem of “sangular drift” prevalent in the LLM stream line. He described how AI agent, a translation of SQL, frequently confused concepts in specific areas, for example, by treating “memberships” in the area of sales and in the area of customer support as the same, thereby creating imperceptible errors in downstream business analysis that could bypass conventional chain tracking.
Similarly, Chu shared a serious malfunction caused by “embedded drift” in the financial compliance system. Following the deployment of AI agent, which was based on a vector similarity threshold to intercept the irregular investment proposal, a backstage update of the bottom embedded model changed the vector distance distribution. This led to a sharp rise in the rate of misreporting and underreporting in online traffic, while all infrastructure health checks were completely green (normal).
In addition, Chandrasekaran points out that the RAG flow line may inadvertently retrieve sensitive documents, while Jarmul refers to models that essentially remember training data, which may be used for anonymous information or leaking API keys. Chandrasekaran also referred to the GitSpawn loophole, which indicates that the malicious warehouse configuration can trigger unauthorized code running in the execution of a regular order.
With the growing volume of production codes generated by code assistants, the traditional manual code review process is facing an unsustainable bottleneck. Chadrasekaran, Head of Technology and Business Services, ThoughtWorks, stressed that a line-by-line manual code review could not keep pace with the rapidly evolving AI agent output. The engineering team is required to automate tactical-level style and grammar checks in the CI/CD stream, with manual reviews focusing only on high-level structures, business intentions and non-functional requirements.
Xiong added that the team had to “left” the work and assess intent at the Harness level before the request was generated. Chandrasekaran also highlighted the resurgence of mutation testing as a key practice. By introducing deliberate code variations, teams can verify whether automated testing packages actually validate bottom logic rather than just code coverage indicators that simply satisfy the surface.
In assessing the AI agent system, Chu stressed that the system should be used as a long-term, continuous infrastructure rather than as the previous static checklist of publish. Effective assessment requires continuous online tracking, log filtering at the span level and the use of LLM to classify failure patterns, thus enabling teams to feed marginal cases in the production environment back into the offline baseline test package.
In response to privacy and security risks, Jarmul warned against relying on the prompt Project for systematic defence. She advocated systematic red team testing, ongoing threat modelling, the definition of a strict functional scope for autonomous AI agent and the enhancement of observable capabilities to deal with larger incident volumes. In order to limit the scope of the autonomous act of AI agent, Chandrasekran suggested that each AI AI agent be treated as a new engineer.
The enterprise should assign an independent non-human identity to AI agent, implement hierarchical access controls, distinguish read-only from roll-back write-up operations, manage the version of third-party MCP dependencies and strictly isolate the API key from the sandbox, away from the AI agent operating environment. Ultimately, the engineering, security and privacy teams are responsible for certifying security protection mechanisms, but deployment clearance remains a strategic operational decision.
See the original English language version at https://www.infoq.com/news/2026/10/ai-systems-prevention/
Auto-fetched and translated by AI Atlas. All rights belong to the original author.