代理工作量打破了软件测试的假设。 这是如何应对
大部分传统企业系统都是围绕三个假设构建的：乔布斯快速完成，重试一个是免费的，同样的输入总是产生相同的输出。
代理工作量打破了所有这些预期，这就是为什么业绩良好的飞行员一旦无人照顾就会变成操作问题。 困难很少是模式；承担这些工作量不再拥有的财产的是周围的基础设施和管理做法。
改变了什么
Agents工作规则与常规软件不同。 有三个重大区别：
工作需要几分钟，而不是毫秒。 代理任务可以持续足够长的时间，以超过您堆栈中从未遇到过的超时阈值。 看上去很稳定的系统以神秘的方式开始失败直到有人检查持续时间
报复现在要花钱传统上，重试逻辑几乎是免费的，因此球队重试慷慨。 但是，每次针对一个计算模型的尝试都会消耗计算结果是否可用的资源。 松散的质量阈值和自动重试相结合，创造了一个预算事件。 由于云和模型应用编程接口的使用在每月周期中同步计费，这些复杂的重试费用会悄悄积累起来，只有在发票到达数周后才能显现出来。
不能重复失败。 这是最大的变化。 当工程师调查不良结果时，标准做法是重新运行，再看一次失败。 当agents参与时，没有agent决定的分步追踪，工具呼叫和应用程序编程接口运行，就没有什么可调查的，审查成为猜测。
为何飞行员要隐藏这一切
经验丰富的团队被抓住是因为评价环境掩盖了这一切。
交互工作时，人类是处理错误的人。 他们读每个结果，注意问题，再试一次。 成本是显而易见的，因为尝试是通过人工计算和解决的。 不需要审计记录或复制准确失败的方法。
相比之下，自动化的agent工作流程无头无尾地运行，有可能出现未记录的故障，导致下游系统破裂。 当一个人与AI聊天并检查每个反应时，一个工作流程表现得可靠，当一个调度员一夜之间发射400次，没有人观看时，工作流程表现得不同。 错误总是存在，但它是隐形的，因为一个人类操作员一次一次吸收并纠正了它。
在使用agents进行自动化生产之前，工程领导必须确定人类手动完成的每一项任务，并明确规定由哪个自动检查或系统接管这一责任。
三个值得你注意的风险
无形成本。 支出不再只是使用量的函数。 自主AI循环自动重试失败的任务，重新生成响应，并在没有人类干预或批准的情况下反复打击API. 一个开发商设定的门槛可以比采购谈判更能移动月账单。 询问每个完成单元的费用，而不是API呼叫的费用，因为第二个数字不包括被放弃的尝试。
失败报告成功。 最昂贵的缺陷不是错误，而是似乎在结构上有效的结果，但实质上是错误的。 概率agents无法识别自己的逻辑错误，下游自动系统接受并处理它而不触发任何警报。 常规监测错过了这些错误，因为没有失败。 AI输出质量必须使用预先定义的方案标准(如断言检查，LLM-as-a-judge规则或语义基准测试)来衡量，而不是依赖标准系统运行时间或错误日志。
这件事没人能解释如果你的团队无法说出哪个模型版本，投入和设置产生了特定的输出，他们无法调查，审计师也无法调查。 这种信息在工程进行期间是廉价的，而且几乎无法在以后重建。
问你的球队什么
5个问题涉及上述大多数情况：
- 哪些定义可以接受的输出，在工作进行之前是否写下来？自动化需要明确，可复制的验收标准。 如果审查期间根据个人个人意见而改变成功标准，软件无法自动验证产出。
- 一个典型的完成单位做了多少次尝试，是否封顶？对模糊标准进行无约束的重试循环是造成超支的最常见来源
- 我们每跑一次都要录什么？六个月后我们能否按要求制作记录？工作数据应包括准确的输入有效载荷、即时/模型版本、时间戳、执行间隔、重试计数、象征性费用和最后产出文物。
- 由谁批准各类产出？不是团队，是个人当出问题时，这是第一个要问的问题。
- 当供应商更新模型时会发生什么？模型更新可以潜移地改变响应格式，准确性或推理逻辑。 这些下游班次可以打破自动管道，这就是为什么在投入生产之前，模型版本的修改必须在中转环境中进行测试。
合并有帮助的地方
由于团队使用传统软件做法管理代理工具，代理工作倾向于分散。 便捷在一队的存放处现场直播，文物在各种桶里
理想的AI工作空间是一个将运行，输入，输出，质量结果和批准记录在一起的地方，作为一个业务表面发挥作用，使团队能够积极管理执行，重新运行投入，跟踪日志以及实时控制agent工作流程。 这个学科不仅仅是产品决策报告和记录必须在每次执行中持续进行，而不是定期进行，因此各小组可以实时追踪异常情况并保持可审计性。
这些同样的问题决定了哪个平台会锁定你的AI工作空间。 平台能否报告产出的产生情况？能从保存的输入中重新执行吗？它是否将失败与拒绝与退化的结果区分开来？服务于生产管道的供应商，包括ImagineArt等平台，越来越多地暴露了这一点。 一个抛光产品演示显示解析，最好的情况结果，但是它没有揭示AI模型在自动操作时如何处理边缘案例，出乎意料的错误或持续的长期任务。
技巧对话
一种模式是很容易错过规划。 团队构建代理特性往往来自应用开发，其中同步请求和响应模式是规范。 这些工作量表现得像数据管道：长期运行，部分失败，重新执行费用昂贵。
如果球队中没有任何人之前运行过这类系统，那么技能差距显示会上升为一系列令人惊讶的事件，而不是表明技能需要升级。 技巧对话比较便宜。
Agentic AI没有造成新的一类操作问题。 它消除了几十年来存在的假设，大多数事件追溯到这些进程差距，而不是模拟质量。
定义工作运行前的正确含义。 计量每个完成单位的费用。 足够调查你无法复制的东西让名人对什么船负责没有什么是异国情调的，但都不是你现在的堆栈所要做的。
M. Touheed是《想象艺术》的成长专家。 他为SiliconANGLE写了这篇文章。
图片：SiliconANGLE/ChatGPT
SiliconANGLE联合创始人John Furrier的留言：
支持我们的使命，通过与CUBE社区接触，保持内容的开放和自由。 加入CUBE的校友信托网络，其中技术领导者相互连接，分享情报，创造机会。
- 15M+观看CUBE视频，为AI、云层、网络安全等各种对话提供动力
- 11.4k+CUBE校友——通过独特的信任网络与11400多名技术和企业领袖建立联系，塑造未来
你是AWS客户吗？在财务上支持SiliconANGLE，从我们的市场门户页面和链接购买您的AWS服务：https://siliconangle.com/aws-marketplace/.
关于 SiliconANGLE 媒体
SiliconANGLE Media由技术视觉家约翰·弗瑞尔(John Furrier)和戴夫·韦兰特(Dave Vellante)创立，构建了产业领先数字媒体品牌的动态生态系统，达到1500+百万精英技术专业人士。 利用CUBEAi.com神经网络，帮助科技公司以数据为导向做出决策
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Agentic workloads break assumptions about software testing. Here’s how to cope
Most traditional enterprise systems were built around three assumptions: Jobs finish quickly, retrying one is free and the same input always produces the same output.
Agentic workloads break all three of these expectations, which is why pilots that perform well turn into operational problems once they run unattended. The difficulty is rarely the model; it’s the surrounding infrastructure and management practices that assume properties these workloads no longer have.
What has changed
Agents work by different rules than conventional software. Here are three significant differences:
Work takes minutes, not milliseconds. An agentic task can run long enough to exceed timeout thresholds nothing in your stack has encountered before. Systems that seemed stable start failing in ways that look mysterious until someone checks the duration.
Retries now cost money. Traditionally, retry logic has been nearly free, so teams retry generously. But every attempt against a metered model consumes compute resources whether the result is usable or not. The combination of loose quality thresholds and automatic retries creates a budget event. Because cloud and model application programming interface usage is billed asynchronously on monthly cycles, these compounded retry costs quietly accumulate and become visible only when the invoice arrives weeks later.
Failures cannot be reproduced. This is the biggest change. When an engineer investigates a bad result, the standard practice is to rerun it and watch it fail again. That doesn’t work when agents are involved. Without a step-by-step trace of agent decisions, tool calls and application programming interface runs, there is nothing to investigate and reviews become speculation.
Why pilots hide all of this
Experienced teams get caught because the evaluation environment conceals all of this.
When working interactively, humans are the error handlers. They read each result, notice problems and try again. Costs are visible because attempts are counted and fixes applied by hand. There’s no need for an audit record or a way to reproduce the exact failure.
In contrast, automated agent workflows run headlessly, with the potential for unrecorded failures to break downstream systems. A workflow that behaved reliably when a person would chat with AI and check each response behaves differently when a scheduler fires it 400 times overnight with nobody watching. The error was always there, but it was invisible because a human operator absorbed and corrected it one response at a time.
Before moving to automated production with agents, engineering leaders must identify every task the human was performing manually and specify which automated check or system will take over that responsibility.
Three risks worth your attention
Invisible costs. Spending is no longer a function of usage volume alone. Autonomous AI loops automatically retry failed tasks, regenerate responses and hit APIs repeatedly without human intervention or approval. A threshold set by a developer can move the monthly bill more than a procurement negotiation. Ask for the cost per completed unit rather than cost per API call, since the second number excludes discarded attempts.
Failures that report success. The most expensive defects are not errors but results that appear structurally valid but are substantively wrong. Probabilistic agents can’t recognize their own logical errors, so they produce correctly formatted outputs with bad data that appears structurally valid. Downstream automated systems accept and process it without triggering any alerts. Conventional monitoring misses these errors because nothing failed. AI output quality must to be measured using predefined programmatic criteria (such as assertion checks, LLM-as-a-judge rules or semantic benchmarks) rather than relying on standard system uptime or error logs.
Incidents nobody can explain. If your team cannot say which model version, inputs and settings produced a specific output, they cannot investigate it and neither can an auditor. That information is cheap to capture while the work runs and close to impossible to reconstruct later.
What to ask your team
Five questions address most of the scenarios just described:
- What defines acceptable output, and is it written down before the work runs? Automation requires clear, reproducible acceptance criteria. If success criteria change based on individual human opinion during review, software cannot validate the output automatically.
- How many attempts does a typical completed unit take, and is that capped? An unbounded retry loop against a vague standard is the most common source of overspending
- What do we record for every run, and could we produce that record on request in six months? Job data should include the exact input payload, prompt/model version, timestamp, execution latency, retry counts, token cost and the final output artifact.
- Who approves various classes of output, by name? Not a team, an individual. When something goes out wrong, that is the first question that will be asked.
- What happens when the provider updates the model? Model updates can subtly alter response formatting, accuracy or reasoning logic. These downstream shifts can break automated pipelines, which is why model version changes must be tested in a staging environment before being deployed to production.
Where consolidation helps
Agentic work tends to be scattered because teams manage agentic tools using traditional software practices. Prompts live in one team’s repository, artifacts are in various buckets, approvals happen in chat threads and nobody can produce an execution record without an afternoon of archaeology.
The desired AI workspace is one place where runs, inputs, outputs, quality results and approvals are recorded together, functioning as an operational surface that allows teams to actively manage execution, rerun inputs, trace logs and control agent workflows in real time. This is a discipline more than a product decision. Reporting and logging must be continuous across every execution run rather than periodic, so teams can trace anomalies and maintain auditability in real time.
These same questions decide which platform anchors your AI workspace. Can the platform report on what produced an output? Can it re-execute from a saved input? Does it distinguish a failure from a refusal from a degraded result? Vendors serving production pipelines, including platforms such as ImagineArt, increasingly expose this. A polished product demo shows curated, best-case results, but it doesn’t reveal how the AI model handles edge cases, unexpected errors or continuous long-running tasks when operating automatically.
The skills conversation
One pattern is easy to miss in planning. Teams building agentic features often come from application development, where synchronous request-and-response patterns are the norm. These workloads behave like data pipelines: long-running, partially failing and expensive to re-execute.
If nobody on the team has run these kinds of systems before, the skills gap shows up as a series of surprising incidents rather than an indication that skills need to be upgraded. The skills conversation is cheaper.
Agentic AI did not create a new class of operational problem. It removed assumptions that have existed for decades, and most incidents trace back to those process gaps rather than to model quality.
Define what correct means before the work runs. Measure cost per completed unit. Record enough to investigate what you cannot reproduce. Keep a named person accountable for what ships. None of it is exotic, but none of it is what your current stack was built to do.
M. Touheed is a growth specialist at Imagine Art. He wrote this article for SiliconANGLE.
Image: SiliconANGLE/ChatGPT
A message from John Furrier, co-founder of SiliconANGLE:
Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.
- 15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity and more
- 11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network
Are you an AWS customer? Support SiliconANGLE financially by buying your AWS services from our Marketplace portal page and links: https://siliconangle.com/aws-marketplace/
About SiliconANGLE Media
Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.