AI 智能体如今可以生成分析代码、从对象存储读取文件，并将运维数据转为结构化报告。这是否意味着数据湖、湖仓一体和 SQL 引擎已经过时？我们在对一场大型容灾演练做自动化分析时遇到了这个问题。我们最初的可行方案是：将 30 秒粒度的服务指标以 Parquet 格式存入兼容 S3 的对象存储，使用 DuckDB 进行确定性聚合，再由 AI 智能体对故障转移结果进行分类并给出证据说明。这个原型在没有共享湖仓的情况下也能工作，但并不是因为查询引擎消失了。恰恰相反，引擎变成了嵌入在智能体工作流内部的组件。本文将讲解这个架构，介绍确定性计算与概率性推理之间的实用划分，并给出四个判断依据，用来决定什么时候用嵌入式引擎就足够，什么时候需要托管湖仓。
原型跑通之后才出现的问题
这个问题并不是头脑风暴空想出来的，而是在一个真实工作流成功产出第一份报告之后出现的。当时我们正在分析一次容灾隔离演练。演练中，一个区域被有意隔离，流量预期会转移到幸存区域。对每一个服务，我们都需要回答三个问题：
- 幸存区域是否吸收了来自被隔离区域的流量？
- 吸收流量之后，服务是否仍然可用？
- 延迟是否保持在可接受范围内，还是额外负载造成了持续性劣化？
我们以 30 秒粒度导出服务指标，存为 Parquet 文件放到兼容 S3 的对象存储中，然后让 AI 智能体生成结构化分析报告。当这个工作流端到端跑通后，一个诱人的想法就出现了：
既然智能体能从对象存储读取文件并给出答案，那为什么我们还需要表、目录、湖仓或 SQL？
这个问题听起来合乎情理，因为面向用户的交互方式已经变了。工程师不再需要打开仪表盘或者手写查询，而是直接让智能体分析演练数据。智能体会识别相关数据，按需生成 DuckDB SQL 或者 Python 代码，通过确定性工具执行分析，再解释发生了什么。举个例子，为了验证流量是否在故障转移时从 A 区转移到了 B 区，智能体会针对预设的表集合与时间范围生成一条只读的 DuckDB 查询。执行前，这条查询会经过白名单校验，仅允许访问许可的数据集与操作，随后在沙箱分析运行时中执行。运行时返回结构化查询结果，再由确定性应用代码规整成事实数据包，里面包含分区域流量、时间戳、变化百分比等字段。一条简化的示例查询如下：
generated_sql = """SELECT region, SUM(request_count) AS requestsFROM traffic_metricsWHERE ts BETWEEN :start_time AND :end_timeGROUP BY region"""validate_sql( generated_sql, allowed_tables={"traffic_metrics"}, allowed_operations={"SELECT"},)rows = sandbox.execute( generated_sql, params={ "start_time": exercise_start, "end_time": exercise_end, },)fact_packet = { "metric": "request_count", "window": [exercise_start, exercise_end], "by_region": { row["region"]: row["requests"] for row in rows },}
在这个简化示例里，智能体只负责生成分析查询。确定性校验层会在执行前拦截对白名单外的表或非只读操作的访问。查询在沙箱环境运行后，返回的数据行会由应用代码转换成结构化事实数据包。大模型拿到的是这个数据包（例如 A 区、B 区的请求量），而不是由模型自己编造或直接估算这些数值。
为什么容灾分析不只是简单扫描指标
容灾分析的难点不在于检索时序数据，而在于判断这个时间序列意味着什么。假设 A 区被隔离，B 区保持正常，一次健康的故障转移应该产生一个可识别的序列：
- A 区域的 QPS 下降。
- B 区域的 QPS 相应增加。
- B 区域的成功率保持在服务目标之上。
- 尾部延迟可能短暂上升，但不应持续居高不下。
每一个结论都需要上下文支撑。QPS 上升不等于故障转移成功，涨幅可能不足以承接隔离区域流失的流量。成功率下降也不代表容灾失败，指标里可能包含预期内的业务拒绝请求。切换瞬间的延迟尖峰也可能无害，只要系统能快速回到稳态。因此我们设置了两组基线和一个稳态窗口：
- 历史基线：前一天同一时间段，用来捕捉正常时段的流量特征。
- 即时基线：演练开始前一小时，反映服务最近的运行状态。
- 稳态窗口：切换完成后的一段时间，排除短暂过渡期，重点观察持续性表现
分析一共收集四大类核心指标：
- qps（每秒请求数）
- err_qps（错误请求数）
- success_rate（成功率）
- latency_pNN，包含 p99 这类尾部延迟指标
针对每个服务，确定性代码会计算出流量吸收率、SLO 违规情况、延迟变化、持续劣化时长等事实。智能体再结合服务、演练配置和已有证据对这些事实进行解读。这个区分至关重要：难点往往在于判断，但判断只有在底层事实是确定性的时候才可信。
架构：对象存储、嵌入式引擎与智能体
该工作流分为三个逻辑层：数据准备、确定性计算和智能体驱动的解释，并将可复现性视为硬性需求。数据准备层使用可复用的 DuckDB 构建逻辑——生成一次即可复用——把原始指标转换为对象存储中的自描述资产。对于每个分析请求，智能体选择相关输入，生成受约束的 SQL 或 Python 代码，并编排执行确定性计算。智能体解释由此产生的结构化事实包，并生成最终报告。在整个工作流中，输入数据、配置、可复用构建逻辑、确定性事实和智能体执行元数据都被保留下来，以便后续能够重建、重放和评估该分析。
图 1. AI 智能体发起并调度分析流程，确定性工具计算事实，智能体解读事实数据包。来源：作者自制示意图，由 ChatGPT 辅助绘制。
存储与自描述元数据
物理数据存放在 S3 兼容对象存储中。每次演练都有一个自描述目录：
s3://<bucket>/<root-prefix>/ index.json <exercise-id>/ manifest.json exercise_config.json metrics-*.parquet service_dimension.parquet
这些文件用途各异：
- index.json 列出可用的演练任务。
- manifest.json 记录 Schema、行数、文件大小和校验和。
- exercise_config.json 记录时间窗口、区域、阈值和演练标识。
- metrics-*.parquet 存放 30 秒粒度的指标事实数据。
- service_dimension.parquet 存放服务元数据，如归属团队和服务等级。
清单文件（manifest）很重要，一堆零散文件不等于可信的数据资产。使用者需要知道里面有什么、使用的是什么 Schema、写入是否完成、后续能不能校验文件完整性。
使用 DuckDB 进行确定性处理
存储边界的两侧都使用了 DuckDB。在摄入阶段，它读取原始指标记录、排序、写入压缩 Parquet 文件，并为 manifest 生成校验统计信息。在分析阶段，它直接读取 Parquet 文件，过滤目标时间窗口，关联服务元数据，并计算供智能体使用的事实包。该架构受益于 DuckDB 的若干特性：
- 运行在进程内，因此可以随智能体工作流一起移动，不需要单独的数据库服务。
- 直接读取 Parquet，并能把过滤和列选择下推到 Parquet 扫描中。
- 为聚合和连接查询提供 SQL 语义。
- 可以把较大操作溢出到本地存储，不需要整个工作集都留在内存中。
这并不是说 DuckDB 在所有场景下都优于 Spark、Trino 或者托管湖仓。核心原因只是：当前负载还不需要一套分布式共享查询引擎。
使用 AI 智能体进行概率性解释
智能体并不会收到数十万条原始指标，然后自己去统计错误或计算延迟均值。相反，它收到的是每个服务精简的、确定性的证据数据包。一个简化的证据包如下例所示：
{"service_id": "service-a","exercise_window": "post-cutover-steady-state","expected_failover_qps": 1200,"observed_qps_increase": 910,"traffic_absorption_ratio": 0.758,"success_rate_baseline": 0.9992,"success_rate_during_exercise": 0.9941,"service_slo": 0.9990,"p99_latency_baseline_ms": 84,"p99_latency_during_exercise_ms": 191,"sustained_degradation_minutes": 18,"data_quality_flags": []}
智能体使用这个证据包来对结果进行分类、解释支持性证据，并提出下一步排查方向。相应的输出可能是：
{"classification": "partial_failover","confidence": "high","evidence": ["The surviving region absorbed approximately 76% of the expected traffic.","Availability remained below the service SLO during the steady-state window.","The p99 latency increase was sustained for 18 minutes."],"recommended_action": "Review routing capacity and downstream dependency limits."}
精确数值由引擎算出。模型的职责是判断这些数字该如何解读、如何表述。
简单的基于阈值的规则并不总是够用。例如，在故障转移后，某个服务的 p99 延迟阈值可能暂时超标，但流量已经正常转移、可用性仍在服务目标之内。一个把任何阈值违规都视为失败的规则会把该服务判定为已劣化。而智能体可以比较过渡期窗口与稳态窗口，识别出延迟在几分钟内回到了基线，判定这是一次伴随短暂抖动的成功切换，而不是持续性容灾故障。
更重要的是，有些情况包含相互冲突的证据，无法通过单一阈值或时间规则来解决。一个服务可能吸收了预期流量并恢复了自身延迟，但由于下游依赖已接近饱和，依然存在风险。反过来，一个 Tier-2 异步服务可能出现临时积压增加，但未违反可用性目标；同样的模式若出现在 Tier-0 同步依赖中，就必须升级处理。在这些情况下，确定性层仍然产生底层事实（即流量转移、延迟、成功率、积压、依赖饱和度、服务等级和 SLO 状态），智能体将这些事实与服务上下文结合，判断哪些证据优先级更高、事件该如何归类。
因此，智能体的价值不在于取代确定性计算，而在于解决依赖上下文、相互冲突的证据——否则我们就要维护越来越多、每个服务单独定制的规则。
确定性切分
这个项目最值得复用的经验，我称之为确定性拆分原则：把精确计算和上下文推理分开。
表 1. 精确计算与上下文推理的实用划分
这种划分可以避免一种常见反模式——把 LLM 当作查询引擎。语言模型不应该读取数十万行数据并统计错误数、百分位数或平均值。这种做法速度慢、Token 消耗高，且难以复现。更重要的是，它把确定性工作交给了概率性系统。更安全的模式是：
- 用确定性引擎计算出少量权威事实。
- 智能体生成受限的 SQL / Python 代码，用来推导这些事实。
- 代码交由确定性工具执行，把结果数据包连同必要上下文交给大模型。
- 要求大模型基于数据包内字段给出结论，同时保留输入、生成代码、执行元数据、输出，用于重放和审计。
智能体可以决定要调查哪个问题、动态生成受约束的 DuckDB SQL 或 Python，并调度执行。代码的生成过程可以带有概率性，但最终数值事实必须来自确定性执行，而不是大模型凭空算出。
从工作负载看架构设计
这次演练数据集包含：
- 30 秒指标粒度
- 1127748 行事实数据
- 1523 个服务
- 约 3.8 MB 压缩 Parquet 事实数据
跨服务上游采集任务由并发 worker 执行，耗时大约一小时。由此产生的分析数据集足够紧凑，嵌入式引擎可以在靠近智能体的位置完成计算，不需要引入共享数据平台。这点很重要：架构决策要贴合真实负载，而不是看到“大数据”标签就上重型组件。一百多万行数据听起来很多，但仅凭行数并不能作为部署分布式湖仓的理由。列数、压缩方式、查询形态、并发量、访问模式、数据迁移成本都需要纳入考量。
一个典型分析示例
下面是一个基于该演练工作流的典型示例，其中服务标识和数值已脱敏。
对于某个服务，确定性执行层产出的事实包显示：幸存区域吸收了约 76% 的预期故障转移流量，可用性在稳态窗口内低于服务 SLO，p99 延迟持续偏高 18 分钟。与此同时，该服务仍然在处理请求，没有完全中断。
智能体将这些事实解释为部分故障转移，而非简单的成功或失败二元判定。它的解释引用了流量吸收率、持续的 SLO 违规和延迟劣化这几点证据，并建议检查路由容量和下游依赖限制。
随后需要结合底层确定性事实和演练时间线来验证该结论。评审者可以把智能体解释中的每一句话追溯到事实包中的字段，并使用保留的输入数据和配置重跑同样的 DuckDB/Python 分析。确认结论后，下一步动作就是排查：调查路由容量或下游瓶颈是否阻碍了幸存区域吸收全部的预期负载。
这一验证步骤很重要：智能体提出解释和下一步行动，但数值证据保持可复现、可独立验证。
让工作流可复现
只有把可复现性当成一等需求，原型才能成为正式的工程系统。每一次演练都会保存一个重放包，其中包含：
- Parquet 输入
- manifest 与校验和
- 演练配置
- 生成的 SQL 或 Python 代码
- 执行元数据
- 推导得到的确定性事实
- 模型输出
- 智能体版本与提示词
- 模型标识符
重放包可用于离线重跑分析、复核有争议的结论或者用同一套证据对比两个智能体版本的表现。我们还显式定义了数据语义。一个容易忽略的点是空值含义：某些场景下，QPS 或者错误计数为空可能代表没有流量；但成功率或延迟为空往往代表没有可供统计的流量，而不是成功率为零、延迟为零。
把所有缺失值填零会造成假失败和假恢复。Schema 和数据质量标志需要保留这种语义差异，保证引擎和智能体对数据集的解释保持一致。
轻量架构什么时候不够用
智能体加 DuckDB 并不是所有分析系统的最终架构。当工作负载具有与之相匹配的形态时它是有效的。有四个信号表明轻量架构开始遇到压力，但即使出现这些信号也并不意味着要立刻采用完整的湖仓方案。每个信号都指向一种可以逐步引入的能力，包括用于共享发现与治理的目录、用于安全并发写入和演进的开放表格式，或当计算必须靠近大型共享数据集时引入的分布式查询引擎。
独立的消费者
如果仪表盘、定时任务、笔记本或其他团队需要直接访问数据，就需要一个稳定的契约。这些消费者不应该去调用原始智能体，或者自己反向解析存储桶目录。共享目录让他们能够按表名查找、理解 Schema、配置权限，并用标准工具查询。
数据大到无法移动到智能体环境
数据达到 TB 级别之后，把文件拉到本地或者开发环境进行分析，效率非常低。查询应该在数据存储侧执行（也就是近数据处理），利用分区裁剪、谓词下推、分布式计算，只把结果传回给智能体。
工作流必须与大型共享数据集连接
当两边数据都不大时，DuckDB 可以高效连接本地文件。当另一侧是组织级权威数据集、已注册在共享目录中、太大或更新太频繁而无法安全导出时，边界就出现了。这时更稳妥的做法是把新数据注册到同一环境中，用共享引擎完成连接查询。
需要并发写入或表级演进
单写入者约定简单有效，适合早期工作流。当多个生产者更新数据集、Schema 独立演进或读取方需要一致快照时，这种方式就会失效。Apache Iceberg 等开放表格式在 Parquet 文件之上增加了表元数据、快照、Schema 演进和事务性提交语义。托管湖仓随后可以结合三项能力：
- 用于发现、归属、权限、Schema 和分区的目录；
- 用于快照、演进和安全写入的开放表格式；
- 用于分布式扫描和连接的共享查询引擎。
这些能力很有价值，但会引入集成、治理、计算和运维成本。只有在真实存在数据共享、规模、事务需求时才引入，不要默认所有 AI 项目上线第一天就需要湖仓。
实用的决策表
表 2. 根据访问模式、数据规模、写入语义选择平台
这不是 AI 与 SQL 的二元选择，也不是文件和表的二元选择。真正要判断的是：
嵌入式、由智能体驱动的分析引擎是否足够，负载是否需要托管的共享数据平台？
结论：AI 改变的是接口，而不是引擎
AI 智能体降低了数据分析的使用门槛。它们能够把业务问题转化为分析步骤，筛选相关依据、对行为进行归类，并输出工程师可直接落地的报告。但它们并不能取代精准执行的环节。
在我们的容灾分析工作流中，Parquet 和 DuckDB 完成了机械性工作：存储、过滤、聚合、校验和连接。智能体负责上下文判断：综合解释流量吸收、可用性和延迟，并输出结论。架构刻意保持轻量，因为真实负载暂时不需要共享湖仓。
跑通这套工作流之后，我们对分析交互入口的看法发生了变化。工程师不再每一次排查都直接和查询引擎交互，但引擎依然是产出可信事实的关键。智能体变成负责规划分析、生成 SQL / Python、解读证据的层。
因此，主要的架构权衡不是 AI 对 SQL，也不是文件对表，而是对于手头的工作负载，智能体到底需要多强的分析基础设施。对于聚焦的工作流，小型嵌入式引擎可能已经足够；只有当共享、规模、写入并发或治理产生具体需求时，才需要引入共享目录、开放表格式和分布式引擎。引擎一直都在，变的只是接口。
查看英文原文：https://www.infoq.com/articles/ai-changes-data-interface/
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
AI AI agent can now generate analytical codes, read files from object storage and transform traffic data into structured reports. Does this mean that the data lakes, the lake barns and the SQL engine are out of date? We encountered this problem when we did an automated analysis of a major disaster response exercise. Our initial viable option is to place 30-second-square service indicators in a Parquet format for compatible object storage, use DuckDB for definitive aggregation, and classify and provide evidence for failure transfer results by AI AI agent. The prototype also works without sharing a lake warehouse, but not because the search engine disappeared. On the contrary, the engine became a component embedded within AI agentworkflow. This paper will explain this structure, describe the functional division between definitive calculations and probability reasoning, and provide four basis of judgement to determine when it is sufficient to use embedded engines and when it is necessary to host the lake.
The problem after the prototype runs.
The problem did not come out of a brainstorm, but arose after a real workflow successful first report. At the time, we were analysing a disaster containment exercise. During the exercise, an area was intentionally isolated and the flow was expected to move to the surviving area. For every service, we need to answer three questions:
- Has the surviving area absorbed flows from the isolated area?
- Are services still available after absorbing flows?
- Are delays kept within acceptable limits or are additional loads causing continuous deterioration?
We export service indicators at a 30-second particle scale and store them as Parquet files in object-compatible S3 storage, then allow AI AI agent to generate structured analysis. When this workflow end-to-end runs, an attractive thought emerges:
Since AI agent can read files from object storage and give answers, why do we need tables, directories, lakes or SQLs?
This question sounds reasonable because the user-oriented interaction has changed. Engineers no longer need to open dashboards or handwritten queries, but direct AI agent analysis exercises. AI agent will identify the data, generate the DuckDB SQL or Python code as required, perform the analysis through a certainty tool, and then explain what happened. For example, in order to verify whether the flow has been moved from Area A to Area B in case of failure, AI agent will generate a read-only DuckDB query about the preset schedule collection and time frame. Before execution, this query is verified by a white list and only allowed access to authorized data sets and operations, and is then executed during the sandbox analysis run. Returns the results of the structured query at the time of running, which is then consolidated by a definitive application code into a fact sheet containing subregional flows, time stamp, percentage change, etc. field. A simple sample query is as follows:
generated_sql = """SELECT region, SUM(request_count) AS requestsFROM traffic_metricsWHERE ts BETWEEN :start_time AND :end_timeGROUP BY region"""validate_sql( generated_sql, allowed_tables={"traffic_metrics"}, allowed_operations={"SELECT"},)rows = sandbox.execute( generated_sql, params={ "start_time": exercise_start, "end_time": exercise_end, },)fact_packet = { "metric": "request_count", "window": [exercise_start, exercise_end], "by_region": { row["region"]: row["requests"] for row in rows },}
In this simplified example, AI agent is only responsible for generating analytical queries. The definitive verification layer intercepts access to surface or non-read-only operations outside the white list before implementation. Queries are converted from application code to structured fact packets after the sandbox environment has run. LLM was provided with the package (e.g., requests for Area A, Area B) rather than the model itself or directly estimating these values.
Why is disaster analysis more than simple scanning indicators?
The difficulty of disaster analysis lies not in retrieving time series data, but in judging what this time series means. Assuming Area A is isolated, Area B remains normal, and a healthy breakdown should produce an identifiable sequence:
- Area A QPS drops.
- Area B has a corresponding increase in QPS.
- Area B success rate remains above service target.
- The tail delay may rise briefly, but should not remain high.
Each of these conclusions needs to be supported by context. The increase in QPS does not amount to a successful transfer of malfunctions and may not be sufficient to absorb the loss of traffic in the area of separation. The decline in success rate also does not represent a failure of the disaster, and the indicators may include expected business refusals. Toggle transient delay peaks may also be harmless, as long as the system is able to return quickly to stability. So we set up two sets of baselines and a steady window:
- Historical baseline: the same period of time the previous day used to capture flow characteristics of normal times.
- Immediate baseline: one hour prior to the start of the exercise, reflecting the latest state of operation of the service.
- Steady-state window: toggle for a period of time after completion, excluding a short transition period and focusing on ongoing performance
The analysis collected four core indicators:
- qps (requested per second)
- Err qps
- Access rate (success rate)
- Latency pNN, containing p99 such end delay indicator
For each service, the certainty code calculates such facts as traffic absorption, SLO non-compliance, delayed change, and prolonged deterioration. AI agent then interprets these facts in conjunction with services, the configuration of the exercise and the evidence available. This distinction is crucial: the difficulty is often in judgement, but it is only credible when the underlying fact is certainty.
Structure: Object storage, embedded engines and AI agent
The workflow is divided into three logical layers: data preparation, certainty calculations and AI agent-driven interpretation, and replicability is considered a hard demand. The data preparation layer uses the reusable DuckDB construction logic - generating a reusable one time - to convert the original indicator into a self-descripted asset in object storage. For each analytical request, AI agent selects the relevant input, generates the bound SQL or Python code and arranges the execution certainty calculation. AI agent explains the resulting structured fact sheets and produces the final report. Throughout workflow, input data, configuration, reusable construction logic, certainty facts and AI agent execution metadata are retained so that the analysis can be reconstructed, replayed and evaluated.
Figure 1. AI AI agent initiates and arranges a process of analysis, definitive tool calculation of facts, AI agent interprets fact sheets. Source: Author's home-made diagram, supported by ChatGPT.
Storage and self-description metadata
Physical data are stored in S3 compatible object storage. Each exercise has a self-description directory:
s3://<bucket>/<root-prefix>/ index.json <exercise-id>/ manifest.json exercise_config.json metrics-*.parquet service_dimension.parquet
The use of these documents varies:
- index.json lists available exercises.
- Manual.json records Schema, line numbers, file size and checksum.
- Exercise config.json records time windows, areas, thresholds and exercise identifiers.
- Metrics-*.parquet has 30 seconds of particle size of indicator fact data.
- Service dimension.parquet stores service metadata, such as belonging to the team and service level.
Listing documents (manifest) are important, and a pile of scattered documents does not amount to a credible data asset. Users need to know what's in it, what's in it, Schema, whether the writing is complete, and whether the document's integrity can be verified later.
Use DuckDB for certainty processing
DuckDB is used on both sides of the storage boundary. At the ingestion stage, it reads the records of the original indicator, sorts it, writes the compressed Parquet file and produces statistical information for validation. During the analysis phase, it directly reads Parquet files, filters target time windows, links service metadata, and calculates fact packages for AI agent. This structure benefits from a number of DuckDB features:
- Runs within the process and can therefore move with AI agentworkflow, without separate database services.
- Read Parquet directly and push filter and column selection down to Parquet scan.
- Provides SQL syntax for aggregation and connection queries.
- Larger operations can be spilled over to local storage without requiring the entire collection to remain in memory.
That's not to say that DuckDB is better than Spark, Trino, or the Trust Lake. The central reason is simply that the current load does not require a distributed shared query engine.
Probability interpretation using AI AI agent
AI agent does not receive hundreds of thousands of original indicators and then calculates the error or delay average by itself. Instead, it receives a streamlined and conclusive evidence package for each service. A simplified portfolio of evidence shows the following examples:
{"service_id": "service-a","exercise_window": "post-cutover-steady-state","expected_failover_qps": 1200,"observed_qps_increase": 910,"traffic_absorption_ratio": 0.758,"success_rate_baseline": 0.9992,"success_rate_during_exercise": 0.9941,"service_slo": 0.9990,"p99_latency_baseline_ms": 84,"p99_latency_during_exercise_ms": 191,"sustained_degradation_minutes": 18,"data_quality_flags": []}
AI agent used this portfolio of evidence to classify the results, explain supporting evidence and provide directions for next step. The corresponding output may be:
{"classification": "partial_failover","confidence": "high","evidence": ["The surviving region absorbed approximately 76% of the expected traffic.","Availability remained below the service SLO during the steady-state window.","The p99 latency increase was sustained for 18 minutes."],"recommended_action": "Review routing capacity and downstream dependency limits."}
The exact values are calculated by the engine. The role of the model is to judge how these figures should be interpreted and presented.
Simple threshold-based rules are not always adequate. For example, the P99 delay threshold for a service may temporarily exceed the threshold following a shift in failure, but the flow is normally transferred and availability remains within the service target. A rule that treats any threshold violation as a failure would render the service inferior. AI agent, on the other hand, compares the transition window with the steady window, identifies the delay in returning to the baseline within a few minutes and concludes that this was a successful transition accompanied by a brief tremor rather than a continuous disaster failure.
More importantly, some cases contain conflicting evidence that cannot be resolved through a single threshold or time rule. One service may absorb expected flows and recover its own delays, but risks remain as downstream dependence approaches saturation. Conversely, there may be an increase in the temporary backlog of a Tier-2 walk service, without violating the goal of usability; the same model would have to be upgraded if it appeared in Tier-0 synchronized dependency. In these cases, the certainty layer still produces bottom facts (i.e., traffic transfer, delay, success rate, backlog, dependence on saturation, service grade and SLO status), which are combined with service context to determine which evidence is given higher priority and how events should be classified.
Therefore, the value of AI agent is not to replace certainty calculations, but to address the reliance on context, conflicting evidence — otherwise we will have to maintain an increasing number of rules tailored to each service.
Severity of certainty
The most valuable experience of this project is what I call the definitive split principle: to separate precise calculations from context reasoning.
Table 1.
This division avoids a common reverse pattern - using LLM as an engine for queries. Language models should not read hundreds of thousands of lines of data and count errors, percentages or averages. The practice is slow, token is high and difficult to repeat. More importantly, it gives certainty work to the probability system. Safer models are:
- Computation of a small number of authoritative facts using a definitive engine.
- AI agent produces restricted SQL/ Python codes to extrapolate these facts.
- The code is to be executed by the certainty tool and the resulting data package is to be handed over to LLM along with the necessary context.
- LLM is required to draw conclusions based on field in the data package, while retaining input, generation code, execution metadata, output for re-laying and auditing.
AI agent can determine which issue to investigate, dynamically generate the subject DuckDB SQL or Python and dispatch execution. The code generation process can be probabilistic, but the final numerical fact must come from certainty execution rather than from LLM.
View architecture design from task load
This exercise data set contains:
- 30 seconds indicator particle size.
- 1127748 Factual data.
1523 services
- About 3.8 MB Compressed Parquet Factual Data
Cross-service upstream collection missions are carried out by co-workers and take about an hour. The resulting analytical data sets are sufficiently compact to allow embedded engines to calculate in close proximity to AI agent without introducing shared data platforms. This is important: structure decision-making should fit the real load, rather than see the big data label with heavy components. More than a million lines of data sound a lot, but line numbers alone do not justify the deployment of the distributive. The number of columns, compression methods, search patterns, co-production, access patterns, and data migration costs need to be considered.
A typical analytical example
The following is a typical example based on the exercise workflow, where service identifiers and values have been dissensitized.
For a service, the fact sheet for the definitive implementation layer output shows that the surviving area absorbed approximately 76 per cent of the expected failure flow, and availability was below service SLO in a steady-state window, with a delay of 18 minutes that continued to be high. At the same time, the service is still processing requests without complete interruption.
AI agent interprets these facts as a partial breakdown transfer rather than a simple successful or unsuccessful dual determination. Its interpretation cites evidence of traffic absorption rates, persistent SLO irregularities and delays, and recommends that route capacity and downstream dependency limits be checked.
The conclusion would then need to be validated in conjunction with the underlying certainty facts and the time line of the exercise. The evaluator can trace each sentence of the AI agent interpretation to field in the fact sheet and rerun the DuckDB/Python analysis using the input data and configuration retained. Following confirmation of the conclusion, next step action is sorting out: investigating whether route capacity or downstream bottlenecks have prevented the surviving area from absorbing the full expected load.
This certification step was important: AI agent provided an explanation and next step action, but the numerical evidence was recapable and independently validated.
Let workflow come back.
The prototype will become a formal engineering system only if it is considered a first-class requirement. Each exercise saves a re-package containing:
- Parquet input
- Check and check.
- Exercise configuration.
- Generated SQL or Python code
- Execute metadata
- The certainty of the facts.
- Model output.
- AI agent version and prompt.
- Model identifier
Repackaging could be used to rerun offline, review contested findings or compare the performance of two AI agent versions with the same set of evidence. We have also defined the semantics of data in an explicit way. One easy point to ignore is the meaning of empty values: in some scenarios, QPS or errors count empty may represent no flow; but success rate or delay may often represent no flow for which statistics are available, rather than success rate being zero or zero.
Filling in all missing values would cause false failure and recovery. Schema and data quality markers need to maintain this semantic difference and ensure that engines and AI agent interpret data sets in a consistent manner.
When will the lightweight structure not be enough?
AI agent and DuckDB are not the final architecture of all the analysis systems. It is valid when the load has a pattern that matches it. Four signals indicate that light-weight structures are beginning to come under pressure, but even if they do, they do not mean that a full-scale lake silo programme should be adopted immediately. Each signal points to a capability that can be gradually introduced, including a catalogue for sharing discoveries and governance, an open-table format for safe simultaneous writing and evolution, or a distributed query engine that must be introduced when computing close to a large shared data set.
Independent consumers
If the dashboard, time assignment, notebook or other team requires direct access to data, a stable contract is required. These consumers should not be calling the original AI agent, or in turn decomposing the storage barrel catalogue. Shared directories allow them to search for, understand Schema by name, configure permission and search with standard tools.
Data is too big to move to AI agent environment
When data reach TB level, it is very inefficient to drag documents to local or development environments for analysis. Query should be performed on the side of data storage (i.e. near data processing), using partition tailoring, push-down, distributed calculations, and only sent back to AI agent.
workflow must be connected to large shared data sets
When data is not big on either side, DuckDB can efficiently connect local files. Borders arise when the other side is an authoritative data set at the organizational level, registered in a shared directory, too large or updated too frequently to be safely exported. It would then be more prudent to register new data in the same environment and use a shared engine to complete the connection search.
Requires simultaneous writing or table evolution
The single writer agrees to be simple and valid and suitable for early workflow. This approach is ineffective when multiple producers update data sets, Schema independently or readers need consistent snapshots. Open table formats such as Apache Iceberg add meta-data, snapshots, Schema evolution and transactional submission syntax to the Parquet file. The hosting lake silo could then combine three capabilities:
- Catalogues for detection, belonging, permission, Schema and partitions;
- Open-table formats for snapshots, evolutions and secure writing;
- Shared query engine for distributed scanning and connection.
These capabilities are valuable but introduce integration, governance, computing and transport costs. Only when there is a real data sharing, size, service demand, do not default that all AI projects need a lake silo on the first day on line.
Practical decision-making tables
Table 2. Selecting the platform according to access patterns, data size, writing syntax
This is not a binary choice between AI and SQL, or between files and tables. The real judge is:
Are embedded analysis engines driven by AI agent sufficient and whether loads require hosting shared data platforms?
Conclusion: AI changed the interface, not the engine.
AI AI agent lowers the threshold for data analysis. They can translate operational issues into analytical steps, screen relevant grounds, classify behaviour and export reports that engineers can land directly. But they are not a substitute for precise implementation.
In our response analysis of workflow, Parquet and DuckDB did mechanical work: storage, filtering, aggregation, validation and connection. AI agent is responsible for context judgement: a comprehensive explanation of traffic absorption, availability and delay, and output of conclusions. The structure is deliberately light because the real load does not require sharing the lake silos for the time being.
After running through this set of workflows, we changed our view of the analytical interface. Engineers no longer interact directly with the search engine for every check, but the engine remains the key to producing credible facts. AI agent became responsible for planning analysis, generating SQL/Python and interpreting evidence.
Thus, the main structural trade-off is not AI versus SQL, nor document pairs, but rather how strong the analytical infrastructure AI agent needs for the job load at hand. For focused workflow, small embedded engines may be sufficient; shared directories, open-table formats and distributed engines need to be introduced only when specific needs arise from sharing, size, writing or governance. The engine's always there, it's just an interface.
Available in English at https://www.infoq.com/articles/ai-changes-data-interface/
Auto-fetched and translated by AI Atlas. All rights belong to the original author.