人工智能
K圆圈如何用Amazon Bedrock加速DevOps事件分解到15分钟以下
这篇文章是与Circle K的Cloud DevOps工程师Marek Klatowicz共同撰写的。
许多DevOps团队都了解事件分门别类的模式：警报火灾，在任何人都能解决问题之前，一名工程师花了45-60分钟去弄清楚发生了什么。 他们抽取日志，检查服务健康情况，以及一些工具的相关症状。 所有这一切发生在补救开始之前。
在Circle K，这个挑战随着业务的展开而扩大。 该公司在27个国家经营超过17,200家便利和燃料零售店。 在店面后面，一个云土基础设施支持售点系统，忠诚应用，以及运行在AWS上的电动车辆充电应用。 随着公司扩大数字业务，分级管理费用随其增长。 更多的服务意味着更多的警报，在开始补救之前，每个部门都需要同样的人工调查。
为了解决这个问题，Circle K使用Amazon Bedrock构建了自动分拣系统，这是一种完全管理的服务，通过单一API使基础模型(FM)可用。 对于AWS区域提供的模型可用性，参考AWS区域在Amazon Bedrock中支持的模型。 系统使用代理AI架构自主调查每起事件，并在警报射击后几分钟内向Slack发送结构化摘要。 而非原始警报，待命工程师现在得到可操作的背景，所以他们可以立即行动，而不是花一个小时去弄清楚发生了什么。
K圈将平均分解时间缩短至15分钟以下
在本文中，我们描述了最初的Amazon Bedrock Flows执行架构以及向Amazon Bedrock Agent Core的迁移，并分享从系统规模化中吸取的教训。
B. 支离破碎间接费用的挑战
Circle K的DevOps团队通过多个使用Grafana的AWS账户监控基础设施。 一个典型的一天带来提醒，用于容器重启，高误差率，延迟突起，以及失败的部署。 每个警报都是一种有用的信号，但随着系统的扩大，团队希望帮助待命工程师迅速看到哪些信号需要注意，有正确的背景。
当警报发出时，分解涉及一个熟悉的人工步骤序列：审查Amazon Cloud Watch中最近的日志，检查Amazon OpenSearch Service中的应用级别痕迹，检查Amazon弹性容器服务(Amazon ECS)任务状态，以及交叉引用GitHub中内部运行本的症状。 这为工程师提供了他们需要的信息，但也表明随着更多的服务投入使用，需要进行多少重复的背景收集。
该小组认识到，积极主动的投资可以帮助两个领域。 首先，每次事件都会重复同样的日志查询、健康检查和运行本检查。 第二，分门别类的深度和速度可能因待命工程师熟悉受影响的服务而不同。 随着警示量和服务所有权的增加，每次调查的多种工具之间的切换会减缓反应速度，使小组的最佳做法难以一致应用。
因此，Circle K的DevOps团队发起了一项研究与开发(R&D)倡议，探索AI如何支持监测和事件分类。 目标是使重复性的背景收集自动化，并提供结构化的、可采取行动的结论，同时在调查期间支持工程师的业务判断。
为什么K圈选择了Amazon Bedrock
小组在建设前评价了几种方法，包括开源 agent自理基础设施框架。 这种办法将意味着拥有模式托管、即时改版、工具协调以及安全控制。
与Amazon Bedrock一起，球队避免了这一高架。 他们可以通过单一的API和互换基础模型访问基础模型，而无需重写代码。 因为Circle K已经在AWS上运行，Amazon Bedrock符合现有的AWS身份和访问管理(IAM)政策，Amazon虚拟私人云(Amazon VPC)配置，以及伐木管道。
两种管理能力包括安排和检索。 兹Amazon 贝德罗克帮助agent管弦乐，给予agents本土工具用于调用AWS API，查询数据库，并检索文档而无需自定义管弦乐代码。 Amazon Bedrock Knowledge Bases，全资管理的检索增强生成(RAG)能力，处理内部产品卡的摄入和检索，因此团队不需要单独运行一个矢量数据库。 这让DevOps团队专注于即时设计和工具定义。
数据隐私完成图片。 Circle K的事件数据包括操作度量衡，基础设施细节，以及可以包含敏感信息的日志。 兹Amazon Bedrock不与模型提供者共享客户数据，也不使用它来培训基础模型。 对于一个将生产记录和内部服务细节输入系统的小组，这种数据处理方法符合Circle K的安全要求。
解决方案概述
本节介绍团队在Amazon Bedrock Flows上建立的初始架构。 后面一节涉及向Core探员的迁移。
该溶液使用Amazon Bedrock Flows作为管弦电路层，协调一组专门的agents，各自处理分析的不同部分。 管道在Grafana警报发布Amazon简单通知服务(Amazon SNS)的通知后开始，该通知将其转发至Amazon简单排队服务(Amazon SQS)队列。 AWS Lambda对队列进行投票，提取警戒有效载荷，并引用Flow.
在流体内部，两个agents被平行引用。 与AWS组织管理的多个工作量账户的AWS资源探索者代理查询基础设施。 它检查 Amazon ECS 任务状态，应用负载平衡器目标健康，以及云表度量和日志。 同时，OpenSearchExplorerAgent查询Amazon OpenSearch Service的应用级别日志和分布痕迹。 此 agent 单独运行，因为 OpenSearch 返回了大量的日志数据。 隔离它可以控制agent的上下文窗口。 它还可以独家进入知识库，其中包含描述每项服务及其依赖性的产品卡。
在agents两人归还其发现后，GitHubexulderAgent运行。 它从受影响的服务GitHub寄存器的前两个agents和发现得到输出。 然后，它搜索它们的应用代码，基础设施定义，以及最近承诺确定问题的根源并提出具体的解决方案。 在平行的agents之后运行它意味着它确切知道要搜索什么，而不是进行广泛的查询。
所有三个输出都输入一个AnalysisPrompt节点，这是一个单一的模型调用，没有附加工具。 它收到AWS资源数据，OpenSearch日志和GitHub上下文，然后产生结构化摘要：可能的根本原因，佐证证据，受影响的服务和所有者，以及建议补救步骤。 即时输出使用一致格式，待命小组可在2分钟内扫描。
Flow)将这个摘要返回到引用Lambda函数，该函数将其发布到Slack频道。 从警报到Slack消息的整个过程需要5–15分钟，这取决于事件的复杂性。
下面的图显示这个架构结束到结束，从最初的Grafana警报到最后的Slack摘要。
示例分类
A Grafana 提醒：服务上HTTP 5xx的误差率连续3分钟超过5%.
AWSResourceExplorerAgent检查目标账户中的ECS任务健康，发现数据库连接池耗尽，ECS服务试图重启。 OpenSearchExplorerAgent 返回显示服务“拒绝连接”错误的应用程序日志到其支持的 PostgreSQL 数据库。 它还从知识库取回受影响服务的产品卡，该知识库记录其对数据库的依赖情况。 GitHubExplorerAgent搜索服务库和基础设施文件以及发现 Amazon关系数据库服务(Amazon RDS)定义。 它看到这项服务打开了每个请求的新数据库连接，而不是重新使用现有的连接。
分析Prompt得到所有三种结果。 它与在警报前10分钟开始的预定营销运动有关。 它与GitHubExplorerAgent关于推荐一个混凝土编码级固定而不是临时补丁的发现相结合。
最后的Slack消息写道：
缩放到 Amazon 贝德洛克代理 Core
Amazon Bedrock Flows 架构几个月来运作良好，简单事件在不到10分钟的时间里被三进制。 随着时间的推移，每次调查的数据量和背景都有所增加，尽管成套服务保持不变。 有些事件需要较长时间的分析，而单个流程节点则无法完成。
团队首先通过更积极的过滤日志，缩短上下文windows，将某些服务分割为单独的流程，在现有设计范围内进行了优化。 这些步骤有所帮助，但记录量和知识库继续增加。
Amazon Bedrock Agent Core满足了这一需要。 AgentCore是一个以任何框架或模型在规模上构建，连接，优化AI agents的平台。 对于这一工作量，关键能力是Core代理运行时间：全面调查在一个环节中进行，所以每个步骤都可能需要它的数据所需要的时间。 Agent Core Runtime支持长达8小时的会话，远远超出了目前调查的需要。 团队将他们的agent逻辑装箱，部署在Agent Core Runtime上。
下图显示移动到Amazon Bedrock Agent Core后的生产架构。
分析逻辑保持不变，但运行方式发生了变化。 Flows版本的3个专业agents现在被整合为在Agent Core Runtime会话中运行的单agent. Lambda函数开始会话，agent执行与前作相同的步骤。 它通过账户查询AWS资源，从Amazon OpenSearch Service提取日志，搜索GitHub，并从知识库检索产品卡。 agent在15分钟内将其发现返回兰布达函数，这在兰布达最大超时范围内。 Lambda 函数然后将结构化摘要发布到 Slack.
Amazon Bedrock Agent Core Gateway，是Amazon Bedrock Agent Core的一种能力，经纪人agent访问其数据源，包括Amazon OpenSearch Service,Amazon Bedrock知识库，以及GitHub. "Gateway"将每个源都暴露为agent引用的工具，它集中了访问控制，使得团队更容易添加或更改数据源。
Agent Core runtime 如何支持球队不断增长的规模
Amazon Bedrock Agent Core Runtime除了更长的运行时间之外，还增加了支持团队不断增长规模的三种能力：会话持久性，基于OpenTeleometry的可观察性和框架灵活性。
会话持久性：agent在整个调查期间保持中间状态。 如果第一个通过日志会露出一些东西，agent可以运行第二个，更具针对性的通道，而不会失去它已经建立的上下文。 这符合对复杂、多服务事件进行调查的迭接方式。
可观察性：Agent Core通过Cloud Watch利用OpenTeleometry标准和表面来捕捉每个会话的痕迹。 团队可以看到每个步骤需要多长时间，哪些知识库询问agent的，以及瓶颈在哪里出现。 这些洞察力直接反馈到迅速的调试中。
框架灵活性：通过Agent Core Runtime，团队可以部署使用多种框架构建的集装箱agents.
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Artificial Intelligence
How Circle K accelerated DevOps incident triage to under 15 minutes with Amazon Bedrock
This post was co-written with Marek Klatowicz, Cloud DevOps Engineer at Circle K.
Many DevOps teams know the incident triage pattern: An alert fires, and before anyone can fix the problem, an engineer spends 45–60 minutes figuring out what happened. They pull logs, check service health, and correlate symptoms across a handful of tools. All of this happens before remediation can begin.
At Circle K, this challenge scaled with the business. The company operates more than 17,200 convenience and fuel retail stores across 27 countries. Behind the storefronts, a cloud-native infrastructure supports point-of-sale systems, loyalty applications, and electric vehicle (EV) charging applications running on AWS. As the company expanded its digital operations, the triage overhead grew with it. More services meant more alerts, and each one required the same manual investigation before remediation could begin.
To address it, Circle K built an automated triage system using Amazon Bedrock, a fully managed service that makes foundation models (FMs) available through a single API. For model availability by AWS Region, refer to Supported models by AWS Region in Amazon Bedrock. The system uses an agentic AI architecture to investigate each incident autonomously and deliver a structured summary to Slack within minutes of an alert firing. Instead of raw alerts, on-call engineers now receive actionable context, so they can act immediately rather than spending an hour figuring out what happened.
Since deploying this system, Circle K has reduced mean triage time to under 15 minutes, a 67–75 percent reduction based on the team’s internal incident-response metrics.
In this post, we describe the architecture for both the initial Amazon Bedrock Flows implementation and the migration to Amazon Bedrock AgentCore, and share lessons learned from scaling the system.
The challenge of triage overhead
Circle K’s DevOps team monitors infrastructure across multiple AWS accounts using Grafana. A typical day brings alerts for container restarts, elevated error rates, latency spikes, and failed deployments. Each alert is a useful signal, but as the system expanded, the team wanted to help on-call engineers quickly see which signals needed attention, with the right context.
When an alert came in, the triage involved a familiar sequence of manual steps: reviewing recent logs in Amazon CloudWatch, examining application-level traces in Amazon OpenSearch Service, checking Amazon Elastic Container Service (Amazon ECS) task status, and cross-referencing symptoms with internal runbooks in GitHub. This gave engineers the information they needed, but it also showed how much repetitive context gathering would be required as more services were onboarded.
The team recognized two areas where a proactive investment could help. First, the same log queries, health checks, and runbook lookups would be repeated for every incident. Second, the depth and speed of triage could vary with the on-call engineer’s familiarity with the affected service. As alert volume and service ownership grew, switching between multiple tools for each investigation could slow response and make it harder to apply the team’s best practices consistently.
Circle K’s DevOps team therefore started a research and development (R&D) initiative to explore how AI could support monitoring and incident triage. The goal was to automate repetitive context gathering and provide structured, actionable findings, while supporting engineers’ operational judgment during an investigation.
Why Circle K chose Amazon Bedrock
The team evaluated several approaches before building, including open source agent frameworks on self-managed infrastructure. That approach would have meant owning model hosting, prompt versioning, tool orchestration, and security controls.
With Amazon Bedrock, the team avoided that overhead. They could access foundation models through a single API and swap foundation models without rewriting code. Because Circle K already ran on AWS, Amazon Bedrock fit into existing AWS Identity and Access Management (IAM) policies, Amazon Virtual Private Cloud (Amazon VPC) configurations, and logging pipelines.
Two managed capabilities covered orchestration and retrieval. Amazon Bedrock helped with agent orchestration, giving the agents native tool use to call AWS APIs, query databases, and retrieve documents without custom orchestration code. Amazon Bedrock Knowledge Bases, the fully managed Retrieval Augmented Generation (RAG) capability, handled ingestion and retrieval of internal product cards, so the team did not need to run a separate vector database. This let the DevOps team focus on prompt design and tool definitions.
Data privacy completed the picture. Circle K’s incident data includes operational metrics, infrastructure details, and logs that can contain sensitive information. Amazon Bedrock does not share customer data with model providers or use it to train foundation models. For a team feeding production logs and internal service details into the system, this data handling approach met Circle K’s security requirements.
Solution overview
This section describes the initial architecture, which the team built on Amazon Bedrock Flows. A later section covers the migration to AgentCore.
The solution uses Amazon Bedrock Flows as the orchestration layer, coordinating a set of specialized agents that each handle a different part of the analysis. The pipeline starts when a Grafana alert publishes a notification to Amazon Simple Notification Service (Amazon SNS), which forwards it to an Amazon Simple Queue Service (Amazon SQS) queue. An AWS Lambda function polls the queue, extracts the alert payload, and invokes the Flow.
Inside the Flow, two agents are invoked in parallel. The AWSResourceExplorerAgent queries infrastructure across multiple workload accounts managed with AWS Organizations. It checks Amazon ECS task status, Application Load Balancer target health, and CloudWatch metrics and logs. At the same time, the OpenSearchExplorerAgent queries Amazon OpenSearch Service for application-level logs and distributed traces. This agent runs separately because OpenSearch returns large volumes of log data. Isolating it keeps each agent’s context window manageable. It also has exclusive access to the knowledge base, which contains product cards describing each service and its dependencies.
After both agents return their findings, the GitHubExplorerAgent runs. It receives output from the previous two agents and finds the affected service’s GitHub repositories. Then it searches their application code, infrastructure definitions, and recent commits to pinpoint the source of the problem and propose a concrete fix. Running it after the parallel agents means it knows exactly what to search for rather than performing a broad query.
All three outputs feed into an AnalysisPrompt node, which is a single model call with no tools attached. It receives the AWS resource data, the OpenSearch logs, and the GitHub context, then produces a structured summary: probable root cause, supporting evidence, affected service and owner, and recommended remediation steps. The prompt output uses a consistent format that the on-call team can scan in under 2 minutes.
The Flow returns this summary to the invoking Lambda function, which posts it to a Slack channel. The entire process from alert to Slack message takes 5–15 minutes depending on incident complexity.
The following diagram shows this architecture end to end, from the initial Grafana alert to the final Slack summary.
Example triage
A Grafana alert: HTTP 5xx error rate on a service exceeds 5% for 3 consecutive minutes.
The AWSResourceExplorerAgent checks ECS task health in the target account and finds the database connection pool exhausted, with the ECS service attempting restarts. The OpenSearchExplorerAgent returns application logs showing “connection refused” errors from the service to its backing PostgreSQL database. It also retrieves the product card for the affected service from the knowledge base, which documents its dependency on that database. The GitHubExplorerAgent searches the service’s repository and infrastructure files and finds the Amazon Relational Database Service (Amazon RDS) definition. It sees that the service opens a new database connection per request rather than reusing existing ones.
The AnalysisPrompt receives all three results. It correlates the timing with a scheduled marketing campaign that started 10 minutes before the alert. It combines that with the GitHubExplorerAgent’s finding to recommend a concrete code-level fix rather than a temporary patch.
The final Slack message reads:
Scaling to Amazon Bedrock AgentCore
The Amazon Bedrock Flows architecture worked well for months, and simple incidents were triaged in under 10 minutes. Over time, the volume of data and the context behind each investigation grew, even though the set of services stayed the same. Some incidents required longer-running analysis than a single Flow node could complete.
The team first optimized within the existing design by filtering logs more aggressively, shortening context windows, and splitting certain services into separate Flows. These steps helped, but log volumes and the knowledge base kept growing.
Amazon Bedrock AgentCore met this need. AgentCore is a platform to build, connect, and optimize AI agents at scale with any framework or model. For this workload, the key capability was AgentCore Runtime: The full investigation runs in a single session, so each step can take the time its data requires. AgentCore Runtime supports sessions of up to 8 hours, well beyond what the current investigations need. The team containerized their agent logic and deployed it on AgentCore Runtime.
The following diagram shows the production architecture after the move to Amazon Bedrock AgentCore.
The analysis logic stays the same, but the way it runs has changed. The three specialized agents from the Flows version are now consolidated into a single agent running in an AgentCore Runtime session. The Lambda function starts the session, and the agent performs the same steps as before. It queries AWS resources across accounts, pulls logs from Amazon OpenSearch Service, searches GitHub, and retrieves product cards from the knowledge base. The agent returns its findings to the Lambda function within 15 minutes, which is within the maximum Lambda timeout. The Lambda function then posts the structured summary to Slack.
Amazon Bedrock AgentCore Gateway, a capability of Amazon Bedrock AgentCore, brokers the agent’s access to its data sources, including Amazon OpenSearch Service, Amazon Bedrock Knowledge Bases, and GitHub. The Gateway exposes each source as a tool the agent invokes, which centralizes access control and makes it simpler for the team to add or change data sources.
How AgentCore Runtime supported the team’s growing scale
Beyond longer-running sessions, Amazon Bedrock AgentCore Runtime added three capabilities that supported the team’s growing scale: session persistence, OpenTelemetry-based observability, and framework flexibility.
Session persistence: The agent keeps an intermediate state throughout an investigation. If a first pass over the logs surfaces something, the agent can run a second, more targeted pass without losing the context it has already built. This fits the iterative way that complex, multi-service incidents are investigated.
Observability: AgentCore captures traces from each session using OpenTelemetry standards and surfaces through CloudWatch. The team can see how long each step takes, which knowledge base queries the agent makes, and where bottlenecks occur. These insights feed directly into prompt tuning.
Framework flexibility: With AgentCore Runtime, the team can deploy containerized agents built with a wide range of frameworks.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.