人工智能
使用 Amazon Nova Act 加速使用代理QA 自动化软件交付 - 第三部分
随着质量保证(QA)自动化的采用在一个组织内部不断增长，一个与写作测试无关的新问题似乎已经出现。 一个平台团队站立测试自动化，然后第二个团队想要加入，然后再加入十几个。 很快，数百个测试案例都属于不同的产品和团队，每个人都可以看到其他人的测试。 没有人可以一眼就能回答一个基本问题：这个应用程序的测试覆盖率现在是否健康？测试作者不再是瓶颈组织测试和控制谁可以触摸它们成为工作。
第二个相关问题紧随其后：该应用程序及其测试由谁拥有，在覆盖范围缩小时由谁负责？没有明确的主人，健康问题就没有人可以降落。
在第一个帖子中，我们推出 QA Studio，一个用Amazon Nova Act建造的参考解决方案。 团队用它来定义自然语言的测试，并通过AI动力的视觉导航运行。 在第二篇文章中，我们增加了测试套件和一个CLI，用于连续集成和连续交付(CI/CD)管道的批量回归测试。 在这个职位上，我们覆盖两个能力帮助QA Studio 跨组织规模。 首先是应用、组织测试和报告健康指标。 第二，基于团队的用户管理，控制谁能看到和改变什么。
组织应用测试
QA Studio中的应用程序代表测试中的系统：网站，网络应用，服务前端。 是容器收集了测试、住宅和所有者。 使用案例(来自第1部分的个人测试)和测试套件(来自第2部分的回归集合)属于一个应用程序，它们继承其所有权。 每支队伍在部署过程中的每次测试中都有一个平面列表，而不是一个平面列表，而是在其拥有的应用程序内工作。
创建应用程序需要一个名称、一个基础URL以及一组可选的环境，如开发、中转和生产。 从应用程序页面中，团队直接在上下文中创建新的使用案例，添加已有的，并将其分组为测试套件。 这也是QA Studio报道其进行测试的健康状况的地方。
用衡量标准来理解应用健康
试验产生结果。 随着时间的推移，了解这些结果可以告诉一个团队应用的开发生命周期是否健康，而不只是单个建筑是否准备释放。 QA Studio将每次执行的结果记录为Amazon Cloud Watch自定义度量衡，以应用和环境为维度。 它将这些衡量标准转化为应用程序页面的健康视图。
页面顶部以三个数字解答了即时问题：选定窗口的通过率，处决总数，以及目前的失败计数。 QA Studio计算健康得分为0–100。 得分从通过率开始，减去对片状测试的处罚。 如此一来，考试通过却在通过和失败之间翻转的应用程序，一个团队扫描一组应用程序，就可以分辨出那些需要注意的应用程序，而无需打开每个结果。
度量衡标签从当前快照移动到趋势。 团队选择一个每小时颗粒度的7天窗口，或者每天颗粒度的30天窗口，QA Studio则绘制该期间的处决，成功，失败以及平均持续时间。 由于基本尺度具有环境维度，团队将视野过滤到单一环境。 例如，它们可以比较同一套试验的生产相比的组装行为。
两种观点针对最需要团队成本的测试。 片面测试通过过去七天的通过和失败之间翻转的频率来查看案例。 这揭示了间歇性故障，这些故障侵蚀了对套房的信任，然后就变成了生产失误。 近期的失败记录了最近因快速分尸而失败的处决。 每个都链接到第1部分描述的Amazon Nova Act轨迹记录和会话录音。 这样，从“这项测试失败”到“agent所看到的就是”只需一击。
与团队和角色控制接触
量子告诉一个团队他们的应用程序是如何做的。 基于团队的用户管理层首先决定哪些应用程序是他们的。 QA Studio在团队周围组织访问，一个应用程序完全属于一个团队。 该团队的成员处理其应用程序。 其他人都看不到他们。
每个团队成员都担任着五个角色之一，每个角色包括下面各角色的许可：
- 管理员：整个工作室访问每个团队、应用程序和用户，包括创建团队和分配应用程序。 Admin是一个全工作室的角色，而不是每个团队的任务。
- 团队领队：管理自己的团队成员，并为团队创建应用程序。
- 编辑：在团队应用程序中创建和编辑案例和测试套件。
- 执行者：运行使用大小写和套件，但不修改。
- 查看器：只读访问团队的应用程序和结果。
由于角色是累积的，一个Team Lead拥有编辑器拥有的一切，等等下到 Viewer. 使用例或测试套件的获取遵循包含该套件的应用程序。 如果一个成员可以从事申请工作，他们可以从事其测试工作，以他们的作用为界限。 这使许可管理保持在应用程序一级，而不是在数千次个人测试上。
一个管理员看到整个工作室。 用户页面列出每个账户及其团队和角色在一个地方。 管理人员可以知道谁拥有团队领导权，谁拥有全工作室管理员的权利。 在此处显示的参考部署中，四个用户跨越两个小组：两个小组领导、一个查看员和一个全工作室管理员。 团队的页面逐组滚动相同数据，显示每个团队的成员数和应用程序数并列。 管理员可以在打开其中任何一个队伍之前看到哪些队伍正在扩张。
由各小组整理的同一成员数据见各小组的网页。
从团队切换到应用程序，整个工作室的应用程序列出了所有团队的每个应用程序的显示。
团队间共享应用程序
团队很少完全孤立地工作。 一个平台团队可能拥有一个功能团队需要运行对抗的应用程序，而无需交出更改它的能力。 QA Studio通过共享访问支持这一点：自有团队在视图，执行或完整关卡允许另一团队访问一个应用程序。
共享使用遵循上限规则。 用户对共享申请的有效许可是其自身角色和补助金水平的较低。 一个被允许完全访问共享应用程序的查看器仍然只查看，因为他们的角色是上限。 这样一来，一个拥有的团队就可以在不给予一个接受者所允许的角色的情况下，广泛分享。
一队人所见
一个团队领导在单个团队而不是整个工作室内工作。 " 开放我的团队 " 提供了四个标签，涵盖所有主要控制：成员、应用程序、使用案例和共享访问。 成员列表列出团队中的人。 领导可以改变任何成员的角色或增加一个新的角色，因此日常访问管理会留在最接近工作的人身边。 在此处显示的AWS团队中，领队管理两名成员，一名团队领队和一名查看员。
应用程序标签显示仅包含团队拥有的应用程序，每个应用程序都有其测试大小写计数和最后运行时间。 这使得主角一直专注于自己的表面而不是完整的工作室库存。 共享访问标签是跨团队共享成为混凝土的地方。 在此，AWS Team已经允许AMZN Team查看一个申请，领先者可以批准，撤销，或改变访问级别。 由于赠款是收视水平，一个AMZN团队编辑仍然认为共享应用程序只读：上一节的上限规则，在实践中适用。
同一团队的共享访问标签记录了其他团队可以到达团队的应用程序，以及哪个级别。
如何运作
QA Studio在之前的帖子中描述的同一无服务器基础上构建基于团队的访问控制和度量衡，在混音中添加了一些AWS服务：
- Amazon Cognito：每个团队向一组Cognitto组地图，每个角色一个。 AWS Lambda授权人每次请求都会将呼叫者团体解析为API范围和团队背景。
- Amazon DynamomDB：团队，成员，应用程序所有权，以及跨团队共享赠款存储在现有的单表设计中。
- Amazon Cloud Watch：每个执行都发出QAStudio/Executions命名空间中自定义的度量衡，以应用和环境为维度。 衡量标准 API 读回它们来构建健康观点和趋势图。
- AWS Lambda和Amazon API Gateway：后端在两个层次上强制访问：一个是呼叫者角色授予的范围，另一个是确认呼叫者团队拥有或已经与应用程序共享的资源检查。
由于访问来源于Cognitto集团会员，因此OAuth客户端证书从第二部分流出，符合同一模式。 CI输油管认证为机体身份，具有所需的瞄准镜，不再使用。
开始
应用程序，度量衡和基于团队的访问控制是QA Studio参考解决方案的一部分，您在自己的AWS账户中用AWS Cloud Development Kit(AWS CDK)部署。 所有测试数据，结果，和度量标准都停留在您的账户范围内。 部署后，管理员会创建团队，指派成员和角色，并向团队分配应用程序。 每个团队的健康标准随着测试的进行而开始流行。
关于部署指令和配置细节，参见QA Studio GitHub寄存器及其文档。
结论
在这个文章中，我们展示了QA Studio如何将代理QA自动化从个人测试到组织。 申请给每个团队一个使用案例和测试套房的家。 他们通过通过通过率、趋势图、微分调整后的健康分数以及基于Amazon Cloud Watch测量的故障分数来报告健康状况。 基于团队的用户管理，在Amazon Cognitto的支持下，让每个团队在自己的应用程序内工作，跨团队共享以明确的上限规则为限制。
这些能力建立在第一编自然语言测试作者以及第二编套件和CI/CD集成的基础上。 它们一起帮助QA自动化在众多团队中成长而不会失去知名度或控制。 为了在自己的环境中尝试，从QA Studio GitHub寄存器中部署参考解决方案，创建你的第一个团队和应用程序，并运行几个使用的案例来观察度量衡的形状。 要进一步了解背后的agent，参见Amazon Nova Act产品页。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Artificial Intelligence
Accelerating software delivery with agentic QA automation using Amazon Nova Act – Part 3
As quality assurance (QA) automation adoption grows inside an organization, a new problem appears that has nothing to do with writing tests. A platform team stands up test automation, then a second team wants to join, then a dozen more. Soon hundreds of test cases belong to different products and squads, and everyone can see everyone else’s tests. No one can answer a basic question at a glance: is this application’s test coverage healthy right now? Test authoring stops being the bottleneck. Organizing tests and controlling who can touch them becomes the work.
A second, related question follows close behind: who owns this application and its tests, and who is accountable when coverage slips? Without a clear owner, health problems have no one to land on.
In the first post, we introduced QA Studio, a reference solution built with Amazon Nova Act. Teams use it to define tests in natural language and run them through AI-powered visual navigation. In the second post, we added test suites and a CLI for batch regression testing in continuous integration and continuous delivery (CI/CD) pipelines. In this post, we cover two capabilities that help QA Studio scale across an organization. The first, applications, organizes tests and reports health metrics. The second, team-based user management, controls who can see and change what.
Organizing tests with applications
An application in QA Studio represents the system under test: a website, a web app, a service front end. It’s the container that gives a collection of tests, a home, and an owner. Use cases (the individual tests from Part 1) and test suites (the regression collections from Part 2) belong to an application, and they inherit its ownership. Instead of one flat list of every test in the deployment, each team works inside the applications they own.
Creating an application takes a name, a base URL, and an optional set of environments such as development, staging, and production. From the application page, teams create new use cases directly in context, add existing ones, and group them into test suites. This is also where QA Studio reports on the health of the tests it runs.
Understanding application health with metrics
Tests produce results. Understanding those results over time is what tells a team whether the development lifecycle for an application is healthy, not only whether a single build is ready for release. QA Studio records the outcome of every execution as Amazon CloudWatch custom metrics, dimensioned by application and by environment. It turns these metrics into a health view on the application page.
The top of the page answers the immediate question with three figures: the pass rate over the selected window, the total number of executions, and the current failure count. Alongside them, QA Studio computes a health score of 0–100. The score starts from the pass rate and subtracts a penalty for flaky tests. That way, an application whose tests pass but flip between pass and fail doesn’t present a falsely perfect picture. A team scanning a portfolio of applications can tell the stable ones from the ones that need attention without opening each result.
The metrics tab moves from the current snapshot to trends. Teams choose a 7-day window with hourly granularity or a 30-day window with daily granularity, and QA Studio charts executions, successes, failures, and average duration across that period. Because the underlying metrics carry an environment dimension, teams filter the view to a single environment. For example, they can compare how staging behaves against production for the same set of tests.
Two views target the tests that cost teams the most time. The flaky tests view ranks use cases by how often they flipped between passing and failing over the last seven days. This surfaces the intermittent failures that erode trust in a suite before they turn into a production miss. The recent failures view lists the latest failed executions for quick triage. Each one links back to the Amazon Nova Act trajectory logs and session recordings described in Part 1. That way, moving from “this test is failing” to “here is exactly what the agent saw” takes one click.
Controlling access with teams and roles
Metrics tell a team how their applications are doing. Team-based user management decides which applications are theirs in the first place. QA Studio organizes access around teams, and an application belongs to exactly one team. Members of that team work with its applications. Everyone else doesn’t see them.
Every team member holds one of five roles, and each role includes the permissions of the ones below it:
- Admin: Studio-wide access to every team, application, and user, including creating teams and assigning applications. Admin is a studio-wide role rather than a per-team assignment.
- Team Lead: Manages their own team members and creates applications for the team.
- Editor: Creates and edits use cases and test suites within the team’s applications.
- Executor: Runs use cases and suites but doesn’t modify them.
- Viewer: Read-only access to the team’s applications and results.
Because roles are cumulative, a Team Lead has everything an Editor has, and so on down to Viewer. Access to a use case or a test suite follows the application that contains it. If a member can work with the application, they can work with its tests, bounded by their role. This keeps permission management at the application level instead of on thousands of individual tests.
An administrator sees the whole studio. The Users page lists every account with its team and role in one place. An admin can tell who holds Team Lead on which team and who carries Studio-wide Administrator rights. In the reference deployment shown here, four users span two teams: two Team Leads, one Viewer, and one Studio-wide Administrator. The Teams page rolls the same data up by team, showing each team’s member count and application count side by side. An admin can see which teams are expanding before opening any one of them.
The same membership data, rolled up by team, appears on the Teams page.
Switching from teams to applications, the studio-wide Applications list shows every application across all teams.
Sharing applications between teams
Teams rarely work in complete isolation. A platform team might own an application that a feature team needs to run against, without handing over the ability to change it. QA Studio supports this with shared access: the owning team grants another team access to an application at a view, execute, or full level.
Shared access follows a ceiling rule. A user’s effective permission on a shared application is the lower of their own role and the level of the grant. A Viewer who is given full access to a shared application remains view-only, because their role is the ceiling. This way, an owning team can share widely without granting more than a recipient’s role allows.
What a team lead sees
A team lead works inside a single team rather than the whole studio. Opening My Team presents four tabs that cover everything the lead controls: Members, Applications, Use Cases, and Shared Access. The Members tab lists the people on the team. The lead can change any member’s role inline or add a new one, so day-to-day access management stays with the person closest to the work. In the AWS Team shown here, the lead manages two members, one Team Lead and one Viewer.
The Applications tab shows only the applications the team owns, each with its test-case count and last-run time. This keeps the lead focused on their own surface rather than the full studio inventory. The Shared Access tab is where cross-team sharing becomes concrete. Here the AWS Team has granted the AMZN Team view access to an application, and the lead can grant, revoke, or change the level of that access. Because the grant was view level, an AMZN Team Editor still sees the shared application read-only: the ceiling rule from the previous section, applied in practice.
The same Team Lead’s Shared Access tab records which other teams can reach the team’s applications, and at what level.
How it works
QA Studio builds team-based access control and metrics on the same serverless foundation described in the earlier posts, adding a few AWS services to the mix:
- Amazon Cognito: Each team maps to a set of Cognito groups, one per role. An AWS Lambda authorizer resolves a caller’s groups into API scopes and team context on every request.
- Amazon DynamoDB: Teams, memberships, application ownership, and cross-team share grants are stored in the existing single-table design.
- Amazon CloudWatch: Every execution emits custom metrics in the QAStudio/Executions namespace, dimensioned by application and environment. The metrics API reads them back to build the health view and trend charts.
- AWS Lambda and Amazon API Gateway: The backend enforces access at two layers: the scope a caller’s role grants, and a resource check that confirms the caller’s team owns or has been shared with the application.
Because access is derived from Cognito group membership, the OAuth client credentials flow from Part 2 fits the same model. A CI pipeline authenticates as a machine identity with the scopes it needs and no more.
Getting started
Applications, metrics, and team-based access control are part of the QA Studio reference solution, which you deploy in your own AWS account with the AWS Cloud Development Kit (AWS CDK). All test data, results, and metrics stay within your account boundaries. After deploying, an administrator creates teams, assigns members and roles, and assigns applications to teams. Each team’s health metrics begin populating as their tests run.
For deployment instructions and configuration details, see the QA Studio GitHub repository and its documentation.
Conclusion
In this post, we showed how QA Studio scales agentic QA automation from individual tests to an organization. Applications give every team a home for their use cases and test suites. They report health through pass rate, trend charts, a flaky-adjusted health score, and failure triage built on Amazon CloudWatch metrics. Team-based user management, backed by Amazon Cognito, keeps each team working within the applications they own, with cross-team sharing bounded by a clear ceiling rule.
These capabilities build on the natural language test authoring from Part 1 and the suites and CI/CD integration from Part 2. Together, they help QA automation grow across many teams without losing visibility or control. To try it in your own environment, deploy the reference solution from the QA Studio GitHub repository, create your first team and application, and run a few use cases to watch the metrics take shape. To learn more about the agent behind it, see the Amazon Nova Act product page.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.