Spotify 扩大了 AI 辅助开发的应用范围，8 月份合并变更量同比增长一倍以上，从约 8100 个增至 17000 个。该公司正在调整质量与可靠性控制体系，以应对生产系统中激增的软件变更数量。
Spotify 运营着近 3000 个生产服务，支撑着 7.77 亿月活用户，约 1 亿并发客户端，每秒处理 1100 万至 1200 万次后端请求。在复盘工作中发现了内容处理、集群变更、计算容量以及移动端应用质量方面的问题。对此，Spotify 对监控、测试、发布防护、回滚能力、容量规划和质量度量进行了优化。
其中一次事故与 Spotify 的内容处理流水线有关，该流水线每天处理超 50 万首新增歌曲、视频、播客及有声书。6 月 24 日，一个计划内的定时批处理任务与新节目内容抢占资源，一项质量优化提升了处理资源需求，而一个调度错误使吞吐量降低了约 10%。原本几分钟内就能上线的节目，延迟了数小时。Spotify 后续新增端到端监控、修复调度器、调低批处理任务优先级、重构任务优先级策略并进行了扩容。
集群自动化带来了不一样的挑战。Spotify 的集群管理系统已合并超过 250 万个自动化维护拉取请求，其中大部分在安全检查后自动合并。公司后来引入了 Honk，这是一个在 Kubernetes Pod 中使用 Agent SDK 运行 Claude 的后台编码智能体。Fleetshift 负责目标选择、调度和进度跟踪，Honk 完成代码修改并可在 CI 中执行构建。Spotify 报告称，最近其后端服务的一次 Java 迁移耗时三天。
不过 Spotify 表示，一次自动化依赖升级虽然通过了现有校验，却在生产环境发生故障。公司正在强化防护手段、扩大回滚能力，并将自动化变更安排在负责团队的工作时段执行。
JumpCloud 软件工程师 Amit Jha 在 LinkedIn 发帖讨论 Spotify 的复盘时提到：
更多的变更意味着测试、评审和可观测性面临更大压力。
Alex Beck 在 LinkedIn 帖子中同样聚焦验证难题：
编写代码变得更容易了。现在的难点是确保代码上线前质量达标。
这些发现为谷歌 2025 年 DORA 研究提供了背景：该研究发现，采用 AI 会提升软件交付吞吐量与产品性能，但同时和交付稳定性呈负相关。研究指出，自动化测试、版本控制与快速反馈闭环是在引入 AI 时管理软件交付的关键实践。
Spotify 将标准化维护交给集群管理系统，Honk 用于复杂的代码变更，Fleetshift 负责编排，CI 承担验证工作。质量和优化工作占合并变更的比例从 27% 上升至 31%，而维护和配置工作从 31% 下降至 25%。Spotify 报告称返工量并未随之增长，但拉取请求体量与代码复杂度正在上升。事件审查发现 AI 编写的代码没有造成实质性的直接影响，但变更数量已经超过了一些验证机制的处理能力。
查看英文原文：https://www.infoq.com/news/2026/10/spotify-ai-quality-controls/
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Spotify expanded the scope of AI-aided development applications, and combined changes more than doubled in August, from approximately 8100 to 17,000. The company is in the process of adjusting its quality and reliability control system to respond to the surge in software changes in the production system.
Spotify operates nearly 300 production services, supporting 777 million live monthly users, with about 100 million simultaneous clients processing 11 million to 12 million back-end requests per second. Problems in content processing, cluster change, computing capacity and the quality of mobile-end applications were identified during the back-up exercise. In response, Spotify optimizes surveillance, testing, publish protection, rollback capacity, capacity planning and quality measures.
One of the incidents was related to Spotify's content-processing line, which handles more than 500,000 new songs, videos, podcasts and sound books daily. On 24 June, resources were seized from a planned scheduled batch processing mission and new programme content, and a quality optimization improved resource requirements for processing, while a scheduling error reduced throughput by about 10 per cent. The programme, which could have been online within minutes, was delayed for hours. Spotify added end-to-end monitoring, repair of dispatchers, reprioritization of batch processing tasks, re-engineering of mission priority strategies and expansion.
Cluster automation presents different challenges. Spotify has consolidated over 2.5 million automated maintenance requests, most of which are automatically consolidated after security checks. The company later introduced Honk, a backstage code AI agent for using Agent SDK running Claude in Kubernetes Pod. Fleetshift is responsible for targeting, scheduling and tracking progress, and Honk completes code changes and can execute construction in CI. Spotify reported that a recent subsequent post-service migration of Java took three days.
However, Spotify states that an automated build-up, although currently validated, has failed in the production environment. The company is in the process of enhancing protection, expanding its roll-back capacity and placing automated changes in the working hours of the team.
Amit Jha, a JumpClaud software engineer, mentioned in LinkedIn's re-discussion that:
More changes mean greater pressure on testing, evaluation and observability.
Alex Beck, in LinkedIn post, also focused on the challenge:
It's easier to write codes. The difficulty now is to ensure that the quality is achieved before the code goes online.
These findings provided the background for the 2025 Google DORA study, which found that the adoption of AI would enhance the delivery of software throughput and product performance, but at the same time would be counterproductive to delivery stability. The study noted that automated testing, version control and fast feedback closed loops were key practices in managing software delivery when AI was introduced.
Spotify transferred standardized maintenance to the cluster management system, Honk for complex code changes, Fleetshift for organization and CI for validation. Quality and optimization increased from 27% to 31% for combined changes, while maintenance and configuration decreased from 31% to 25%. Spotify reported that back-to-work volumes had not increased, but the volume and complexity of pull requests were increasing. The incident review found that the codes produced by AI did not have a substantial direct impact, but that the number of changes had exceeded the processing capacity of some certification mechanisms.
See the original English language version: https://www.infoq.com/news/2026/10/page-ai-equality-controls/
Auto-fetched and translated by AI Atlas. All rights belong to the original author.