迅速扩大在线存储范围，为10亿ChatGPT用户提供服务(第二部分)

迅速扩大在线存储范围，为10亿ChatGPT用户提供服务(第二部分)

我们如何优化Azure宇宙DB 在我们的应用程序存储平台，生境，管理前所未有的增长。

OpenAI的所有产品都需要快速，可靠地获取数据，由我们的存储平台"生境"提供。 它必须在毫秒内回应40个地理区域的10亿用户。 我们的基础设施面临一个非同寻常的挑战：支持爆炸性增长和新的使用案例，同时继续提高可靠性、性能和产品开发者的速度。 解决这一挑战需要重新考虑我们的在线存储平台是如何设计和运作的。 我们不断评估权衡情况，其模式是：首先，对任务关键产品流量来说，变得足够可靠，然后，对全球用户来说，速度足够快，最后，可以大规模地进行精细操作。

这个分为两部分的系列解释了我们如何从一个简单的客户图书馆和数据库发展成一个全球分布式存储平台，服务于500多位微字节的数据。 第一部分着重阐述生境从Python客户端向服务的演变。 它还包括了使我们能够将基于Python的服务推向远远超出许多人认为实用的技术。

在本文第二部分中，我们走过我们如何缩放基本的存储组件：Azure Cosmos DB. 通过与Azure团队的密切工作伙伴关系，我们达成了一种可行的多租借办法，即共同管理的集装箱，其出色的阅读性能在全球规模上是巨大的。

随着公司规模的扩大，基层团队往往从运行的单租户系统演变为多租户系统。 单租，每个产品表面团队都维持自己的系统，减少爆炸半径，提供容易的拥有界限和工作量隔离。 随着球队和产品表面的增长，这变得昂贵运行。 单租制使每个产品组都负责能力规划，往往导致供应严重过剩。

相反，多租期将控制权分配给一个集中的数据库小组，该小组在提供资源方面有更好的知识和经验。 但是，多租房本身就会产生问题，如吵闹邻里的问题，一个租户消耗了异常大的资源份额，对另一个租户不利。

在人居中心，我们必须权衡所有这些权衡，以设计一个可靠、高效、规模化的储存系统。 今天，生境是许多OpenAI产品的多租户数据库层，但我们经历了一个循环的旅程，从使用一个多租户Azure宇宙DB容器到提供单租户辅助集装箱到默认恢复到多租户集装箱。 我们对于人居的多租期故事还很早：我们支持一大批使用案例

在通过我们的Azure宇宙DB旅程之前，一些基本背景：

- Azure Cosmos DB账户界定一套集装箱和复制政策

- 每个集装箱每秒按请求单位提供一定的吞吐量

- 每个容器内的数据以典型的硬度-逐项-逻辑-分区-键方式存储，并记录每个逻辑分区键的存储和吞吐量限制

- Azure Cosmos DB 手柄将逻辑分区键压在较少的支撑物理分区上，在存储超过阈值或容器需要更多物理分区以维持供给的吞吐量时将其分割成多个物理分区。

我们最初允许任何客户端的负载通过，依靠每个逻辑分区键的10K RU/s限制来提供后压。 当我们第一次出发时，我们还在每个账户中只使用了一个默认的Azure宇宙DB容器。 这样做非常有效，但有以下几个原因证明存在问题：

由于早期API的选择，我们直接向客户端披露逻辑分区密钥选择。 因此，我们经常在常用分区键上发生碰撞，例如用户 id或工作空间 id，这些碰撞涉及不同的产品使用案例。 因为这些使用大小写共享了Azure Cosmos DB限制在同一逻辑分区键上，设计限制了给定用户的整体吞吐量。

在迁移到生境和Azure宇宙数据库之前，我们在扩大Postgres系统方面面临的一个共同问题是，一个非常热的组织或热键——一个租户产生不成比例的高负荷——提出的许多要求能够把整个数据库取出来。 为缓存这些密钥和将租赁建立为缓存缺失，作出了重大集体努力，以便由此产生的缓存缺失负载不会踩在基本数据库上。

虽然有文献记载的数据大小和吞吐量限制适用于逻辑分区键，但物理分区实际上是执行限制的地方。 在实践中，这意味着逻辑分区不能达到10,000 RU/s. 相反，它们与在物理分区上共用的所有其他逻辑分区密钥共用物理分区的界限。

这使得吵闹的邻居可以影响许多无关的逻辑分区键，在特殊产品特性的数据上操作的工作量会干扰核心登录和注册流量。

特定物理分区的吞吐量限制也是可变的，取决于分配给Azure Cosmos DB容器的总吞吐量。 集装箱供应的吞吐量大致均匀地分布在该集装箱的物理分区。

随着数据量的增长，物理分区分裂，增加分区总数，而提供的总吞吐量保持不变。 结果是，随着数据大小的增长，每个分区的有效吞吐量限制会缩小。

对于我们最重的、类似blob的使用案例来说，这造成了一种失败模式，即仅储存增长就减少了现行工作量的吞吐能力，即使请求率和使用率保持稳定。

单独的容器有助于部分解决这些分区密钥限制和吵闹的邻居。 通过使用单独的容器，每个容器都可以被分配到自己的独立的物理分区，只有该容器本身的不同工作量才可能是吵闹的邻居或分区密钥空间的重叠。 在下滑方面，在提供新集装箱方面存在着非零的间接费用——每个集装箱都需要个性化的能力规划——因此，我们希望只在服务或使用有道理的案例边界上建立集装箱。

我们开始使用单独的Azure Cosmos DB容器，用于需要强烈隔离的最关键的新用途。 结合管理弱分区的脚本，通过将每个容器提供的总吞吐量与其物理分区数同步，我们大部分避免了我们所面临的所有问题。

然而，自从我们从一个默认的容器开始以来，许多遗留的关键用途案例不幸被困在那里，而没有数据迁移。 为了保护仍然依赖共享默认容器中数据的关键服务，我们经常建议各小组为新的服务或主要产品特征单独创建容器。 随着我们的扩大，对默认集装箱的压力增加，生产事件也增加。 我们开始为任何新的服务或大型产品提供集装箱，只是为了保护默认集装箱免受更多的负荷。

这拯救了我们免于许多整个场地的停电，将爆炸半径限制在特征或服务退化停电。 然而，提供新集装箱的间接费用最终会使我们陷入困境：

在旧的共享默认容器中，从0QPS到100万QPS的任何特定的新用途案例都不太可能大大改变总使用量。 然而，在一个孤立的集装箱上，一个新的使用量从零到100万检疫和装运装置，其负荷可能比仅因内部狗食而导致的基线多出1 000x。

使用专用集装箱，各小组现在需要记住预先提供集装箱，使其发布的吞吐量达到预期水平。 产品团队往往会将它们的特性运送到生产上，并忘记预缩放

这是多租户系统的一项核心工作量混合效益，我们丧失了这一效益，更倾向于采用这种集装箱战略。

随着时间的推移，监测专用集装箱的使用也需要熟悉运输何时会转移或增加的产品。

我们可以提供一个基本的控制平面脚本，根据增长趋势扩大每个容器的供给吞吐量，但是如果没有稳定的总增长，这些缩放使OpenAI的基本上每个产品组都能参与能力规划。

这未能在组织上扩大规模，在每一服务自己的团队向其Azure宇宙DB供应支付多少勤勉费方面存在很大差异。

随着本组织的发展，这些业务问题难以扩大规模。 没有孤军奋战的中央团队进行规划，导致许多团队规划不佳。 因此，我们最终转向了多租户解决方案，并与Azure团队密切合作，以共享租户设计为我们最初的多租户问题设计修复：

先按最热的键，防止吵闹的邻居为给定物理分区消耗所有可用的吞吐量这样可以降低邻居的噪音效应，一个热键可以有一个全场爆炸半径。

要求分级QoS同样防止了脱机工作量，如批量插入或数据提取，无法将所有吞吐量用于特定的物理分区。 通过将这些工作量的请求列为较低优先事项，这些请求首先被节制，防止了对更高优先事项的节制，即为终端用户服务的互动请求。

Azure Cosmos DB 已经有一个无服务器模式，它会用提供模式解决我们的弱分区问题。 然而，出现了一个重大倒退：仅根据存储使用量加以扩大，每个物理分区的RU使用量限制为5K RU/s。 这将解决我们薄弱的分隔问题，但阻止我们扩大RU-重型使用案例。

一个新的改进后的无服务器模式Azure Cosmos DB团队为我们建造，除了基于存储-用法的拆分外，允许每个物理分区扩大至10K RU/s和推出 RU-usage-based 拆分，使我们能够停止监控，手动提升容器。

这些变化使我们得以转向使用一个共用的容器，几个月以来，我们没有发生任何供应或吵闹的邻居事件，尽管产品正在加速开发。

当公司发展到OpenAI的规模时，管理的服务变得不那么“管理”了。 许多工作量在云提供商的基础设施中变得超出标准。 由于OpenAI成为Azure Cosmos DB的最大用户之一，成功操作它需要不同的模型：共享测试，共享仪表板，直接工程通道，以及仅出现在我们规模的故障模式的联合所有权。 今天，我们最大的Azure Cosmos DB账户存储了10s的petabytes数据，并服务于20M以上的QPS. 据我们所知，我们是唯一部署Azure宇宙DB的公司。

2024年末，一场区域停电事件明确了为何有必要进行这种合作。 当时，我国最大的Azure宇宙DB账户之一的主要Azure区域停电。 在接下来的失败演习中，我们意识到，我们最大的账户要花几个小时才能在二级区域出现故障，而我们的团队和Azure宇宙DB团队都没有在我们的规模上进行测试。

大约一个月后，虽然这些账户无法回到原地区，但我们又引发了另一场严重的停电。 连接到原区域超过原区域的 Azure 宇宙

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)

How we optimized Azure Cosmos DB in our application storage platform, Habitat, to manage unprecedented growth.

All OpenAI products require fast, reliable access to data, provided by our storage platform, Habitat. It has to respond in milliseconds to over 1 billion users across 40 geographic regions. Our infrastructure faced an extraordinary challenge: supporting explosive growth and new use cases while continuing to improve reliability, performance, and product developer velocity. Solving that challenge required rethinking how our online storage platform was designed and operated. We constantly evaluated tradeoffs, following the pattern of: first become reliable enough for mission-critical product traffic, then fast enough for global users, and, finally, to deftly operate at massive scale.

This two-part series explains how we developed Habitat from a simple client library and database into a globally distributed storage platform serving over 500 petabytes of data. Part I focused on Habitat’s evolution from a Python client to a service. It also covered the techniques that allowed us to push a Python-based service far beyond what many would consider practical.

In this post, Part II, we walk through how we scaled the underlying storage component: Azure Cosmos DB. Through a close working partnership with the Azure team, we arrived at a viable multi-tenancy approach of co-managed containers, with excellent read performance at massive, global scale.

As companies scale, infra teams often evolve from running single-tenant systems to multi-tenant ones. Single-tenancy, where every product surface team maintains their own system, reduces blast radius and provides easy ownership boundaries and workload isolation. As teams and product surfaces grow, this becomes expensive to run. Single-tenancy makes every product team responsible for capacity planning and often results in significant over-provisioning.

Conversely, multi-tenancy assigns control to a centralized database team, with better knowledge and experience for provisioning resources. But multi-tenancy creates its own problems, such as noisy-neighbor issues, where one tenant consumes an unusually large share of the resources to the detriment of the other tenants.

In Habitat, we had to weigh all these tradeoffs to design a reliable, efficient, storage system at scale. Today, Habitat is a multi-tenant database layer for many OpenAI products, but we went through a circular journey from using one multi-tenant Azure Cosmos DB container to provisioning single-tenant backing containers to reverting to a multi-tenant container by default. We’re still very early in our multi-tenancy story on Habitat: we support a large set of use cases, but we’ve been able to defer investment in this area by leveraging Azure Cosmos DB’s built-in primitives wherever possible.

Before walking through our Azure Cosmos DB journey, some basic context:

- An Azure Cosmos DB account defines a set of containers and a replication policy

- Each container is provisioned for a certain amount of throughput in request units per second (RU/s)

- Data within each container is stored in a typical shard-by-specified-logical-partition-key manner with documented limits on storage and throughput per logical partition key

- Azure Cosmos DB handles sharding logical partition keys across fewer backing physical partitions, splitting them into multiple physical partitions when storage exceeds a threshold or when the container requires more physical partitions to sustain the provisioned throughput

We originally allowed any client load through, relying on the stated 10K RU/s limit per logical partition key to provide back pressure. When we first set out, we also used only one default Azure Cosmos DB container in each account. This worked well enough, but proved problematic for a few reasons:

Due to early API choices, we exposed logical partition key selection directly to clients. As a result, we have frequent collisions on commonly used partition keys, such as user_id or workspace_id, across different product use cases. Because these use cases shared Azure Cosmos DB limits on the same logical partition keys, the design constrained overall throughput for a given user.

One common problem we faced in scaling Postgres before we moved to Habitat and Azure Cosmos DB was the ability for many requests from a very hot organization or hot key—one tenant generating a disproportionately high load—to take out the whole database. Significant collective effort was spent to cache those keys and build leases into cache misses so that the resulting cache miss load does not stomp on the underlying database.

While documented data size and throughput limits apply to logical partition keys, the physical partition is actually where limits are enforced. In practice, this means logical partitions are unable to reach 10,000 RU/s. Instead, they share the physical partition’s limits with every other logical partition key colocated on that physical partition.

This allowed noisy neighbors to impact many unrelated logical partition keys, with workloads operating on data for niche product features disrupting core login and signup flows.

The throughput limit for a given physical partition is also variable, dependent on the total throughput assigned to the Azure Cosmos DB container. The container’s provisioned throughput is distributed roughly evenly across the physical partitions for that container.

As data volume grows, physical partitions split, increasing the total partition count while total provisioned throughput remains unchanged. The result is that the effective throughput limit per partition shrinks as data size grows.

For our most storage-heavy, blob-like use cases, this creates a failure mode where storage growth alone reduces throughput capacity for active workloads even while request rates and usage remain stable.

Separate containers help partially solve these partition key limits and noisy neighbors. By using separate containers, each can be assigned its own independent physical partitions and only the varying workloads on that container itself could be noisy neighbors or overlap in partition key space. On the downside, there is non-zero overhead in provisioning new containers—each container requires individualized capacity planning—so we want to create containers only at a service or use case boundary that makes sense.

We began to use separate Azure Cosmos DB containers for the most critical new use cases requiring strong isolation. Combined with a script for managing weak partitions by synchronizing the total throughput provisioned for each container with its number of physical partitions, we mostly avoided all the problems we were facing.

However, since we started with a default container, many legacy critical use cases were unfortunately stuck there without a data migration. To protect critical services that still relied on data in the shared default container, we often advised teams to create separate containers for new services or major product features. As we scaled, pressure on the default container grew and production incidents also grew. We began to provision containers for any new service or large product feature just to protect the default container from more load.

This saved us from many site-wide outages, limiting the blast radius to feature or service degradation outages. However, the overhead of provisioning new containers eventually caught up to us:

In the old shared default container, any particular new use case going from 0 QPS to 1 million QPS was unlikely to significantly shift the total aggregate usage. On an isolated container however, a new use case going from 0 to 1 million QPS might represent 1,000x more load than the baseline attributed to only internal dogfooding.

With dedicated containers, teams now needed to remember to pre-provision containers to the expected level of throughput for their launches. Product teams would often ship their features to production and forget to pre-scale and we’d sit in SEV rooms on launch day attempting to scale up containers last minute.

This is a core workload mixing benefit of multi-tenant systems that we lose in favor of better isolation with this container strategy.

Over time, monitoring usage of dedicated containers also required intimate product knowledge of when traffic would shift or grow.

We can provide a basic control plane script to scale up the provisioned throughput for each container based on growth trends, but without the stable aggregate growth, scaling these correctly enabled essentially every product team at OpenAI to be involved in capacity planning.

This failed to scale organizationally with a wide variance in how much diligence each service’s owning team paid to their Azure Cosmos DB provisioning.

As the organization grew, those operational problems became difficult to scale. No lone, central team doing planning resulted in many teams planning poorly. As a result, we ended up shifting back towards a multi-tenant solution and worked closely with the Azure team to engineer fixes for our original multi-tenancy issues with a shared-tenancy design:

Throttling the hottest keys first prevents noisy neighbors from consuming all of the available throughput for a given physical partition This reduces the noisy neighbor effect, where one hot key can have a site-wide blast radius.

Request tiering QoS similarly prevents offline workloads, such as bulk inserts or data extracts from utilizing all of the throughput for a given physical partition. By assigning the requests for these workloads a lower priority, they are throttled first, preventing throttling on higher priority, interactive requests that serve end users.

Azure Cosmos DB already had a serverless mode that would’ve fixed our weak partitions problem with the provisioned mode. However, there was a major downside: it scaled based on storage usage only and limited RU usage to 5K RU/s per physical partition. This would solve our weak partitions problem, but prevent us from scaling up RU-heavy use cases.

A new improved serverless mode Azure Cosmos DB team built for us allowed every physical partition to scale up to 10K RU/s and introduced RU-usage-based splits in addition to storage-usage-based splits allowed us to stop monitoring and scaling up containers manually.

With these changes, we’ve been able to shift back to using one shared container which has significantly simplified the need to do any upfront or continued capacity planning for product teams while preserving isolation between workloads. We have not had any provisioning or noisy neighbor incidents in the months since, despite accelerating product development.

When companies grow to OpenAI’s scale, managed services become less “managed.” Many workloads become outliers in the cloud provider’s infrastructure. As OpenAI became one of the largest users of Azure Cosmos DB, operating it successfully required a different model: shared tests, shared dashboards, direct engineering channels, and joint ownership of failure modes that only appear at our scale. Today, our largest Azure Cosmos DB account stores 10s of petabytes of data and serves more than 20M QPS. To our knowledge, we are the only company which has deployed Azure Cosmos DB at this scale.

In late 2024, a regional outage made it clear why this collaboration was necessary. At the time, a power outage struck the primary Azure region for one of our largest Azure Cosmos DB accounts. In the resulting failover exercise, we realized that our largest accounts would take several hours to fail over to a secondary region, something neither our team nor the Azure Cosmos DB team had tested at our scale.

About a month later, while failing those accounts back to the original region, we triggered another serious outage. A connection stampede to the original region overwhelmed the original region’s Azure Cosmos

Auto-fetched and translated by AI Atlas. All rights belong to the original author.