通往agent级基特基础设施的桥梁

通往agent级基特基础设施的桥梁

我们把99.999999%的GitHub存储器移到了Azure. 以下是参考的法定人数将如何带来其余部分

在过去一年里，虽然Git的流量呈指数增长，但GitHub上的99.9999%的Git寄存器开始同时生活在两个地方：我们自己的数据中心和Azure. 其他的都是最繁忙的平台

在更早的一篇博客文章中

- 尽量减少协调

- 从计算中解析存储

我们将会看到旧设计与新设计之间的桥梁。 如何让最繁忙的储存库进入Azure

我们怎么来到这里的？

动向的时机和形态是由动力转变驱动的：交通量迅速增长，我们的数据中心和周边地理区域的电力能力受到硬顶，以及有形的供应链制约。 在我们开始移徙时，剩下的能力，加上所有可能的扩展，给我们足够的时间安全移徙，但不足以缓慢移徙。

虽然我们以新的建筑方法为原型，在云基础设施上推广Git，但与此同时，在这种增长下，改变存储结构和托管平台的风险太大。 鉴于我们的规模和这项服务对社区的重要性，我们需要仔细地安排工作顺序：先移动数据，然后在数据到达后发展结构。

Git操作是一Tier 0服务：它们是GitHub的核心功能，GitHub的几乎一切都依赖于寄存器数据。 这意味着数据、服务和每个依赖必须首先移动。 首份寄存器于2025年8月登陆Azure的中美地区。 如今，大部分的寄存器在Azure有其五种复制品中的两种，Azure服务于大约一半的吉特交通。 在这段时间的大部分时间里，Azure吸收了我们的成长，而我们自己的数据中心却保持稳定。

在迁移期间，流量遵循数据。 大多数储存库有五套复制品，以平衡耐久性、可用性和规模。 为了尽量减少读取空闲，Spokes使用优惠的路线，将交通引向同一区域和提供区的复制品。 随着复制品向Azure移动，从Azure可以服务到的流量份额随每个部分增长，直到最后的复制品到达，老区完全停止接收请求。 每个寄存器都经过这个过程：

没有留下寄存器

那么，剩下的0.001%的储存物呢？

在我们目前的架构中，一个以上区域托管的存储器的书写吞吐量上限较低。 发言人的法定人数只需要多数人才能写出来，但要等到每个健康的复制品(直到超时)才能回应，在实际操作中，这意味着最慢的健康复制设定了每个写作的耐久性，由于每个写作在承诺之前都持有一个全存储器的锁，它也设定了吞吐量上限。 一旦一个复制品生活在目标Azure地区，它离我们的数据中心大约1000英里，使得每个网络往返大约25ms，一个已经接近其吞吐量上限的存储器无法吸收每个序列式的书写；它会随着每次推力而更落后。

这是我们尽量减少协调的第一个原则发挥作用的地方。 具体来说，Git的一致要求适用于每个参考。 一旦服务器接受更新到一个引用，任何客户端都不应该看到一个更古老的值，只有该值或更新的值。 这决定了更新是快速前进、推动还是删除。 但在大多数情况下，不同参考文献的更新并不互相依赖，可以独立应用。

因此，我们正在把法定人数的范围从整个存储处改为个别参考。

拥有一个全存储器的锁，用于整个参考更新，包括往返，将法定人数变成一个单一的，无序的请求队列，一次在飞行中写一个字。 每一个写在队列上需要只要最慢的健康复制品。 随着写速率的增长，队列也随之增长，每个写法的等待时间都比之前的要长。

在逐个引用序列化的情况下，只有对共同引用进行争辩的写作才能互相等待。 在实际操作中，大多数写到寄存器时会触碰不同的引用——例如agents和开发者推向自己的分支——所以他们不再互相等待。 每个写作仍然经历跨区域时空，这可以消除存储器上的单一瓶颈。 我们可以通过在类似负载下同时比较两种方法来想象排队和超时如何积累：

这些变化提高了存储器的书写吞吐量上限，大致与同时写出多少参考文献的比例，并将最后剩余的存储器放入Azure. 在具有现实流量的内部测试中，不同参考文献一致地看到3–4x的吞吐量在低纬度时较高。

缩放自读

大部分的寄存器都是大量读取的，每张书都有100到1000个读物。 今天，阅读和写作都撞到了同样的复制品，所以它们争抢同样的资源，不能独立地缩放。

将存储从计算中解析改变了这一点。 一旦寄存器数据被长期存储在Azure Blob存储中，并用每次收到的写入进行更新，我们就可以开始将读取流量与写入法定人数脱钩，并独立地缩放每个版本所需的计算。 例如，阅读会获益于比我们今天给予的多得多的记忆缓存，但这种记忆对于收缩也是必要的。 一旦他们分开

第一步是让存储对象成为法定人数的一员，但特别的，我们称之为虚拟复制品。 代表它的计算是第一个完全麻风节点。 它充当了用于对象存储的桥梁，流传到的写作和为不读服务。 因为它是法定人数的成员之一

新增对每个写作都有些耐久，但就像跨区域往返一样，每一个参考序列化可以防止它成为一个吞吐瓶颈。 虚拟写作复制品还将脱节引用的交易汇成一组网络 I/Os 来存储对象，因此一个繁忙的寄存器支付许多同时写作的往返费，而不是每个写作一次。 现在，虚拟写作复制品也处理收缩，读作复制品继承收缩状态。

随着虚拟写作复制件的到位，我们可以按照寄存器的需要，按比例推广许多虚拟读作复制件。 每个读取的复制品都会从Azure Blob存储器中温取其缓存，并在不加入法定人数的情况下开始接受读取流量，将对象存储作为真相的来源，并在背景中调和其本地状态。

此步骤解锁了显著的读比例。 在内部测试中，我们的分布式载荷测试装置经常在目前的架构中找到天花板。 与虚拟阅读复制品相对应的是，这把吊带首先击中了自己的极限。 它还带来一些挑战，这些挑战在以后的步骤中得到解决：

- 虚拟读取复制品开始与背景中的对象存储进行调和，这使得它们非常适合一些读取工作量，但并非全部。 为各种读数服务，需要将校对线上移动，只获取自复制件上次被夹住以来变化的参考状态和数据。

尽管我们逐渐增加了书写量

我们怎样才能进一步推进文字输入？

接下来会怎样？

今天的GitHub是建立在Spokes法定人数的基础上的，多年来，Spokes已经不仅仅是法定人数协调员。 它也是一个智能的路由和负载平衡层，一个统一的API，由内部客户每天在远超过1000亿个操作时使用访问和交易寄存器数据，以及一组我们称之为“扫描器”的背景系统，它们处理寄存器的放置、修理和存储节点之间的重新平衡。

在Spokes服务伞下生活的绝大多数与现有结构一样适用于新结构。 但我们也认识到，承诺协议已经把GitHub带到了十亿个储存库

我们还没说完。 随着这些步骤的落实，下一个极限是写吞吐量。 每一个书写仍然在等待复制品，每个复制品都持有一个完整的寄存器副本，然而实际上几乎没有一个书写需要订购。 基特物体是不可改变的，并按其内装物命名，因此可以在没有人同意的情况下存放。 剩下的是小的，困难的部分正在尽快订购，在白板上设计是一回事开发商和agents 24小时写到的十亿个寄存器下面，这是另一个。 进入那里迫使我们重新思考二阶效应，比如持久参照状态的所在，以及收缩和垃圾收集的规模。

Agent流量并没有减慢

下个星期来：在GitHub宇宙和本系列最后的一篇文章中，我们将巡视一下最终状态，如何记录吞吐量

感谢布赖恩·塞伦扎对这个职位的贡献。

标记：

编剧：

有关员额

常绿化：我们如何在共享的绿地上实现安全进出和重新获得卫生的自动化

将本地的 GitHub 特性用于工作，以审查外部的合作者访问和存档的 stale 寄存器。

从2500次请求到24次：重建没有星座的恒星历史

我们新的Repo恒星历史 API现在提供的趋势不返回个人身份或需要不受限制的数据库工作。 这些是我们在工程方面的权衡

为agent规模发展建设吉特基础设施

我们重建GitHub的基特基础设施，同时GitHub不断运行，为agent规模的软件开发打下基础。

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

The bridge to agent-scale Git infrastructure

We moved a copy of 99.99999% of GitHub’s repositories to Azure. Here’s how per-reference quorums will bring over the rest, and how object storage lets reads scale independently.

Over the past year, while Git traffic grew exponentially, 99.99999% of the Git repositories on GitHub began living in two places at once: our own data centers and Azure. The rest are among the busiest on the platform, and they’re held back by just one thing: a 25ms round trip.

In an earlier blog post, we laid out the two principles behind our next-generation storage architecture:

- Minimize coordination

- Decouple storage from compute

In this post, we’ll look at the immediate bridge between our old design and our new one. We’ll talk about how a change to the way our quorum serializes writes gets the busiest repositories into Azure, and how that same step opens the door to replicas that keep no persistent state of their own.

How we got here

The timing and shape of the move were driven by the agentic shift: rapidly growing traffic, a hard ceiling on the power capacity of our data centers and the surrounding geographical region, and physical supply chain constraints. At the beginning of our migration, the remaining capacity with all possible expansion gave us enough time to migrate safely, but not enough to do it slowly.

While we had prototyped new architectural approaches for scaling Git on cloud infrastructure, changing the storage architecture and the hosting platform at the same time, under that growth, carried too much risk. Given our scale and how critical the service is to the community, we needed to sequence the work carefully: move the data first, then evolve the architecture once it arrived.

Git operations are a Tier 0 service: they’re a core function of GitHub, and nearly everything in GitHub depends on the repository data. That meant the data, services, and every dependency had to move first. The first copy of a repository landed in Azure’s Central US region in August 2025. Today, most repositories have two of their five replicas in Azure, and Azure serves about half of Git traffic. For most of that time, Azure absorbed our growth while our own data centers held steady.

During the migration, traffic follows the data. Most repositories have five replicas to balance durability, availability, and scale. To minimize read latency, Spokes uses preferential routing to direct traffic to a replica in the same region and availability zone as the request. As replicas move to Azure, the share of the traffic that can be served from Azure grows with each one, until the last replica arrives and the old region stops receiving requests entirely. Every repository goes through this process:

No repository left behind

So, what about the remaining 0.00001% of repositories?

In our current architecture, a repository hosted in more than one region has a lower write throughput ceiling. Spokes’ quorum only needs a majority to commit a write, but it waits for every healthy replica to respond (up to a timeout) before moving on, so slow replicas delay every write rather than falling behind. In practice, that means the slowest healthy replica sets the latency of every write, and because each write holds a repository-wide lock until it commits, it also sets the throughput ceiling. Once a replica lives in the target Azure region, it’s about 1,000 miles from our data centers, making each network round trip around 25ms, and a write requires multiple round trips. A repository already near its throughput ceiling can’t absorb that on every serialized write; it would fall further behind with every push.

This is where our first principle, minimizing coordination, comes into play. Specifically for this case: don’t serialize what doesn’t need to be. Git’s consistency requirements apply to each reference individually. Once the server accepts an update to a reference, no client should see an older value afterward, only that value or a newer one. That holds whether the update is a fast-forward, a force push, or a delete. But in most cases, updates to different references don’t depend on each other and can be applied independently.

So, we’re changing the scope of the quorum from the whole repository to individual references.

Holding a repository-wide lock for the entire reference update, round trips included, turns the quorum into a single, unordered request queue with one write in flight at a time. Each write in that queue takes as long as the slowest healthy replica. As the write rate grows, so does the queue, and each write waits longer than the one before it.

With per-reference serialization, only writes that contend on a shared reference must wait for each other. In practice, most writes to a repository touch different references—for example, agents and developers pushing to their own branches—so they no longer wait on each other. Each write still experiences the cross-region latency but writes to different references can be in flight at the same time. That removes the single bottleneck on the repository. We can visualize how queueing and timeouts build up by comparing the two approaches side by side under similar load:

These changes raise a repository’s write throughput ceiling roughly in proportion to how many references are being written at once, and get the last remaining repositories into Azure. In internal testing with realistic traffic, writes to different references consistently saw 3–4x higher throughput at lower latency.

Scaling reads on their own

Most repositories are heavily read-skewed, with somewhere between 100 and 1,000 reads for every write. Today, reads and writes hit the same replicas, so they compete for the same resources and can’t be scaled independently.

Decoupling storage from compute changes that. Once repository data is durably stored in Azure Blob Storage and updated with each incoming write, we can begin to decouple read traffic from the write quorum, and independently scale the compute needed for each. For example, reads would benefit from far more in-memory cache than we give them today, but that memory is also needed for compaction. Once they’re separated, each gets its own configuration.

The first step is making object storage a member of the quorum, but a special one that we call a virtual replica. The compute that represents it is our first fully ephemeral node. It serves as a bridge to object storage, streaming incoming writes and serving no reads. Because it’s a member of the quorum, a write isn’t acknowledged until it’s durable in object storage.

That adds a little latency to each write, but just like the cross-region round trip, per-reference serialization keeps it from becoming a throughput bottleneck. The virtual write replica also coalesces transactions on disjoint references into a single set of network I/Os to object storage, so a busy repository pays for one round trip across many concurrent writes instead of one per write. For now, the virtual write replica also handles compaction, and read replicas inherit the compacted state.

With the virtual write replica in place, we can scale out as many virtual read replicas as a repository needs, on demand. Each read replica warms its cache from Azure Blob Storage and starts taking read traffic without joining the quorum, using object storage as the source of truth and reconciling its local state against it in the background.

This step unlocks significant read scale. In internal testing, our distributed load testing harness routinely found ceilings in the current architecture. Against virtual read replicas, the harness hit its own limits first. It also comes with a few challenges that are solved in subsequent steps:

- Virtual read replicas start out reconciling with object storage in the background, which makes them a good fit for some read workloads but not all of them. Serving every kind of read requires moving that reconciliation in-line, fetching only the reference state and data that changed since the replica last caught up.

- While we’ve incrementally increased write throughput by shifting to a per-reference model, the quorum remains the bounding factor for increasing further.

How can we push write throughput further?

What comes next

The GitHub of today was built on the Spokes quorum, and over the years Spokes has become much more than a quorum coordinator. It’s also an intelligent routing and load balancing layer, a unified API that internal clients use to access and transact repository data at well over 100 billion operations per day, and a collection of background systems we call “sweepers” that handle repository placement, repair, and rebalancing between storage nodes.

The vast majority of what lives under the Spokes service umbrella applies just as well to the new architecture as it does to the existing one. But we also recognize that the commit protocol that has taken GitHub to a billion repositories won’t take it to the next billion and beyond.

We’re not done yet. With these steps in place, the next limit is write throughput. Every write still waits on replicas that each hold a full copy of the repository, yet very little of a write actually needs to be ordered. Git objects are immutable and named by their contents, so they can be stored without anyone agreeing on them. What’s left is small, and the hard part is ordering it as fast as possible without moving the bottleneck somewhere else. Designing for that on a whiteboard is one thing. Doing it underneath a billion repositories that developers and agents are writing to around the clock is another. Getting there forced us to rethink the second-order effects too, like where durable reference state lives and how compaction and garbage collection scale.

Agent traffic isn’t slowing down, and each of these steps gives repositories more headroom as it lands, without waiting for the end state.

Coming next week: At GitHub Universe and in the final post in this series, we’ll take a tour through that end state, what it does for write throughput, the second-order problems we had to solve to get there, and why there’s still a (small) consensus group somewhere in the stack.

Thanks to Brian Celenza for their contributions to this post.

Tags:

Written by

Related posts

Staying Evergreen: How we automated secure access and repo hygiene in a shared org

Put native GitHub features to work to review outside collaborator access and archive stale repositories.

From 2,500 requests to 24: Rebuilding star history without the stargazers

Our new repo star history API now provides trends without returning individual identities or requiring unbounded database work. These are the engineering tradeoffs we made.

Building Git infrastructure for agent-scale development

We’re rebuilding GitHub’s Git infrastructure while GitHub keeps running, creating a foundation for agent-scale software development.

Auto-fetched and translated by AI Atlas. All rights belong to the original author.