OpenAI 缩放 Git

中文版正在随流水线自动整理，稍后刷新即可；可先阅读下方英文原文。

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Scaling Git at OpenAI

Building agentic-scale scale source control while investing in the Git ecosystem.

Over the past year, the size and demands on our monorepo have increased at a staggering pace. It was clear that we needed to revamp our source control system to scale to the level of concurrent work that Codex makes possible.

We built Pando, a new backend for Git repositories, without interrupting any engineering work or changing how our team uses Git. Pando now powers all development in OpenAI’s monorepo. Pando is designed to meet two ambitious goals: first, horizontally scalable writes, optimized for concurrent pushes to many branches. And second, maximize read throughput, so clients can download data fast enough to fully saturate their available network bandwidth.

We met these goals while retaining compatibility with the Git protocol and client, enabling both our engineers and agents to seamlessly adopt Pando into their existing workflows. In production, Pando cut median push latency by nearly 5x. It now delivers nearly a petabyte of Git data per day.

Here, we’ll cover our motivations for building a new source-control system, along with why we chose to retain compatibility with Git. We’ll compare our system with common approaches to scaling Git, describing why Pando is uniquely capable of meeting OpenAI’s scalability needs. We also discuss what’s next, and how we’ll continue contributing to Git to support agent-scale work.

Codex has changed the scale and pace of development at OpenAI, enabling engineers to do far more work in parallel. That means more branches and pushes, more merges, and more fetches and clones from development environments, research jobs, and CI.

These demands compound. One engineer now has several changes ready for CI at once, and each push can set several jobs fetching the same commit.

Starting around the end of last year, the systems supporting our monorepo were struggling to keep up. We suffered slow fetches and pushes along with failed merges, with reliability at times falling below 90%. Had we reached a limit of Git itself, or of the way we served our monorepo? Git grew out of a community that exchanged patches to the kernel via email. For more than two decades, developers have adapted their tools around Git as the way they build software has changed. When we started to feel friction in our own monorepo, it mattered to us that we could keep building on that work.

We saw a way to keep that familiar workflow while rebuilding the backend to handle pushes to different branches in parallel and serve the same history to many readers concurrently. Building on Git as the foundation of our source-control workflow meant that improvements to the client could be useful beyond OpenAI. We wanted to contribute those improvements from our copy of Git(opens in a new window) to upstream, sharing them with the broader community.

Scaling a source control system has two essential dimensions: many writers pushing to different branches, and many readers fetching the same history. Pando optimizes for both.

First, Pando coordinates updates to each branch independently. Updates to unrelated branches can proceed in parallel, and a reader fetching a branch gets the history needed for that revision. The tradeoff is that a reader looking at several branches can see them at different points of time.

Second, Pando reuses packfiles (Git’s format for compressing multiple objects together) across fetches instead of generating an individual packfile for each client. A client may download some objects it already has, but the server can serve the same bytes to many clients with little computation.

These choices let us scale writes across branches and serve reads from caches. To see how they fit together, let’s start with how Git stores and transfers history.

Git must be able to keep two parts of a repository in sync: objects and references.

A Git repository records snapshots of files and the history connecting them as a graph of objects. An object may be either a blob, tree, commit, or tag. Blobs hold the content of an individual file, and trees associate names with blobs or other trees to form a directory structure. Commits point to a root tree and zero or more parent commits, forming the history of your repository.

Objects are identified by a hash of their type and contents. Objects are also immutable: recording a changed file creates a new blob, while unchanged files and directories can reuse existing objects.

Branches and tags are references, or “refs”, that point to commits. Committing on a branch creates a new commit and updates the branch’s ref to point to it.

Git stores objects in either a “loose” or “packed” state. Loose objects are stored in individual files identified by their object ID. Packed objects are stored together in a packfile, enabling similar objects to be stored as deltas against one another.

In Git, an object is reachable from a commit if Git can find it by following links from that commit through its parents, trees, and blobs. During a fetch, the client tells the server which commits it wants and which it already has. The server finds the objects reachable from the wanted commits, excludes those it knows the client already has, and sends the remainder as a packfile in the same format Git uses on disk. A full clone starts without any objects, so it needs all the objects reachable from the history being cloned.

A push transfers objects in the other direction by asking the server to update one or more refs. The client uses the server’s set of advertised refs to avoid sending objects the server already has. The pack being sent can therefore depend on existing objects: receiving the pack alone is not enough to reconstruct the new history.

Replicating a Git repository means keeping both objects and references available across multiple servers. Before a replica can serve a reference’s new value, it must be able to supply the objects reachable from that value. The service also needs to coordinate competing reference updates, i.e., pushes that try to move the same branch from the same old tip to different new commits. Systems that impose a single order on all writes to a repository can make pushes to unrelated branches wait on one another.

While designing Pando, we evaluated conventional approaches for scaling source control using Git: one that coordinates ref updates across repository replicas using three-phase commit (3PC, a protocol for coordinating a commit across multiple nodes), and one that orders updates through a shared write-ahead log (WAL, a record of changes written before they’re applied) in blob storage.

In the three-phase commit design we modeled, the Git service keeps a complete copy of the repository on several replicas. When any server receives a push, the incoming packfile is copied and sent to all replicas in parallel. To publish the push, a majority of replicas must accept the ref update. The server managing this is the coordinator, and it requires multiple rounds of communication between the coordinator and each other replica.

To accept a reference update, each replica locks its copy of the ref and checks that it still points to the commit the client expects. The coordinator then sends a pre-commit message to the replicas that accept. Once a majority acknowledges the pre-commit message, the coordinator tells them to perform the update. Any up-to-date replica can then advertise the new branch tip from its local copy of the repository.

Keeping complete replicas makes it possible to serve reads locally, and guarantees that clients get a consistent view of all references no matter which replica they pick. As a consequence, references are updated one at a time, and each push requires multiple round trips to coordinate the ref update among replicas.

Waiting for a quorum puts replica disk performance and communication latency on the push path. As write volume grows, the replicas must keep up with incoming packs as well as read traffic.

In the WAL-based design we modeled, the Git service uses a shared manifest in blob storage to order updates across the repository. When a server receives a push, it uploads the incoming packfile to a blob store and prepares the ref update in its local copy of the repository. To publish the push, it adds an entry to a shared manifest stored in the blob store that records the ref update and points to the packfile. The Git server then updates the manifest using a compare-and-swap operation. This atomically replaces the manifest, or otherwise rejects the change. If another writer has updated it in the meantime, the server reads the intervening entries, checks that its ref update is still valid, and retries using the packfile it already uploaded.

Writers can upload their packfiles in parallel, but even pushes to unrelated branches require updating the same manifest. The round-trip time for a successful compare-and-swap limits how often the repository can publish a new manifest. A low-latency blob store makes this step fast. Batching lets one manifest update several refs at the same time, at the cost of queueing updates and increasing latency.

The blob store can receive packfiles in parallel, but updates to the shared manifest still compete with one another. Increasing the blob store’s capacity alone doesn’t resolve that bottleneck. Each replica still needs to read and index packs from the blob store before it can serve them locally. If incoming writes outpace that work, replicas fall behind and clients wait longer to fetch newly pushed commits.

That brings us to Pando. Pando is a new architecture for serving Git repositories with the goal of scaling to many clients pushing simultaneously. Here’s how it works:

Prior to developing Pando, we had a hunch that we could meet our requirements without compromising on Git, but we did need to compromise on some of the things that most Git systems assume to be requirements:

- Repository-wide consistency: Providing a consistent, point-in-time snapshot of all refs requires coordination across the repository. In a system that orders all ref updates together, pushes to unrelated branches contend on a single coordination point.

- Unique packfiles for every client: If two clients with different local histories fetch the same branch, the server may need to generate a different pack for each to avoid sending objects they already have. Creating these responses costs CPU and disk I/O, making it difficult to scale to high volumes of readers.

Instead, Pando provides a relaxed consistency model (causal consistency) to scale writes, and reuses packfiles to scale reads.

When a client pushes to a branch, Pando stores the incoming packfile as an immutable blob in a blob store. Each time the branch moves, Pando records a ref version containing the new tip, the pack that supplied it, and any earlier ref versions the pack depends on in a metadata store. Both the blob store and metadata store are horizontally scalable.

Fetching a commit or reference from Pando requires replaying a set of ref versions (packfiles) to the client, reconstructing the state of the client that created the ref version being fetched. This provides a weaker form of consistency across the entire repository, while providing serializability for updates to individual refs. Clients only conflict when pushing to the same ref simultaneously. A client reading several branches cannot assume those reads describe a single point in time across the repository.

This is a form of causal consistency. In order to fetch a reference, the reader observes the history of the writer that produced it. During a push, Pando controls what shared history the client and server agree on, ensuring clients produce more easily shared packfiles.

As a consequence, Pando doesn’t provide a point-in-time state of all references during Git operations or an ordering of ref updates within a clone or fetch. Concretely, suppose that branches foo and bar are both updated

Auto-fetched and translated by AI Atlas. All rights belong to the original author.