OpenAI introduces Pando to scale Git backend for monorepo

OpenAI has announced the development of Pando, a new backend for Git repositories designed to scale with the increasing size and demands of the company's monorepo. According to OpenAI, the system was built to meet the needs of concurrent work made possible by Codex, which has changed the scale and pace of development at the company.

Development Drivers

OpenAI reports that the size and demands on its monorepo have increased at a "staggering pace" over the past year. The company states that starting around the end of last year, the systems supporting the monorepo were struggling to keep up, resulting in slow fetches and pushes, as well as failed merges, with reliability at times falling below 90%.

OpenAI notes that Codex has enabled engineers to do more work in parallel, leading to more branches, pushes, merges, and fetches from research jobs, development environments, and CI. These demands compound as a single engineer may have several changes ready for CI at once, and each push can trigger several jobs fetching the same commit.

Technical Architecture of Pando

To address these scaling challenges, OpenAI designed Pando to achieve two primary goals: horizontally scalable writes optimized for concurrent pushes to many branches, and maximized read throughput to allow clients to download data fast enough to saturate available network bandwidth.

Unlike conventional approaches, Pando utilizes a relaxed consistency model known as causal consistency (CC). According to OpenAI, this allows the system to scale writes by coordinating updates to each branch independently. While this provides serializability for updates to individual refs, a client reading several branches cannot assume those reads describe a single point in time across the entire repository.

Key technical features of Pando include:

• Write Optimization: Pando validates incoming packfiles while streaming them to blob storage without unpacking them into a local repository. It uses a multi-key transaction to record a ref version in a metadata store and atomically update the ref pointer.
• Read Optimization: Pando reuses packfiles across fetches rather than generating individual packfiles for each client. This allows the server to serve the same bytes to many clients with little computation.
• Compatibility: The system retains compatibility with the Git protocol and client, allowing engineers and agents to adopt Pando into existing workflows.

Performance and Comparison

OpenAI evaluated Pando against two other common approaches for scaling Git: a three-phase commit (3PC) design and a write-ahead log (WAL) design. In simulations, OpenAI found that the 3PC design showed significantly greater write latency under load as disks saturated, while WAL systems showed significantly greater read latency as servers struggled to keep up with reading from the journal.

In production, OpenAI states that Pando has cut median push latency by nearly 5x and now delivers nearly a petabyte of Git data per day.

Why it matters

As development scales to include more agents and parallel work, traditional Git service patterns face bottlenecks. OpenAI's move to a custom backend aims to support "agent-scale work" while contributing improvements back to the upstream Git community.

---

Sources

• openai.com: Scaling Git at OpenAI | OpenAI

This article was drafted with AI assistance and checked against the sources above. Company claims are reported as claims. Cover image is AI-generated.