OpenAI develops Pando to scale Git backend for large-scale development

OpenAI has developed Pando, a new backend for Git repositories designed to manage the increasing scale and demands of the company's monorepo. According to OpenAI, the system was built to support the level of concurrent work enabled by Codex without interrupting existing engineering workflows or changing how the team uses Git.

Scalability and Performance

OpenAI states that Pando was designed to achieve two primary goals: providing horizontally scalable writes optimized for concurrent pushes to many branches, and maximizing read throughput to allow clients to download data at speeds that saturate available network bandwidth.

In production, OpenAI reports that Pando has achieved the following:

• Cut median push latency by nearly 5x.
• Delivers nearly a petabyte of Git data per day.

Technical Architecture

To achieve these goals, Pando utilizes a relaxed consistency model known as causal consistency. This approach allows updates to unrelated branches to proceed in parallel by coordinating updates to each branch independently. According to OpenAI, this means a reader looking at several branches may see them at different points in time.

Unlike traditional Git service patterns, Pando employs the following methods:

• Write Optimization: Pando validates incoming packfiles while streaming them to blob storage without unpacking them into a local repository. It uses a multi-key transaction to record ref versions in a metadata store, which OpenAI says requires $O(1)$ metadata reads and writes.
• Read Optimization: Pando reuses packfiles across fetches rather than generating individual packfiles for each client. This allows the server to serve the same bytes to many clients with minimal computation.

Comparison to Existing Methods

OpenAI evaluated two conventional approaches to scaling Git: three-phase commit (3PC) and write-ahead log (WAL) based designs.

According to OpenAI's simulations and testing:

• Three-phase commit (3PC): This method requires multiple rounds of communication between a coordinator and replicas, which OpenAI says leads to significantly greater write latency under load as disks saturate.
• Write-ahead log (WAL): This method uses a shared manifest in blob storage to order updates. OpenAI reports that these systems show significantly greater read latency because servers struggle to keep up with reading from the journal.

OpenAI states that Pando supports many thousands of clients pushing concurrently without impacting performance under load.

---

Sources

• openai.com: Scaling Git at OpenAI | OpenAI

This article was drafted with AI assistance and checked against the sources above. Company claims are reported as claims. Cover image is AI-generated.