DEVOPS
S3 keeps the source of truth while local NVMe repositories do the latency-sensitive work
Developers recently burned by GitHub’s system outages should take note that there are other ways to manage Git at scale. One approach, recently put into action by Cursor, is to build the distributed version control system on object storage.
A recent post from Cursor principal systems engineer Vicent Martí explains how the SpaceX subsidiary worked through its scaling issues with the notoriously fickle Git distributed version control system.
The post explains how Cursor arrived at an architecture for its own Git-based repository service called Origin, which is powered by an internal engine called Continuity. A beta of the service is available with paid Cursor plans.
“Agents have fundamentally changed the way we work with software, and in many ways they’ve made this situation worse. More code, more PRs, more CI runs. Version control is at the core of all of this, and it is possibly the hardest thing to change overnight,” Martí wrote.
Martí speaks from experience, having worked at GitHub through much of the last decade when the company arrived at its own current architecture for managing Git.
Synchronization is a bitch
Git creator Linus Torvalds designed his software to work as a content-addressable data store, where all the objects are stored and indexed by the SHA-1 hash of their contents.
Git sees a repository as a directed acyclic graph (DAG), with each commit being a node in a graph of nodes all connected by pointers. A Git server can look up an object directly by its SHA, but if it doesn’t have the SHA, then it “must actually walk the DAG step by step,” Martí noted.
A client may just want to fetch or clone a consolidated packfile of the repository, or even get a list of recent changes, but to fulfill these requests, the server must traverse the entire graph only to assemble the necessary objects.
Now, imagine providing such a service for over 400 million repositories, and you’ll get an idea of the scale at which GitHub operates (or struggles to do).
After some fiddling about, GitHub engineers landed on what they called Spokes, which basically involves keeping at least three tightly synchronized copies of every repository on speedy NVMe disks.
The approach became an industry standard, though over time its limits became apparent; the chief one is that the more replicas you make, the longer the synchronization takes. And Git does not play well with “eventual consistency,” Martí explained.
Plus, these days agents bring their own mayhem.
“When agents work with Git repositories at scale, they often operate outside of a monorepo by creating vast numbers of small repositories, many of them throwaway, and most of them barely touched,” Martí wrote.
Object stores to the rescue
When Cursor set out to build its own Git repository, it turned to object storage. Unlike file or block storage, object storage gives each chunk of bits its own unique identifier and files it with all the others in a single namespace (no directories).
The most popular object storage today is Amazon Web Services’ Simple Storage Service (S3), which is increasingly used as a foundational layer for enterprise software such as databases, container registries and message brokers, thanks to its low cost, built-in redundancy and – for all practical purposes – endless scalability.
With Origin, pushes are uploaded into S3 in a write-ahead log (WAL), capturing all changes as immutable objects. Wherever possible, changes are bundled together for faster throughput.
Simultaneously, the pushes are written to the local “reference” copy of the repository (usually an NVMe disk). Once both actions complete, other replicas of the repository can download the changes as needed.
“With the only requirement of having to synchronize the reference transaction with a single local repository instead of a quorum of replicas, we have a system that can ingest pushes as fast as our disk allows,” Martí wrote.
Git will still have to do DAG traversal for many operations, but it is better to do it locally on a speedy solid-state drive than over a network.
“Where does every repository live? The answer is ‘anywhere’. It doesn’t matter! We treat repositories like a warm cache on disk, but the source of truth is always the write-ahead log,” Martí noted.
We will see how well this approach plays out as Origin moves into being a production service. But if we don’t see stories about Origin outages, then Git managers will know to give object storage a serious look. ®
