Gitaly at Scalingo: Explaining the complete redesign of how we host your application git repositories

Do you want to learn about Gitaly, the open-source solution to host Git repositories created by GitLab?

To give a bit of context, I work for the cloud hosting platform Scalingo. Scalingo is a European Platform as a Service and we host git repositories from a lot of users.

We thought that the process we used can help other developers and this is why we wrote this article.

In this article I will share the reasons behind this redesign (bye bye NFS), the choice we made (good morning Gitaly), and how we implemented Gitaly.

Lets get our hands dirty!

What is Gitaly?

Gitaly is an open-source solution developed by GitLab. It provides high-level RPC access to Git repositories. It is obviously used by GitLab to read and write Git data.

You can learn more about it on GitLab Documentation.

How Does Gitaly Works

Gitaly offers a couple of components: Praefect and Gitaly.

Gitaly is a daemon which runs on all nodes intended to store the Git repositories data (no more NFS!). It offers a high level Git API with gRPC. This high level API is also interesting so that our operators do not need to understand too much of Git internal working. They would be able to work with some high level API calls.

Praefect is intended at distributing the incoming connections to the Gitaly nodes. It detects a Gitaly node failure to be able to dispatch the connections to healthy nodes. It also ensures a repository is replicated on all Gitaly nodes. This is the component responsible of the high availability. It is really important for Scalingo to ensure all components are highly available!

How did we proceed to choose Gitaly and how can this help you to choose a technology?

Our infrastructure before Gitaly

The source code of all applications hosted on Scalingo is on a NFS shared folder. This folder is shared among all the builders.

This architecture may lead to various problems. For example, a problem may arise if multiple deployments are triggered on an application in a short time interval. Under some race condition, we may unmount the NFS folder in the middle of the second deployment.

But NFS was also known at Scalingo to imply other issues:

• Performance issues
• Network latency for file access
• Process reading the files may end up in an uninterruptible sleep
• Etc

To make a long story short: NFS was used in various places in Scalingo infancy. We are slowly moving away from it everywhere possible. And Git repositories hosting is one of the last bastions using NFS at Scalingo.

The criteria we used to choose and how this can help you

From what we read, we are not alone trying to find a good solution for repositories hosting. Especially, the most well known source code hosting company communicated about this:

• GitHub announced the creation of a home made solution named Spokes to get rid of NFS.
• GitLab released the open source software Gitaly to manage their repositories.

We also explored other options such as:

• Updating the current software to be more up-to-date and easier to maintain, and get rid of NFS by using an object storage for the storage.
• Using Git Ketch which used to be in use at Google.
• Using Stemma to manage the Git repositories and implement our own logic for automatic failover.

We further investigated, and compared various criteria that you can discover in the image below.

These criteria range from all the Git features we need (push, reupload and fetch) to more fuzzy criteria such as the perceived complexity of a solution or our perceived confidence in the solution.

Out of those criteria we always give our best to understand the time that is necessary to implement the solution.

We eventually settled on Gitaly:

• They developed it to get rid of NFS, just like us \o/
• Horizontally scalable: one can easily add more storage capacity by adding new nodes to the cluster.
• Prevents write errors.
• Highly available: with NFS, a server outage leads to a downtime for all the deployments on the platform. In the Gitaly world, the server is more resilient to failure.
• The Gitaly project is open source with a license allowing us to use it.

Our infrastructure with Gitaly

As seen on this figure, the new architecture uses Praefect and Gitaly for the Git repositories storage. It also includes a new software developed internally called Git Repository Core.

This software acts as a router: it receives incoming SSH connection coming from the git push scalingo master command issued by a customer, and direct it either to the NFS repository storage or to the Gitaly storage.

This was a strong requisite for us so that we could release this feature using the canary release pattern: release Gitaly backend for a small share of our customers and increase the share of customers using it over time.

It is also in charge of triggering the deployment of the code hosted on Gitaly.

With its central place, Git Repository Core allows us to intercept the Git traffic, and potentially filter it. We plan to add more security features, but also convenient ones for people deploying their application using this Git integration. For instance, adding the ability to customize the main branch name (e.g. main rather than the default master branch).

On a more internal perspective, Git Repository Core allows us to develop administrative endpoint to ease the work from the operator in their day-to-day work.

All elements on the figure are duplicated to improve the resiliency to server failures.

Conclusion

Gitaly is an open-source solution that offers a new way to handle git repositories.

For us it was a big change and this will help us to handle our current growth and avoid

If you want to read the full article or discover Scalingo we invite you to read it here.