Monorepos - pnpm: a performant npm

By the end of 2020, npm released the v7 of their package manager. The most noticeable change was that it now offered support for workspaces. That meant you could finally manage a monorepo with the Node built-in package managerwithout the need for extra steps or tools.

At Hotjar, a platform that gives people insights into user and customer behavior, we decided to switch back from yarn to npm to simplify our toolingso, npm would become our package manager again, now that it gave our team the ability to our team manage a monorepo.

But, it didnt take long for us to realize that it probably wasnt the best decision.

What is npm workspaces?

Npm workspaces provide the tooling you need to manage your dependencies in a monorepo. The setup is pretty simple with a new property in your package.json:

// package.json{  "name": "workspace-example",  "workspaces": [    "./packages/*"  ]}

It automates the installation and linking process, and theres a special --workspace flag to run commands in the context of monorepo projects.

# example: run the tests for the workspace "foo"npm run test --workspace foo

When Hotjar started using npm workspaces, we had less than 20 workspaces in our monorepo. Today, we have more than 50!

So, we realized npm wasnt a scalable monorepo toolthe hard way. Our CI started taking way too long (where we got to the point of surpassing our max average runtime), the node_modules size got out of hand, and a simple npm install became a dense daily process. We actually werent adding that many dependencies to our monorepo, but working with the workspaces and installing their dependencies became tiring and tedious.

For reference: our node_modules size took 3.2GB of space disk, and installing the dependencies on CI took around 4 minutes. This wasnt desirable at all, so we started digging into how npm CLI works.

The disadvantages of npm

Npm docs are great in general. However, when it comes to understanding how their CLI works under the hood, youre going in blind. After many undesirable hours of research, we came to the conclusion that the npm workspace was a great approach for simple monorepos, but not for massive repos like ours at Hotjar. Heres why:

• Npm holds a copy of a dependency every time its installed: if we have 100 projects using a dependency, well have 100 copies of that dependency saved on disk. This doesnt work exactly like that for monorepos, since npm can dedupe dependencies with the same version, but itll still duplicate some sub-dependencies with different versions across the dependencies tree.
• Npm has blocking stages of installation: for example, it cant parallelize the installation, so each dependency follows the Resolving Fetching Writing flow, and each dependency must finish one stage before we can start another
• Npm doesnt have a proper mechanism to realize a dependency is another workspace from your monorepo: this means that every time you want to install your dependencies, npm will request such a dependency to the registry, then will try to link it locally when no dependency was found there.
• Npm hoists dependencies you install to the root of the modules directory: as a result, the source code has access to dependencies that are not added as dependencies to the project. This is known as Phantom Dependencies and might break your dependencies or code in several unexpected ways.

A performant npm

The way npm works makes it complicated (if not impossible) to escalate properly for workspaces, causing our dependencies process to slow down, and become unmanageable and unsafe. Thats why Hotjar needed to find a faster, lighter, and safer alternative to npm that also scaled properly for our monoreposo adding more workspaces didnt degrade the whole dependencies installation process.

Performant npm (Pnpm), which is sponsored by Vercel and Prisma and is making a name for itself within the Node ecosystem, is an alternative to npm that caught Hotjars special attention. With its promise of being fast, efficient, strict, and supporting monorepos, it looked like the perfect package manager candidate.

You can see a full benchmark comparison of JavaScript package managers here.

Switching to fast monorepos

After running some tests, we estimated that the improvement in terms of installation, speed, and disk space usage would be massive. It was time to leave npm behind again, so we got down to work. We spent a few weeks working on this migration, and then finally made the switch to pnpm.

The setup is quite simple too:

# pnpm-workspace.yamlpackages:- 'packages/*'

The whole installation and linking are intelligently automated, and you can run all commands in the context of workspaces with the --filter flag:

# example: run the tests for the workspace "foo"pnpm --filter foo test

Pnpm also provides advanced features like workspace protocol, or doing partial installs for your workspaces, giving you granular control over your monorepo.

However, the best improvement came in the form of performance. Lets take a look at what changed after Hotjar made the switch:

Disk space comparison

Installation time comparison (in local dev env)

The result? We eventually got our CI back into its average runtime.

We couldnt be happier about pnpm and cant recommend it enough as we continue to use itit makes your monorepo scalable and maintainable without adding any complexity. Even if you have a small or simple monorepo, you can take advantage of pnpm speed installation improvements. In our next post, well talk about Phantom Dependencies and how they complicated our migration to pnpm.

Cover photo by Mak on Unsplash