What is pnpm and is it really so fast and space-efficient?

We announced at ViteConf that our WebContainers now support pnpm. It was a major achievement in our commitment to support the Vite ecosystem as many projects running on Vite also use pnpm (examples include Vue, Astro, SvelteKit).

In this post, Id like to talk about what sets pnpm apart from other package managers - and why we need package managers at all. As a person who best operates through analogy, I will also talk about: sushi, cemetery, birthdays, and memes.

What are package managers?

I've also written an Explain like Im 5 version of this section!

Lets start with the terminology.

package, dependency, node module

A package (or dependency) is one or a few files bundled neatly together that can be downloaded from a package registry. Once its downloaded and installed, it resides in the node_modules folder in a projects directory. In this post, I will use the words a Node module, a package, and a dependency interchangeably as many folks in the tech community do. That being said, the npm docs offer a differing opinion so check it out if youre interested in trivia and precision.

package.json

Each package has a file called package.json which contains information on the package author, its version, its dependencies, and so on. A package can have many dependencies, which in turn can also have many dependencies. This sounds like a lot of files, doesnt it! It really is, especially if your project requires bigger packages.

In the olden times, youd manually download, install, and configure the dependencies of your app. Remember that most packages are oftentimes updated, which means that youd have to always make sure that your dependencies (and their dependencies!) are up to date - or that in a given project you use a correct version of a given package. This sounds not only stressful but also unnecessarily repetitive! This is where a package manager comes in.

package managers

A package manager is a tool that helps you keep track of all dependencies of your app in a consistent way. It automates the tasks of dependency installation, upgrading, configuration, and removal. It makes sure that all the packages that your app needs are installed, of the correct version, and up-to-date. This frees so much time for coding and procrastination

The first package manager for the Node runtime was introduced only in 2010 and it was npm, (which, contrary to popular belief, doesnt stand for Node package manager and, in fact, is an empty acronym). Twelve years later, npm is now managed by Microsoft and there are two other contenders to the iron throne of the package universe: yarn and pnpm.

What do package managers do?

A package manager grabs the published source files from the online registry and installs them into a directory (a folder) called node_modules. This is the folder where your app will look for the packages when you call the require() method.

You can think about the node_modules folder as a box with helpful tools that you can plug into your app so it works in a desired way. Except that all the tools are made of iron (or osmium!) and the box is massive. What I mean to say is that, sadly, prior to pnpm the node_modules folder would usually be bulky and take a lot of disk space. This resulted in a collection of memes such as:

Imagine that you have numerous projects on your local machine, and each project comes with its own node_modules folder. Most likely, many of these projects use a similar stack. In such a case, you end up with the same packages in numerous projects. Duplication and redundancy! Think about all those inspired side projects that never left the boilerplate stage, all the past attempts to check out all the hot new frameworks - all of them are chilling on your precious disk space.

Who cursed the node_modules folder?

To understand these jokes, we need to take a stroll down memory lane and look at how package management has traditionally been handled.

To the best of its intentions and abilities, npm handled the dependencies by splitting the installation (which is triggered by, for example, npm install ) process into three phases:

1. Resolving, when the package manager is checking all the projects dependencies (and their dependencies) listed in the package.json file, finds a version that satisfies the version specifier (for instance, ^1.0.0 would install any future minor/patch versions). Afterwards, it creates a file like package-lock.json for npm and pnpm-lock.yaml for pnpm. you can think about it like a projects equivalent of a birthday gift wishlist.
2. Fetch, when the package manager takes the list of resolved dependencies and fetches all the packages from the package registry, which is an equivalent of the feverish shopping in a crowded mall two hours before the party for the gifts all of your friends will give to the birthday person.
3. Linking, when the package manager writes all the dependencies into the projects node_modules folder, which, finally, is an equivalent of placing all the gifts in the corner of the party room for when they are needed.

In this scenario, each phase needs to end for the next one to begin. This means that if one dependency has twenty dependencies itself or if one package takes forever to be downloaded, you may have to wait for a long time. If we follow the birthday analogy, imagine that the wishlist included a new version of a popular phone and a box of chocolates. You end up queuing for a few days in front of the phone store to only then buy the chocolates. In such a system, you cant save your spot in the queue, go buy the chocolates and join the party, even if that looked like a better use of your time. You see that t*he way npm manages its tasks is just not efficient.*

As we said earlier, it is also the matter of the disk space. You will end up with numerous duplications of the same packages. And even if there are different versions of the same package, it is rarely the case that all of the files have been changed from one version to another. You still end up with some number of copies. We can agree that the way npm manages the disc space is not efficient either.

How is pnpm a solution to all our worries and troubles?

On its docs page you can read that pnpm is a fast, disk space efficient package manager. It really is fast - locally, up to three times faster than the alternatives - and space-efficient. But - how?

pnpm is fast

Do you remember the birthday wishlist analogy? As a gofer, npm first collected all the wishes, then bought all the gifts, then placed them in the corner of the party room. Each phase needed to end for another to begin. This is how the npm gofer organizes its way through life.

Well, if pnpm were the gofer, theyd buy the gift as soon as theyve read the wish, and designated the spot in the room as soon as they placed the order. They may not even have the physical gift yet to already be able to mark the space in the room where the gift will stand. This happens for each gift separately and independently from others. By design, pnpm doesnt have the blocking stages of installation - the processes run for each of the packages independently.

pnpm is disc space efficient

Lets talk about food. I love veggie sushi. I order a lot of veggie sushi. I also eat a lot of wasabi and I dont like waste. Even though it would be fair to assume that such portions surely are to be shared, I usually order just for myself. My order always arrives with two pairs of chopsticks even though I say on the phone that I dont need them - I have metal ones at home. I wouldnt just throw unused chopsticks away and now I have a drawer full of them, and you can also see them laying around in the most surprising places in the kitchen. Similarly, I have my own wasabi tube but would I throw away those little sachets? Never. They chill on the shelf in my fridge in an ever-expanding fashion.

My chopsticks and wasabi could work as an analogy for how npm manages disc space. Oh, youve already installed React two hundred times? Im SURE you NEED the 201st copy!

To my relief, pnpm checks what is already available on your disk and only adds what you additionally need for your project to run. All the dependencies are located in one global location (for instance, ~/.pnpm-store/ - you can check it by running pnpm store path in the terminal) called a content-addressable store. In your projects node_modules folder there is a .pnpm file that contains the virtual store with many so-called hard links. it creates one hard link for each file of each package. I like this explanation that pnpm docs offer:

So, for example, if you have foo in your project as a dependency and it occupies 1MB of space, then it will look like it occupies 1MB of space in the project's node_modules folder and the same amount of space in the global store. However, that 1MB is the same space on the disk addressed from two different locations. So in total foo occupies 1MB, not 2MB.

This makes the node_modules folder more like a portal (like the wardrobe in Narnia) to files located in different nooks of the global store, and not like a bloated storage unit. This diagram illustrates the strategy employed by pnpm:

But, does pnpm know if there are duplicate files between two versions of the same package? Yes, because, just like git, it identifies the files by a hash id (called also content integrity or checksum) and not by the filename. This means that two same files will have identical hash id and pnpm will determine that theres no reason for duplication.

pnpm has got your back

There are so many ways that pnpm looks out for you. One of them is that it is impossible to invite silly bugs by trying to use modules that are not directly specified in the project'spackage.json but, for instance, are required by your projects dependencies. While its not the end of the world if everything works well, what happens if your projects dependency no longer requires one of the packages and as a result it is no longer available in the the node_modules folder? Well, everything crashes and you dont even know why.

Such bugs may happen in npm and Yarn Classic because of the flat node_modules directory because during installation, they hoist (elevate) all of the packages to the node_modules , regardless of whether they are directly required by your app or not. As a result, your project gets access to distant relatives of its dependencies.

but what if my project doesnt support symlinks?

And even if your project presents an edge case and doesnt work well with symbolic linking (or symlinks), worry not! You can still use pnpm but set it to an npm-like mode. While it will not be space-efficient, it will still be faster.

But what about Yarn?

There are two editions of Yarn:

• Yarn Classic, which comprises versions below v2 and is no longer maintained,
• Yarn Berry, which is v2 and higher.

Yarn Classic operates similarly to npm with regards to managing the node_modules folder. Yarn Berry, on the other hand, offers three solutions:

1. the npm-like mode
2. the Plug'n'Play mode
3. the pnpm-like mode, which uses hard links to reduce the disc space (not available by default and not thoroughly documented)

The PlugnPlay seems like an intriguing option but it has been met with the communitys apprehension as the .zip files are not so easily accessible during the dev workflow. The earlier-mentioned post by TakeShape explains some of the other challenges.

Closing remarks

Considering how space-efficient and fast pnpm is, it is not a surprise that more and more projects make a move towards it and that its popularity is rapidly growing. This year, the weekly download rate for pnpm was seven times as high as last year!

/screenshots/

If youre interested in learning about why projects choose to switch to pnpm, check out these posts by Gestalt and TakeShape.

Further reading

Here are further readings that may give you a better understanding of the current landscape of package managers:

• feature comparison between the three package managers
• an in-depth comparison of the three package managers by LogRocket
• package managers speed benchmarks on the pnpm page and yarn page

Come work with us on package managers!

While youre here, an announcement: we are looking for a new team member who will work on package managers including pnpm, npm, yarn, and our own Turbo. Interested? Apply today or reach out to me on Twitter!