On Maintaining a Third-party VS Code Extension Marketplace

Amongst the various things that I worked on at Coder.com, one of the most interesting projects was that of the internal extension marketplace. This marketplace interfaced with code-server clients, allowing users to install extensions on their server-powered VS Code instances. Since the official VS Code Marketplace was not technically or legally usable for third-parties (it still isn't), we had to roll our own!

Our internal marketplace leveraged the Google Cloud Storage platform to support over 4.5 thousand extensions for code-server users.

That's fantastic, however this project was particularly challenging for three reasons:

1. The official Marketplace could not be used directly
2. The official Marketplace did not appear curated in any major way
3. Some extensions were not as...easily supported...as others

Side-note as I'm writing this: let's see how many times I can say "official" or "officially".

1. Officially Off-Limits

Due to the official Marketplace being off limits, the internal system had to supply valid extensions including all their metadata. Such as:

• Manifest
  • Author, Repository, Version, etc.
• Icons
• Description
• License
• Extension Archive
• Miscellaneous Files
  • Changelog

Obviously this is a lot of data to process manually, so we had a secondary service to collect all of it. That's where the extension scraper came in. This service was in charge of pulling open source extension repositories and packaging them for distribution.

2. Officially Mysterious

Reliability is the name of the game, so the scraper's job was not only to pull the sources down, package them, and dump them out into the storage bucket. It needed to ensure that the extensions were: valid according to the official standards; and functioned.

The official standards can technically be checked by Microsoft's vsce CLI tool. This tool is actually what outputs the *.VSIX Zip archives. If it encounters any abnormalities, it won't package the extension.

Having run this command manually and programmatically on several thousand extensions, I can safely say that the most common error I've seen among the extensions that don't make the cut is, when the extension includes the vscode module as a dependency. The vscode module is provided by the VS Code runtime environment, so it's only needed as a dev-dependency. This occurs with extensions that are actually on the official Marketplace at this moment.

So, taking other errors into consideration, especially sketchy repository URLs, or incomplete metadata, my only conclusion is that the Marketplace doesn't have any curation. However, this is only an issue for less popular extensions. This isn't something that becomes apparent with any of the officially supported and maintained extensions from Microsoft. Think Python or PowerShell. Or, even extensions with strong communities backing them, like Vim. Those were all fantastic to work with, and any issues were easily reported and dealt with.

It makes sense. All of the popular extensions used by roughly a million people need to be stable and conform to the standards. It's the extensions that are used by smaller, but still significant, groups that are worrisome.

In addition to packaging, the scraper needed to ensure that the extensions would function for users. To check this, the scraper loaded each extension source into a Docker container and attempted to load all the extension's dependencies. This needed to include local and global dependencies. Again, for the vast majority this worked great. However, some of the outliers were concerning.

There was an extension that was still referencing and attempting to download an old NPM module that experienced severe security issues in the past. I got a hold of the extension maintainer, and it was dealt with. Still, that made me wonder what other potential security risks there are with the overlooked extensions on the Marketplace.

It would be nice if there was some tool or automated submission pipeline for extensions. Ideally such a pipeline would ensure that the extension being submitted conformed to the same standards that we would expect of the incredibly popular extensions I mentioned previously.

3. Caution: Flammable

One of the unfortunate drawbacks of creating a third-party fill for a third-party fork of a widely used open source project is that you have to roll with the punches. You're at the whim of the official maintainers, and you're at the whim of all the extension maintainers that you want to support on your platform.

So, in some cases like Settings Sync (sidenote: I personally love it ) the extension maintainers were blocked by a lack of custom functionality in VS Code itself. Which means then that our users either had to cross their fingers and hope that the functionality was delivered, or take some convoluted steps to support it! Neither were ideal.

There were some repositories, such as the localization extension pack, that contain extensions that have extraordinarily aggressive versioning conventions. Every time VS Code Insiders updated, the extensions in question would update their version number to a version of VS Code we obviously couldn't support!

That lead me to create version constraints for the extension scraper. Such that aggressively versioned extensions, or extensions that needed to be rolled back due to incompatibilities (e.g. https://github.com/VSCodeVim/Vim/issues/3522), could continue functioning as expected for our users.

Overall, it was a lot of fun to work on! Thanks for reading.

Looks like a total of 9 instances of "official" and 4 instances of "officially"!

Header photo Credit: Photo by Bernard Hermant on Unsplash