Your Web News in One Place

Help Webnuz

Referal links:

Sign up for GreenGeeks web hosting
December 9, 2020 05:59 am GMT

Cleanup the node_modules for a lighter Lambda Function

Any nodejs project carries a bulky folder - the node_modules - that carries all the modules and dependencies that the application would need. If you try to peep into that folder, you can see a huge chunk of folders and files. That often makes me wonder - are these really required? Does my application use so much?

Not just that, each of these modules come with several versions of the code - the dist, prod, and the elaborate bulky src folder. Along with that, it has a ton of readme files and license agreements. Few of them also have a photograph of the developers!
With due regards to each of these, I feel these are not required on my production deployment. That is a big waste of disk space.

People who deploy on a bare server or an EC2 instance, may not mind all of this. Not because the cost and compute are free, but they have already resigned to overprovisioning. So such problems may be a low priority.

But, for someone who is conscious and goes for Lambda functions, it may be a big concern - where each millisecond of compute time is valuable, and so is the memory used.

One may get generous about provisioning RAM, but the deployment package has to restrict to 500MB. An ugly node_modules folder can easily grow well beyond that - and put us in trouble. Also, larger deployment size means longer warmup times. So we should do everything to ensure a compact node_modules folder to get a cleaner deployments.

Here are some of the techniques that helped me.

Check the Dependencies

First of all, we have to overcome the shock - why is my node_modules so huge?

{  "name": "layerjs",  "version": "1.0.0",  "description": "Lambda function triggered by event, to generate daily reports",  "main": "index.js",  "scripts": {    "test": "echo \"Error: no test specified\" && exit 1"  },  "keywords": [],  "author": "",  "license": "ISC",  "dependencies": {    "aws-sdk": "^2.805.0",    "jsonwebtoken": "^8.5.1",    "pdfkit": "^0.11.0",    "uuid4": "^2.0.2",    "xlsx": "^0.16.9"  }}
Enter fullscreen mode Exit fullscreen mode

Consider for example, this simple and small package.json. It pulls in a node_modules folder of 117 MB!

$  sudo du -sh node_modules117M    node_modules
Enter fullscreen mode Exit fullscreen mode

I need to know what is going on here. What does it pull in?

I found a very good tool for this. NPM Graph. Very simple to use, it provides a graphical view of all that goes into the node_modules. Just drop the package.json in there and it will show all that goes into the node_modules

layerjs_dependencies (1).jpg

That's HUGE! Let's try to reduce it now.

AWS SDK modules

This is a very common mistake. A lot of developers - who want to test stuff locally, include the AWS SDK in the package.json. This is great. But, problem starts when we have this pushed into our deployment package.

The Lambda runtime environment carries its own AWS SDK. Unless you have to make a lot of tweaks in there an need a highly customized version, this is really not required in your deployment package. This can be simply achieved by making it a dev-dependency

$ npm install PACKAGE --save-dev
Enter fullscreen mode Exit fullscreen mode

This will make the package a dev dependency. We can use it for development and testing. But it is purned off when we make a production deployment

We can do the same about many other modules that we need only in our development environment.

Production Flag

This follows from the previous one. It is the simplest and yet ignored one. Just delete the node_modules folder and install it again using the --production flag

Any package that we have marked as dev dependencies will not be a part of the deployment. Not just that, any dev-dependency of the our prod dependencies will also drop off.

With this, the package.json becomes

{  "name": "layerjs",  "version": "1.0.0",  "description": "This is the lambda layer generated for the service",  "main": "index.js",  "scripts": {    "test": "echo \"Error: no test specified\" && exit 1"  },  "keywords": [],  "author": "",  "license": "ISC",  "dependencies": {    "jsonwebtoken": "^8.5.1",    "pdfkit": "^0.11.0",    "uuid4": "^2.0.2",    "xlsx": "^0.16.9"  },  "devDependencies": {    "aws-sdk": "^2.805.0"  }}
Enter fullscreen mode Exit fullscreen mode

Now, we install it with the production flag

$ rm -rf node_modules$ npm install --production
Enter fullscreen mode Exit fullscreen mode
$ sudo du -sh node_modules59M     node_modules
Enter fullscreen mode Exit fullscreen mode

Now, the node_modules folder is 40 MB. Note that this chunk is mainly because of the SWS SDK. If everyone had followed the good coding practices, this would have made a huge difference. But... So we may not see miracles here, but it can reduce the deployment size to some extent.

Remove Unnecessary Files

Now that we have dropped the unnecessary packages, we have to start with cleaning the packages themselves.
For that, we have some good utilities.

Node Prune

$ npm install -g node-prune
Enter fullscreen mode Exit fullscreen mode

When we run this in the root folder of the project, it will again tear off what is not useful.

$ node-pruneBefore: 59M .Files: 5696After: 47M .Files: 4115
Enter fullscreen mode Exit fullscreen mode

That was good. But it could be better. Let's top it up with other utilities.

ModClean

npm install modclean -g
Enter fullscreen mode Exit fullscreen mode

Then, use it to cleanup the node_modules

$ modclean -n default:safe,default:caution -rMODCLEAN  Version 3.0.0-beta.1 Found 689 files to remove[==============================] 100% (689/689) 0.0s Found 546 empty directories to remove[==============================] 100% (546/546) 0.0sFILES/FOLDERS DELETED    Total:    1235    Skipped:  0    Empty:    546$
Enter fullscreen mode Exit fullscreen mode

It did some work. Now, the size is 43MB

$ sudo du -sh node_modules43M     node_modules
Enter fullscreen mode Exit fullscreen mode

Uglify Code

We have come down from 98MB to 43MB. That is good, but not as much as one would want. Considering the amount of junk in the node_modules folder, we need something better. And white space is what occupies most space. So we work on that. Uglifying code certainly reduces the file size.

There are several node modules that can help you uglify code. But a lot of them are not compatible with the ES2015 and above. Uglify ES is a good one. Let's start with installing that

npm install uglify-es -g
Enter fullscreen mode Exit fullscreen mode

With this in, let's uglify each JavaScript file in the node_modules folder.

find node_modules -name *.js | while read a> do> echo $a> uglifyjs $a -o $a> done
Enter fullscreen mode Exit fullscreen mode

This takes a long time, as it has to access and analyze each JS file in there.

At times, this generates a heap overflow error. Because uglifyjs is asynchronous, running in a loop spawn too many of them - causing trouble. Adding a sleep 1 in the loop can solve the problem. But it will increase the runtime further. In any case, it is worth all the effort.

$ sudo du -sh node_modules37M     node_modules
Enter fullscreen mode Exit fullscreen mode

There, now we have 37MB. That is good! Reduces my warmup time and


Original Link: https://dev.to/solegaonkar/cleanup-the-nodemodules-for-a-lighter-lambda-function-20jk

Share this article:    Share on Facebook
View Full Article

Dev To

An online community for sharing and discovering great ideas, having debates, and making friends

More About this Source Visit Dev To