Use puppeteer on the server in non-headless mode

TL:DR

Using puppeteer on the server is necessary use case when you want to

1. Crawl millions of web pages of a certain URL
2. Perform Web Automation
3. Automate the testing of web apps
4. Implement Bot-like activities on web sites etc.

But, using it in non - headless mode is such a pain. Especially, when you have any Unix-like OS i.e. ubuntu, centos installed on the remote machine and you are using the machine over SSH or any other remote connection tool like putty.

How to do it :

So, in spite of all these extreme cases we can still use it and in this article we will take a look at one of the ways to implement it.

Some Background :

Puppeteer at it's core uses the chromium browser, which does all the heavy lifting stuff like

1. Exposing DOM API's for interactions
2. Perform DOM manipulations and code injection
3. Cookie management and session handling
4. Navigation and context maintenance etc.

Chromium needs the display adapters to launch the chromium window on the host machine. Regardless of the host OS, it uses the available display adapter API available.

When we are serving puppeteer using virtual machine or any other remote machine, by default we don't have any displays because all the work is done by Bash (AKA command line).

If we have windows server, then there is no big deal. Just connect the server using Remote Desktop Connection and you are good to go. But, if you have Ubuntu, centos or any other UNIX based OS installed on the remote machine, then you have to do some extra efforts.

Steps

1. Regular stpes :-

Before doing anything else, lets update host by some OLD school stuff

sudo apt-get updatesudo apt-get upgrade 

2. Install desktop-like enviroments

For installing the desktop based utility known as XRDP, I have installed core utilities like ubuntu-mate-core and ubuntu-mate-desktop by using this

sudo apt-get install --no-install-recommends ubuntu-mate-core ubuntu-mate-desktop -y

Ubuntu Mate Core :-

This utility converts the basic Ubuntu machine to complete workstation by adding some extra applications.

Ubuntu Mate Desktop :-

This is on such desktop like environments for Ubuntu. This exposes the interface for handling the local as well as networked files, perform calculations etc.

3. Install XRDP :-

After doing above steps, we have to install XRDP server which will give us the access of Remote Desktop Connection for host machine. We will also install some secondary dependencies for implementing complete desktop like environment.

sudo apt-get install mate-core mate-desktop-environment mate-notification-daemon xrdp -y

4. Create remote user :-

As suggested by numerous manuals, we will now create non-root user for performing all of our remaining tasks.

Create a XYZ user and grant the sudo access to newly created user for avoiding any permission conflicts.

adduser xsevenusermod -aG admin xsevenusermod -aG sudo xsevensu - xseven

5. Configuring Mate Session :

We now will configure the necessary Mate Session. This will help use to customize the available desktop like enviroment as per our needs

echo mate-session> ~/.xsessionsudo cp /home/xseven/.xsession /etc/skel

6. Restart XRDP :-

Final step is to restart XRDP to apply all the configrations.

sudo service xrdp restart

Once restarted, we can connect to the remote machine by using RDC or any other compatible tool

This is how we can run puppteer in non-headless mode on server. There are many other alternatives like

1. Installing GNome or KDE for desktop machines
2. Host applications on desktop ubuntu enviroment

etc.

Thanks For Reading....