Practical Puppeteer: Using proxy to browse a page

Today Puppeteer topic will be related to proxy. Using proxy when browse a page is useful when we want to hide our origin access location. That's only one reason. Another reason it can be used to protect our privacy and other use case is to open a website with geographical restriction.

According to Wikipedia,

In computer networks, a proxy server is a server (a computer system or an application) that acts as an intermediary for requests from clients seeking resources from other servers. A client connects to the proxy server, requesting some service, such as a file, connection, web page, or other resource available from a different server and the proxy server evaluates the request as a way to simplify and control its complexity. Proxies were invented to add structure and encapsulation to distributed systems.

In Puppetter we can use a proxy when we browse a page on internet. I will use several sample of proxy, such as SOCKS4, SOCKS5 and HTTP proxy.

Let's start.

Preparation

Install Puppeteer

npm i puppeteer

We also need some proxy sample. For this I will use list of free proxy from https://hidemy.name/en/proxy-list/ and we can pick several proxy from there.

The code

We will use SOCKS4 proxy and IP location of this proxy at Cambodia. Proxy IP address 96.9.77.192 and port 55796. I hope the proxy address still working when you try the example.

File proxy_with_puppeteer.js

const puppeteer = require('puppeteer');(async () => {    // set some options (set headless to false so we can see     // this automated browsing experience)    let launchOptions = { headless: false,                           args: ['--start-maximized',                                 '--proxy-server=socks4://96.9.77.192:55796'] // this is where we set the proxy                        };    const browser = await puppeteer.launch(launchOptions);    const page = await browser.newPage();    // set viewport and user agent (just in case for nice viewing)    await page.setViewport({width: 1366, height: 768});    await page.setUserAgent('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36');    // go to whatismycountry.com to see if proxy works (based on geography location)    await page.goto('https://whatismycountry.com');    // close the browser    // await browser.close();})();

Run it with

node proxy_with_puppeteer.js

It will open website https://whatismycountry.com and it will show like below.

Ow nice, it means the proxy works.

How about SOCKS5 proxy? It's easy, just change the code that set the proxy like below.

'--proxy-server=socks5://PROXY_IP_ADDRESS:PROXY_PORT'

For HTTP or HTTPS proxy we can do like below.

'--proxy-server=PROXY_IP_ADDRESS:PROXY_PORT'

If the proxy need authentication, we can add this code to support authentication. Put it before page.goto() part.

    // set the proxy credential    await page.authenticate({'username':'YOUR_USERNAME', 'password': 'YOUR_PASSWORD'});

That's it.

Thank you and I hope you enjoy it.

Reference

• https://en.wikipedia.org/wiki/Proxy_server
• https://hidemy.name/en/proxy-list/
• https://free-proxy-list.net/
• https://pptr.dev