Sources Contact Advanced Search Tutorials

An Interest In:

Web News this Week

Search Archive

Some of Our Sources

View All Sources

Help Webnuz

Referal links:

July 13, 2022 12:49 pm GMT

Web Scraping Google News with Python

What will be scraped
Prerequisites
Full Code
- Code Explanation
Links

What will be scraped

Prerequisites (could be skipped)

Install libraries:

pip install requests bs4 google-search-results

google-search-results is a SerpApi API package.

Basic knowledge scraping with CSS selectors

CSS selectors declare which part of the markup a style applies to thus allowing to extract data from matching tags and attributes.

If you haven't scraped with CSS selectors, there's a dedicated blog post of mine
about how to use CSS selectors when web-scraping that covers what it is, its pros and cons, and why they matter from a web-scraping perspective.

Separate virtual environment

In short, it's a thing that creates an independent set of installed libraries including different Python versions that can coexist with each other in the same system thus preventing libraries or Python version conflicts.

If you didn't work with a virtual environment before, have a look at the
dedicated Python virtual environments tutorial using Virtualenv and Poetry blog post of mine to get a little bit more familiar.

Note: this is not a strict requirement for this blog post.

Reduce the chance of being blocked

There's a chance that a request might be blocked. Have a look
at how to reduce the chance of being blocked while web-scraping, there are eleven methods to bypass blocks from most websites.

Make sure to pass User-Agent, because Google might block your requests eventually and you'll receive a different HTML thus empty output.

User-Agent identifies the browser, its version number, and its host operating system that represents a person (browser) in a Web context that lets servers and network peers identify if it's a bot or not. And we're faking "real" user visit. Check what is your user-agent.

Full Code

import requests, json, refrom parsel import Selectorheaders = {    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36"}params = {    "q": "gta san andreas",  # search query    "hl": "en",              # language of the search    "gl": "us",              # country of the search    "num": "100",            # number of search results per page    "tbm": "nws"             # news results}html = requests.get("https://www.google.com/search", headers=headers, params=params, timeout=30)selector = Selector(text=html.text)news_results = []# extract thumbnailsall_script_tags = selector.css("script::text").getall()for result, thumbnail_id in zip(selector.css(".xuvV6b"), selector.css(".FAkayc img::attr(id)")):    thumbnails = re.findall(r"s=\'([^']+)\'\;var\s?ii\=\['{_id}'\];".format(_id=thumbnail_id.get()), str(all_script_tags))    decoded_thumbnail = "".join([        bytes(bytes(img, "ascii").decode("unicode-escape"), "ascii").decode("unicode-escape") for img in thumbnails    ])    news_results.append(        {            "title": result.css(".MBeuO::text").get(),            "link": result.css("a.WlydOe::attr(href)").get(),            "source": result.css(".NUnG9d span::text").get(),            "snippet": result.css(".GI74Re::text").get(),            "date_published": result.css(".ZE0LJd span::text").get(),            "thumbnail": None if decoded_thumbnail == "" else decoded_thumbnail        }    )print(json.dumps(news_results, indent=2, ensure_ascii=False))

Code Explanation

Import libraries:

import requests, json, refrom parsel import Selector

Library	Purpose
`requests`	to make a request to the website.
`json`	to convert extracted data to a JSON object.
`re`	to extract parts of the data via regular expression.
`parsel`	to parse data from HTML/XML documents. Similar to `BeautifulSoup` but supports XPath.

Create request headers and URL parameters:

headers = {    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36"}params = {    "q": "gta san andreas",  # search query    "hl": "en",              # language of the search    "gl": "us",              # country of the search    "num": "100",            # number of search results per page    "tbm": "nws"             # news results}

Code	Explanation
`params`	a prettier way of passing URL parameters to a request.
`user-agent`	to act as a "real" user request from the browser by passing it to request headers. Default `requests` user-agent is a `python-reqeusts` so websites might understand that it's a bot or a script and block the request to the website. Check what's your `user-agent`.

Make a request, pass created request parameters and headers. Pass returned HTML to parsel:

html = requests.get("https://www.google.com/search", headers=headers, params=params, timeout=30)selector = Selector(text=html.text)

Code	Explanation
`timeout=30`	to stop waiting for response after 30 seconds.
`Selector(text=html.text)`	where passed HTML from the response will be processed by `parsel`.

Create an empty list to store extracted news results:

news_results = []

Create a variable that will hold store <script> tags from the page:

all_script_tags = selector.css("script::text").getall()

Code	Explanation
`css()`	is a `parsel` method that extracts nodes based on a given CSS selector(s).
`::text`	is a `parsel` own pseudo-element support that extracts textual data, which will translate every CSS query to XPath. In this case `::text` would become `/text()` if using XPath directly.
`getall()`	returns a `list` of matched nodes.

Iterate over news results and extract thumbnails data (skip to the next step if you don't want thumbnails):

for result, thumbnail_id in zip(selector.css(".xuvV6b"), selector.css(".FAkayc img::attr(id)")):    thumbnails = re.findall(r"s=\'([^']+)\'\;var\s?ii\=\['{_id}'\];".format(_id=thumbnail_id.get()), str(all_script_tags))    decoded_thumbnail = "".join([        bytes(bytes(img, "ascii").decode("unicode-escape"), "ascii").decode("unicode-escape") for img in thumbnails    ])

Code	Explanation
`zip()`	iterate over several iterables in parallel. In this case `zip` is used to also extract thumbnails that are located in the `<script>` tags.
`::attr(id)`	`parsel` own pseudo-element support that will extract given attribute from an HTML node.
`re.findall()`	match parts of the data from HTML using regular expression pattern. In this case, we want to match thumbnails. If you parse thumbnails directly from the HTML, you'll get a 1x1 image placeholder, not thumbnail. `findall` returns a `list` of matches.
`format(_id=thumbnail_id.get())`	`format` is a Python string format that insert passed values inside the string's placeholder, which is `_id` in this case: `\['{_id}'\];`
`str(all_script_tags)`	used to type cast returned value to a `string` type.
`"".join()`	join all items into a single string. Since this example uses list comprehension, the returned output would be a `list` of each processed element: `[thumbnail_1]` `[thumbnail_2]` `[thumbnail_3]` or `[]` if empty. `join` will ~~convert~~ join `list` to `str`
`bytes(img, "ascii").decode("unicode-escape")`	to decode parsed image data.

Append extracted results to a temporary list as a dict:

news_results.append(    {        "title": result.css(".MBeuO::text").get(),        "link": result.css("a.WlydOe::attr(href)").get(),        "source": result.css(".NUnG9d span::text").get(),        "snippet": result.css(".GI74Re::text").get(),        "date_published": result.css(".ZE0LJd span::text").get(),        "thumbnail": None if decoded_thumbnail == "" else decoded_thumbnail    })

Print extracted data:

print(json.dumps(news_results, indent=2, ensure_ascii=False))

Using Google News Result API

The main difference is that it's a quicker approach if you don't want to create the parser from scratch and maintain it over time or figure out how to scale the number of requests without being blocked.

Basic Hello World example:

from serpapi import GoogleSearchimport jsonparams = {    "api_key": "YOUR_API_KEY",  # your serpapi api key    "engine": "google",         # serpapi parsing engine    "q": "gta san andreas",     # search query    "gl": "us",                 # country from where search comes from    "tbm": "nws"                # news results    # other parameters such as language `hl` and number of news results `num`, etc.}search = GoogleSearch(params)   # where data extraction happens on the backendresults = search.get_dict()     # JSON - > Python dictionaryfor result in results["news_results"]:    print(json.dumps(results, indent=2))

Outputs:

{   "position":1,   "link":"https://www.sportskeeda.com/gta/5-strange-gta-san-andreas-glitches",   "title":"5 strange GTA San Andreas glitches",   "source":"Sportskeeda",   "date":"9 hours ago",   "snippet": "GTA San Andreas has a wide assortment of interesting and strange glitches.",   "thumbnail":"https://serpapi.com/searches/60e71e1f8b7ed2dfbde7629b/images/1394ee64917c752bdbe711e1e56e90b20906b4761045c01a2cefb327f91d40bb.jpeg"}

Google News Results API with Pagination

If there's a need to extract all results from all pages, SerpApi has a great Python pagination() method that iterates over all pages under the hood and returns an iterator:

# https://github.com/serpapi/google-search-results-pythonfrom serpapi import GoogleSearchimport jsonparams = {    "api_key": "YOUR_API_KEY", # your serpapi api key    "engine": "google",       # serpapi parsing engine    "q": "coca cola",         # search query    "tbm": "nws"              # news results}search = GoogleSearch(params) # where data extraction happenspages = search.pagination()   # returns an iterator of all pagesfor page in pages:    print(f"Current page: {page['serpapi_pagination']['current']}")    for result in page["news_results"]:        print(f"Title: {result['title']}
Link: {result['link']}
")

Outputs:

Current page: 1Title: PepsiCo's Many Troubles Now Have Me Focused on Coca-ColaLink: https://realmoney.thestreet.com/investing/pepsico-s-many-troubles-now-have-me-focused-on-coca-cola-16050336...Current page: 26Title: What You Can Learn About NFTs From Coca-Cola, Acura, and ...Link: https://www.entrepreneur.com/article/425166

Links

Join us on Twitter | YouTube

Add a Feature Request or a Bug

Original Link: https://dev.to/serpapi/web-scraping-google-news-with-python-19el

Share this article:

View Full Article

Dev To

An online community for sharing and discovering great ideas, having debates, and making friends

More About this Source Visit Dev To