Your Web News in One Place

Help Webnuz

Referal links:

Sign up for GreenGeeks web hosting
April 26, 2022 11:13 pm GMT

Creating a Skyrim API with Python and webscraping.

A friend of mine asked me to make a small project with him. He is actually, a fan of the The Elder Scrolls V - Skyrim game, but, when we searched for an API for it, we didn't find any. Maybe that's because nobody actually plays it anymore, but I doubt it, so I made my own API for it, scraping a few things of the wiki!

For these project, our steps are:

  • Access the wiki programatically
  • Scrape the data from there and store it
  • Create a server and a few endpoints that show this data

So, for the first two steps, we are talking about webscraping! I never made a really great project with webscraping before, so it was a good opportunity for me to learn as well.

First of all, let's get to the wiki!

Here you can see an example of a wiki page:
Skyrim - Factions

After looking through it a bit, you can see that there's a list of all factions in the page:
Wiki Image

So, looking in the site source code, we find that all tables with the content we need have a few elements in common, that we can use to filter out the junk text that is not going to compose our API.

Wiki's Source Code

After that, it was just a matter of using bs4 to get and clean our data!

from bs4 import BeautifulSoupimport requestsimport jsondef getLinkData(link):    return requests.get(link).contentfactions = getLinkData(    "https://elderscrolls.fandom.com/wiki/Factions_(Skyrim)")    data = []    soup = BeautifulSoup(factions, 'html.parser')    table = soup.find_all('table', attrs={'class': 'wikitable'})

After that, we go through all elements in the page, scrape all rows and columns of the table and put it in a way that actually makes sense for our API.
Last but not least, we save that to one variable so that we can pass it to our API later.

for wikiTable in table:        table_body = wikiTable.find('tbody')        rows = table_body.find_all('tr')        for row in rows:            cols = row.find_all('td')            cols = [ele.text.strip() for ele in cols]            # Get rid of empty values            data.append([ele for ele in cols if ele])    cleanData = list(filter(lambda x: x != [], data))    skyrim_data[html] = cleanData

Okay, with the scraping part done, we should then move to the next part which is create a server and display the data we scraped in an API manner.

For that, I used FastAPI which is one awesome idea for generating one API fast!

And it is as simple as it gets, first of all, we import it and then set our endpoints!

from fastapi import FastAPIfrom fastapi.responses import HTMLResponsefrom skyrim import skyrim_dataapp = FastAPI()@app.get("/", response_class=HTMLResponse)def home():    return("""    <html>        <head>            <title>Skyrim API</title>        </head>        <body>            <h1>API DO SKYRIM</h1>            <h2>Rotas disponveis:</h2>            <ul>                <a href="https://dev.to/factions"><li>/factions</li></a>            </ul>        </body>    </html>    """)@app.get("/factions")def factions():    return skyrim_data["Factions"]

I defined the / route to be an actual list of all available endpoints and there we have it!

I deployed to Heroku so you can see it in action here. It can be a little slow to open because it needs to start the container, before running the code.

I hope this has been enlightning and that's all folks!


Original Link: https://dev.to/llxd/creating-a-skyrim-api-with-python-and-webscraping-36lc

Share this article:    Share on Facebook
View Full Article

Dev To

An online community for sharing and discovering great ideas, having debates, and making friends

More About this Source Visit Dev To