An Interest In:
Web News this Week
- March 22, 2024
- March 21, 2024
- March 20, 2024
- March 19, 2024
- March 18, 2024
- March 17, 2024
- March 16, 2024
Creating a Skyrim API with Python and webscraping.
A friend of mine asked me to make a small project with him. He is actually, a fan of the The Elder Scrolls V - Skyrim game, but, when we searched for an API for it, we didn't find any. Maybe that's because nobody actually plays it anymore, but I doubt it, so I made my own API for it, scraping a few things of the wiki!
For these project, our steps are:
- Access the wiki programatically
- Scrape the data from there and store it
- Create a server and a few endpoints that show this data
So, for the first two steps, we are talking about webscraping! I never made a really great project with webscraping before, so it was a good opportunity for me to learn as well.
First of all, let's get to the wiki!
Here you can see an example of a wiki page:
Skyrim - Factions
After looking through it a bit, you can see that there's a list of all factions in the page:
So, looking in the site source code, we find that all tables with the content we need have a few elements in common, that we can use to filter out the junk text that is not going to compose our API.
After that, it was just a matter of using bs4 to get and clean our data!
from bs4 import BeautifulSoupimport requestsimport jsondef getLinkData(link): return requests.get(link).contentfactions = getLinkData( "https://elderscrolls.fandom.com/wiki/Factions_(Skyrim)") data = [] soup = BeautifulSoup(factions, 'html.parser') table = soup.find_all('table', attrs={'class': 'wikitable'})
After that, we go through all elements in the page, scrape all rows and columns of the table and put it in a way that actually makes sense for our API.
Last but not least, we save that to one variable so that we can pass it to our API later.
for wikiTable in table: table_body = wikiTable.find('tbody') rows = table_body.find_all('tr') for row in rows: cols = row.find_all('td') cols = [ele.text.strip() for ele in cols] # Get rid of empty values data.append([ele for ele in cols if ele]) cleanData = list(filter(lambda x: x != [], data)) skyrim_data[html] = cleanData
Okay, with the scraping part done, we should then move to the next part which is create a server and display the data we scraped in an API manner.
For that, I used FastAPI which is one awesome idea for generating one API fast!
And it is as simple as it gets, first of all, we import it and then set our endpoints!
from fastapi import FastAPIfrom fastapi.responses import HTMLResponsefrom skyrim import skyrim_dataapp = FastAPI()@app.get("/", response_class=HTMLResponse)def home(): return(""" <html> <head> <title>Skyrim API</title> </head> <body> <h1>API DO SKYRIM</h1> <h2>Rotas disponveis:</h2> <ul> <a href="https://dev.to/factions"><li>/factions</li></a> </ul> </body> </html> """)@app.get("/factions")def factions(): return skyrim_data["Factions"]
I defined the / route to be an actual list of all available endpoints and there we have it!
I deployed to Heroku so you can see it in action here. It can be a little slow to open because it needs to start the container, before running the code.
I hope this has been enlightning and that's all folks!
Original Link: https://dev.to/llxd/creating-a-skyrim-api-with-python-and-webscraping-36lc
Dev To
An online community for sharing and discovering great ideas, having debates, and making friendsMore About this Source Visit Dev To