Python Flask: Interacts with Docker Containers
Python3 Flask: Interacts with Docker Containers
Last Friday, I was talking with my co-worker at my work. Just about work, you know. He told me he had to implement a server for serving his features. Although I didn't understand what he was going to do exactly, it sounded really interesting and I wanted to try. Since I got a lot of work that make many pages, I had written a bunch of HTML things with React. It wasn't technically interesting at the moment(I love React but the work was almost the same at all). I got tired of that. While I was in there, the coworker said me what he was doing.
He had to do
- Make an API server that gets an image file
- Execute a docker container and pass the image file to the container and the container will create the text file that is generated from the image file
- Execute another docker container and pass the text
- The API Server knows that the container has done with its work and notify to another server(The backend that servers the data to users).
It sounds interesting, ha?, I decided to implement the system on the weekend. I thought it was going to be fun. But it's been really a long time since I coded with python. So, I had to learn a lot of stuff. During the weekends, I couldn't learn all the things, so, I would use the minimum skills for implementing the system. If you're a python developer or working with the docker, it might be not in your mind at some points.
Anyways, Let's get started!
Prerequisite
- pipenv
Pipenv is a tool that aims to bring the best of all packaging worlds (bundler, composer, npm, cargo, yarn, etc.) to the Python world. Windows is a first-class citizen, in our world.
When I first use python, it was about 5 years ago, I used conda or venv for managing packages. While I was searching for package management, I found this. It seems likenpm
, I'm not sure this is the best solution, but in my opinion, it must be worth it to try.
- docker
Docker is a platform designed to help developers build, share, and run modern applications. We handle the tedious setup, so you can focus on the code.
As I introduced at first, this is the main topic in this post
- Flask
Flask is a web application framework written in Python. It was developed by Armin Ronacher, who led a team of international Python enthusiasts called Poocco.
To implement a simple API web server, I usedFlask
. there were other libraries like Fast API. For reducing my hours, I went with Flask.
This is the process that I'm going to implement.
The API server receives a file from the user
Container A processes A file then saves it to the B File
Container B processes the B File then saves it to the C File
Users can see the result through API
Python Apps and Docker Images
I made two images and uploaded to my them repository in docker-hub.
A tokenizer and word-counting app.
[Tokenizer]
import sys, json, os, requestsfrom textblob import TextBlobdef extract_nouns(text): blob = TextBlob(text) filtered_tags = list(filter(lambda tag: tag[1] == "NN", blob.tags)) nouns = list(map(lambda tag: tag[0], filtered_tags)) return nounsdef read_file(path): with open(path) as f: contents = f.read() return contentsdef save_data(path, data): with open(path, "w") as f: json.dump(data, f)def get_filename_from_path(path): return os.path.splitext(os.path.basename(path))[0]def notify_done(url, file_name): requests.get(f"{url}/docker/tokenizer_done?file_name={file_name}")if __name__ == "__main__": if len(sys.argv) < 4: print("You must pass file path as an argument") print("python3 main.py [file path to read] [dir to save] [notification api]") print("Example) python3 main.py ./test.txt ./ http://host.docker.internal:20000") sys.exit() api_url = sys.argv[3] file_path = sys.argv[1] file_name = get_filename_from_path(file_path) target_path = os.path.join(sys.argv[2], file_name + ".json") text = read_file(file_path) nouns = extract_nouns(text) save_data(target_path, {"nouns": nouns}) notify_done(api_url, file_name) print("Done")
[word-counting]
import sys, json, os, requestsdef count_word(nouns_list): count_dict = dict() for noun in nouns_list: if noun in count_dict: count_dict[noun] += 1 else: count_dict[noun] = 1 return count_dictdef load_data(path): with open(path) as f: json_data = json.load(f) return json_datadef save_data(path, data): with open(path, "w") as f: json.dump(data, f)def get_filename_from_path(path): return os.path.splitext(os.path.basename(path))[0]def notify_done(url, file_name): requests.get(f"{url}/docker/word_count_done?file_name={file_name}")if __name__ == "__main__": if len(sys.argv) < 4: print("You must pass file path as an argument") print("python3 main.py [file path to read] [dir to save] [notification api]") print("Example) python3 main.py ./test.txt ./ http://host.docker.internal:20000") sys.exit() api_url = sys.argv[3] file_path = sys.argv[1] file_name = get_filename_from_path(file_path) target_path = os.path.join(sys.argv[2], file_name + ".json") json_data = load_data(file_path) count_dict = count_word(json_data["nouns"]) save_data(target_path, {"result": count_dict}) notify_done(api_url, file_name) print("Done")
For running the apps from the API server, I built both python files with below Dockerfiles.
[Tokenizer]
FROM python:3.9WORKDIR /appCOPY . .RUN pip install pipenvRUN pipenv installRUN pipenv run python3 -m textblob.download_corporaENTRYPOINT ["pipenv", "run", "python3", "./main.py"]
[word-counting]
FROM python:3.9WORKDIR /appCOPY . .RUN pip install pipenvRUN pipenv installENTRYPOINT ["pipenv", "run", "python3", "./main.py"]
API Server
This is the main code and it's kind of simple.
from flask import Flaskfrom dotenv import load_dotenvload_dotenv()app = Flask(__name__)import routes
Routes
[routes/docker.py]
import osfrom flask import jsonify, requestfrom server import appfrom lib import docker, jsonresult = []@app.route('/docker/tokenizer_done')def get_tokenizer_done(): file_name = request.args.get("file_name") docker.run_word_count_container(file_name) return "run a word_count container"@app.route('/docker/word_count_done')def get_word_count_done(): file_name = request.args.get("file_name") json_data = json.load_data( os.path.join(os.getenv("SHARED_VOLUME_PATH"), "word_count_output", f"{file_name}.json" )) result.append(json_data) return "all works done"@app.route('/docker/result')def get_result(): file_name = request.args.get("file_name") return jsonify({ "result": result })
[routes/upload.py]
import osfrom flask import jsonify, requestfrom werkzeug.utils import secure_filenamefrom server import appfrom lib import docker@app.route("/upload", methods=["POST"])def upload_file(): f = request.files["file"] file_name = secure_filename(f.filename) f.save(os.path.join(os.getenv("SHARED_VOLUME_PATH"), "input", file_name)) docker.run_tokenizer_container(file_name) return "succeed to upload"
[routes/__init__.py]
from routes import docker, upload
[lib/docker.py]
import osAPI_URL = os.getenv("API_URL")VOLUME_ROOT_PATH = os.getenv("SHARED_VOLUME_PATH")RUN_TOKENIZER_CONTAINER = 'docker run -it --add-host=host.docker.internal:host-gateway -v "' + VOLUME_ROOT_PATH + ':/shared_volume" hskcoder/tokenizer:0.2 /shared_volume/input/{FILE_NAME_WITH_EXTENSION} /shared_volume/tokenizer_output ' + API_URLRUN_WORD_COUNT_CONTAINER = 'docker run -it --add-host=host.docker.internal:host-gateway -v "' + VOLUME_ROOT_PATH + ':/shared_volume" hskcoder/word_count:0.2 /shared_volume/tokenizer_output/{FILE_NAME_WITHOUT_EXTENSION}.json /shared_volume/word_count_output ' + API_URLdef run_tokenizer_container(file_name): print(RUN_TOKENIZER_CONTAINER.format( FILE_NAME_WITH_EXTENSION = file_name )) os.popen(RUN_TOKENIZER_CONTAINER.format( FILE_NAME_WITH_EXTENSION = file_name ))def run_word_count_container(file_name): os.popen(RUN_WORD_COUNT_CONTAINER.format( FILE_NAME_WITHOUT_EXTENSION = file_name ))
[iib/json.py]
import jsondef load_data(path): with open(path) as f: json_data = json.load(f) return json_data
This app read environment variables from .env
file, so you need to set up like this.
The below variables just fit my system.
API_URL=http://host.docker.internal:20000ROOT_PATH=C:\Users\hskco\OneDrive\ \stuff\docker\apiSHARED_VOLUME_PATH=C:\Users\hskco\OneDrive\ \stuff\docker\api\shared_volume
You can run the server with this script
python3 -m pipenv run flask run -h 0.0.0.0 --port 20000
Before running this command, you need to set an environment variable FLASK_APP
.
Since I was developing in Windows, I ran this command in api
dir.
$env:FLASK_APP = './server.py'
If you enter http://127.0.0.1/docker/result
you must see this page.
Let's send a file to the API server and see the result.
Conclusion
It was really fun. I've learned a lot of things.
Regardless of your position, I think It would be really good to try anything you're interested in.
This example is really basic, I mean.
It should've considered like
- Authorization
- Communication between containers (In this example, the API server exposes all routes to the public)
- Management containers (Containers that have done need to be deleted)
- Deployment the API server
Without these, there must be many things that you need to consider. (I respect for backend developers) I was just focusing on implementing the system, Honestly, I didn't care much about others. If I did, I couldn't write this article. I'm going to be back to work tomorrow.
Anyways, it was fun, it's true :) I hope it'll be helpful for someone.
References
Github Source Code
Python
- https://realpython.com/python-modules-packages/
- https://www.geeksforgeeks.org/reading-and-writing-json-to-a-file-in-python/
Pipenv
Flask
Docker
- https://www.atlassian.com/microservices/microservices-architecture/kubernetes-vs-docker
- https://blog.payara.fish/do-you-need-kubernetes
- https://docs.docker.com/build/building/multi-stage/
- https://textblob.readthedocs.io/en/dev/quickstart.html#noun-phrase-extraction
- https://www.baeldung.com/ops/dockerfile-run-cmd-entrypoint
- https://www.howtogeek.com/devops/how-to-connect-to-localhost-within-a-docker-container/
Original Link: https://dev.to/lico/python-3-flask-interacts-with-docker-containers-3nce
Dev To
An online community for sharing and discovering great ideas, having debates, and making friendsMore About this Source Visit Dev To