Sources Contact Advanced Search Tutorials

An Interest In:

Web News this Week

Search Archive

Some of Our Sources

View All Sources

Help Webnuz

Referal links:

September 27, 2019 09:59 pm GMT

Export big amount of data using Celery With Django | Reverse Python Lab

By REVERSE PYTHON

What's up DEVs?

In this tutorial we are going to build Random Posts Generator app with Django using Celery and RabbitMQ.

Why should we use Celery?

Web applications usually start out simple but can become quite complex, and most of them quickly exceed the responsibility of only responding to HTTP requests. Assume that your web application sends confirmation mail to each registered user. Once your application overloaded with traffic, your web server can only handle certain number of requests and leave the user waiting for way too long.

At a certain point SQLite becomes too "lite" for real-world applications, if you are planning to use heavy tasks in your web application it is better to switch another database backend to avoid any errors.

Celery is the best choice for doing background task processing in the Python/Django ecosystem. It has a simple and clear API, and it integrates beautifully with Django. So, we are using Celery to handle the time-consuming tasks by passing them to queue to be executed in the background and always keep the server ready to respond to new requests.

Random Posts Generator

We are going to build random posts generator app which will generate hundreds of posts at once and let's see how Celery will handle this task.

Celery requires a solution to send and receive messages; usually this comes in the form of a separate service called a message broker. We will be configuring celery to use the RabbitMQ messaging system, as it provides robust, stable performance and interacts well with celery.

We can install RabbitMQ through Ubuntus repositories by following command:

sudo apt-get install rabbitmq-server

Then enable and start the RabbitMQ service:

sudo systemctl enable rabbitmq-serversudo systemctl start rabbitmq-server

Install RabbitMQ on Mac

Create Django project named randompostgenerator with an app named posts and run the following command to install Celery:

pip3 install Celery

Once installation completed, add the CELERY_BROKER_URL configuration to the settings.py file:

CELERY_BROKER_URL = 'amqp://localhost'

Setting up the username, password and vhost.

Then, create celery.py inside your project.

celery.py

import osfrom celery import Celeryos.environ.setdefault('DJANGO_SETTINGS_MODULE', 'randompostgenerator.settings')app = Celery('randompostgenerator')app.config_from_object('django.conf:settings', namespace='CELERY')app.autodiscover_tasks()

We are setting the default Django settings module for the 'celery' program and loading task modules from all registered Django app configs.

Now inside your __init__.py import the celery:

from .celery import app as celery_app__all__ = ['celery_app']

This will make sure our Celery app loaded every time Django starts.

Alright! Let's create our Post model in models.py:

from django.db import modelsclass Post(models.Model):    title = models.CharField(max_length=200)    description = models.TextField()    author = models.CharField(max_length=200)    def __str__(self):        return self.title

We are going to use Python package named Faker which will generate fake data for our model. Install the Faker by following command:

pip3 install Faker

Now, create new file named tasks.py inside posts app and create a Celery task that generates a number of random posts.

from faker import Fakerfrom .models import Postfrom celery import shared_task@shared_taskdef create_random_posts(number_of_posts):    fake = Faker()    for i in range(number_of_posts):        title = fake.sentence()        description = fake.text()        author = fake.name()        Post.objects.create(            title = title,            description = description,            author = author        )    return f'{number_of_posts} posts successfully generated!'

@shared_task will create the independent instance of the task for each app, making task reusable. This makes the @shared_task decorator useful for libraries and reusable apps, since they will not have access to the app of the user.

Faker will generate fake data for model fields. Learn more about Faker

Then in out forms.py:

from django import formsfrom django.core.validators import MinValueValidator, MaxValueValidatorclass RandomPostGeneratorForm(forms.Form):    number_of_posts = forms.IntegerField(        validators=[            MinValueValidator(50),            MaxValueValidator(500)        ]    )

This form will expect a positive integer field between 50 and 500.

Then, in our views.py:

from django.shortcuts import render, redirectfrom django.contrib import messagesfrom .models import Postfrom .forms import RandomPostGeneratorFormfrom .tasks import create_random_postsdef posts_view(request):    posts = Post.objects.all()    return render(request, 'posts.html', {'posts':posts})def generate_view(request):    form = RandomPostGeneratorForm(request.POST)    if form.is_valid():        number_of_posts = form.cleaned_data.get('number_of_posts')        #executing this function in the background        create_random_posts.delay(number_of_posts)        messages.success(request, 'Random posts generated! Refresh the page after few seconds to see the result.')        return redirect('posts')    return render(request, 'generate.html', {'form':form})

By calling create_random_posts with delay() method, we are instructing Celery to execute this function in the background.

Finally, let's see templates:

base.html

<html><head>  <meta charset="utf-8">  <title>Random</title>  <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous"></head><body>    <div class="container py-4">    <h1>Random Post Generator</h1>    <a href="{% url 'posts' %}">Post List</a> |    <a href="{% url 'generate' %}">Generate Random Posts</a>    <hr>    {% if messages %}        {% for message in messages %}        <div class="alert alert-success" role="alert">            {{message}}        </div>        {% endfor %}        <hr>    {% endif %}    {% block content %}    {% endblock %}    </div></body></html>

generate.html

{% extends 'base.html' %}{% block content %}  <h2>Generate Random Posts</h2>  <form method="post">    {% csrf_token %}    {% for field in form %}      <div class="form-group">{{ field }}</div>    {% endfor %}    <button class="btn btn-primary" type="submit">Generate</button>  </form>{% endblock %}

posts.html

{% extends 'base.html' %}{% block content %}  <h2>Posts List</h2>  <ul>    {% for post in posts %}        <h3 class="mt-5">{{post.title}}</h3>        <p class="lead">{{post.description|truncatewords:25}}</p>        <p>{{post.author}}</p>    {% endfor %}  </ul>{% endblock %}

Great! Now it is time to start the Celery worker process. Open your terminal and run the following command:

celery -A randompostgenerator worker -l info

Change the randompostgenerator to your project name if it is different. After running command, your terminal should look similar to this:

Also don't forget to run your Django project. Here is the final result:

You can clone or download the project on my GitHub

raszidzie / RandomPostGenerator-Celery-RabbitMQ

Random Post Generator App with Django using Celery & RabbitMQ

RandomPostGenerator-Celery-RabbitMQ

Random Post Generator App with Django using Celery & RabbitMQ

Getting Started

This tutorial works on Python 3+ and Django 2+.

Install dependencies:

python3 -m pip3 install -r requirements.txt

start Celery worker:

celery -A randompostgenerator worker -l info

and run following commands:

python3 manage.py makemigrations postspython3 manage.py migratepython3 manage.py runserver

View on GitHub

That's it! Make sure you are following me on social media and please support me by buying me a coffee so I can upload more tutorials like this. See you in next post DEVs!

Reverse Python
Instagram
Twitter

Original Link: https://dev.to/coderasha/export-big-amount-of-data-using-celery-with-django-reverse-python-lab-e6c

Share this article:

View Full Article

Dev To

An online community for sharing and discovering great ideas, having debates, and making friends

More About this Source Visit Dev To