Making Things Faster With Gearman and Supervisor

Sometimes our services need to perform some huge tasks after user interaction. For example, we need to send a letter, generate a report file, or call external APIs. These kinds of tasks can be slow because of third parties and can consume the resources of your server.

In this case, an application can become a snake eating an elephant, as in the book The Little Prince. You take some data from a user and make him wait because the snake needs some time to digest an elephant (or something else that your app needs to do):

To process this functionality faster, you need to make the parts of your application asynchronous. You can achieve this by delegating this task to a more powerful server or running it in a background process.

And Gearman is a proper tool that can be used to do this.

What Are We Going to Do?

In this tutorial, we will create a simple application that will delegate a task from a client to the Gearman worker. Our application will calculate a Fibonacci sequence in three processes. To run worker processes, we will install and configure Supervisor.

Please note that the examples in this tutorial need PHP7 to run.

So What Is Gearman Anyway?

First, let's discover what is Gearman from its homepage:

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel, to load balance processing, and to call functions between languages. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events. In other words, it is the nervous system for how distributed processing communicates.

In other words, Gearman is a queuing system that is easy to scale on many servers and flexible to use because of multi-language support.

Install Gearman

If you are running Debian/Ubuntu, run the following command to install Gearman with the necessary tools and PHP extension:

sudo apt-get install gearman php-gearman gearman-tools

After that, run the Gearman server and check the status:

sudo gearmand -d
gearadmin --status

But you will not see anything helpful after the status command because we haven't started any worker yet. Just remember this until we need it.

Create a Client

And we are ready to start a script called client.php. This script will create a Gearman client and send information to a server on the same machine:

<?php

// create gearman client

$client = new GearmanClient();
$client->addServer('127.0.0.1');

// config

$numbers = [
    1, 2
];

// do a task with gearman worker

$res = $client->doNormal('get_sequence', json_encode($numbers));

You may have noticed that we sent numbers in a JSON format. Gearman clients and workers talk to each other in a string format, so one of the ways to serialize an array is to use the json_encode() function or something similar.

After receiving an answer from the worker, we will unserialize it with json_decode() and output as CSV rows:

<?php

// decode answer

$lines = json_decode($res, true);
if (empty($lines)) {
    die('empty answer');
}

// output as csv

foreach ($lines as $numbers) {
    $line = implode(', ', $numbers);
    echo($line . PHP_EOL);
}

We have just finished our client script, so let's run it from terminal:

php /vagrant/tuts-gearman-supervisor/code/client.php

But it will be stuck without any output. Why? It is waiting for a worker to connect.

Create a Worker

It's time to create a worker to do the job that was ordered by the client. We will require a file with the fibonacci() function and create a new Gearman worker on the current server:

<?php

require_once __DIR__ . '/lib.php';

$worker = new GearmanWorker();
$worker->addServer('127.0.0.1');

After this, we will add a new function called the same as we called it in the client code:

<?php

$worker->addFunction('get_sequence', function ($job) {
    // decode input
    $content = $job->workload();
    $data = json_decode($content, true);

    // calculate sequence and return result
    $rows = fibonacci($data);
    return json_encode($rows);
});

And, of course, don't forget to wrap your answer to JSON format. The last thing to do is loop the worker script to use it many times without restarting:

<?php

for ($i = 0; $i < 100; ++$i) {
    $worker->work();
}

We can run the worker script in the background:

php /vagrant/tuts-gearman-supervisor/code/worker.php &

At this moment, you may already have observed that the client script has ended its job and written something like this:

vagrant@localserver:~$ /vagrant/tuts-gearman-supervisor/code/client.php
1, 2, 3
2, 3, 5
3, 5, 8
5, 8, 13
8, 13, 21

Check the Gearman Status

Finally, we have our worker running, so we can check the status again:

vagrant@localserver:~$ gearadmin  --status
get_sequence    0       1       2
vagrant@localserver:~$ ps -aux | grep worker.php
root      4595  0.0  1.5  98928  7764 ?        S    21:52   0:00 php /vagrant/tuts-gearman-supervisor/code/worker.php
root      4596  0.0  1.5  98928  7764 ?        S    21:52   0:00 php /vagrant/tuts-gearman-supervisor/code/worker.php

In each row, there is a function name and three numbers: the number of tasks in the queue (0), the number of jobs running (1), and the number of capable workers (2).

Of course, to add more workers, you can run more worker scripts. To stop each of them, you can use killall.  But there is a great tool to manage workers, and it is called Supervisor.

A Few Words About Supervisor

As the manual says:

Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems.

Let's install it and create the basic configuration file:

sudo apt-get install supervisor
sudo nano /etc/supervisor/conf.d/supervisor.conf

In the editor that opens, we will create a basic configuration for a Gearman worker:

[program:gearman-worker]
command=php /vagrant/tuts-gearman-supervisor/code/worker.php
autostart=true
autorestart=true
numprocs=3
process_name=gearman-worker-%(process_num)s

This will say to Supervisor that the worker must run in three processes and restart when ended. Now save the configuration file, reload Supervisor, and check the status of the running processes:

vagrant@localserver:~$ sudo supervisorctl reload
Restarted supervisord
vagrant@localserver:~$ sudo supervisorctl status
gearman-worker:gearman-worker-0  RUNNING    pid 4596, uptime 0:01:03
gearman-worker:gearman-worker-1  RUNNING    pid 4595, uptime 0:01:03
gearman-worker:gearman-worker-2  RUNNING    pid 4597, uptime 0:01:03

We can see three workers that are ready to take jobs from client scripts.

Conclusion

We've completed the basic tasks to install and configure Gearman. Now you are free to play with example code, so try to make the following changes to the code:

• Add some worker process in the background, like sending an e-mail.


• Play with task priorities using GearmanClient::doHigh.


• Chunk data using GearmanJob::sendData, which can be useful in the case of long tasks that can be observed by the status bar.

Also, you can scale the power of your workers by increasing the number of processes or running them on a faster server. And don't forget to use Supervisor to make your workers run.

If you have any questions, don't hesitate to ask questions in the comments to the article. 

Further Reading and Related Links

• Gearman Job Server


• 
  Gearman Service on php.net


• 
  Gearman UI to monitor Gearman tasks


• Supervisor: A Process Control System