Django + Celery
Task queuing and asynchronous jobs, + user notification
Within our project we have identified the following issues:
- Some tasks take a long time to run (generating reports being the most obvious)
- Optimisations so far have only been able to reduce the larger reports to single digit minutes (still far from responsive as far as the user is concerned)
- Due to the nature of the reports, as more services are added, the generation of these reports will take longer.
One proposed solution is to move the generation of such reports to an asynchronous task runner to free up the user to do 'other things' while the reports are generated.
Celery
Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.
Given the above, Celery seems to fit our needs fairly well. Generation of reports can be queued and users notified when the reports have been generated, this leaves us with a couple of options to choose from:
- How to notify the user?
- In browser?
- Via Email?
- ???
- How to deliver the file?
- Via expiring link for direct download
- Send the document as email attachment
- Insert the document into the users securedocs
Celery also has the option of working with various backends to handle the storage of task queues, the most common of which being RabbitMQ and redis
Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.
RabbitMQ doesn't have a handy copypastable tagline, but it is a higher level messaging system for applications.
I favour redis, as it is a lower level data store and can possibly be utilised for other needs within the project (ie. browser messages, discussed later).
Example Implementation
An example implementation has been committed at
This example uses redis as the backend for celery, and provides a system to inject messages into the Django Messaging Framework.
System Setup
We need to install redis-server to hold our tasks and messages, so:
sudo apt-get install redis-server
That's it as far as redis setup goes. Further configuration information can be found at http://redis.io/topics/config
Celery Setup
We can install celery and its required libraries for redis using pip
pip install celery[redis]
Code Walkthough
More complete information can be found here. But what follows is a simple explanation of my implementation.
Celery requires a celery.py settings file at the project level (same level as settings.py) This file instantiates the celery instance and iterates through installed apps looking for a tasks.py file to load tasks.
We then register the celery instance in the project's init.py, so that the instance is created when django starts, and is usable through all apps.
At the application level, we have blog/tasks.py which holds the tasks to be created. I the example, busywaitTask simply waits for a number of seconds, then completes.
To start the task, I have created a view that does nothing but start the task. blog/views.py:StartTaskView extends the generic detailview and simply adds the task to the celery queue:
busywaitTask.delay(self.request.user.username)
That's it. The task has been added to the queue and will be dealt with by the celery workers.
Messaging
As an aside, I needed an easy way to notify a user when their task has been completed. The Django Messaging Framework provides a neat way of showing messages to users in the browser, but unfortunately provides no hooks for apps outside of the session to use it.
This was failry trivial to work around using django middleware. I created a simple middleware class middleware/tests.py:getMessagefromRedisMiddleware and added it to middleware classes in settings.py. By overriding the process_request method we have access to the session, and can add messages using the framework. There is a function addUserMessage() which serializes a message and message level and pushes it to the redis db with a key of the sessions username. Then on every request, we lookup the key of the username, and pop messages until there are none left, adding them to the regular messaging framework.