<metaname="description" content="Task queues handle asynchronous jobs outside the HTTP request-response cycle. Learn about task queues on Full Stack Python.">
<divstyle="padding: 0 0 20px 0; margin: 0 0 20px 0; background-color: #22B24C;">
<divclass="container">
<h2style="color: #fff; margin: 20px 40px 0 0;"><ahref="https://www.gumroad.com/l/python-deployments" style="color: #fff">The Full Stack Python Guide to Deployments book</a> has been released!</h2>
</div>
</div><ahref="https://github.com/makaimc/fullstackpython.com"><imgstyle="position: absolute; top: 0; right: 0; border: 0;" src="/theme/img/fork.png" alt="Fork me on GitHub"></a>
is a presentation for what task queues are and why they are needed. </p>
</li>
<li>
<p>Flask by Example <ahref="https://realpython.com/blog/python/flask-by-example-implementing-a-redis-task-queue/">Implementing a Redis Task Queue</a>
provides a detailed walkthrough of setting up workers to use RQ with
Redis.</p>
</li>
<li>
<p><ahref="https://www.digitalocean.com/community/articles/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps">How to use Celery with RabbitMQ</a>
is a detailed walkthrough for using these tools on an Ubuntu VPS.</p>
</li>
<li>
<p>Heroku has a clear walkthrough for using
<ahref="https://devcenter.heroku.com/articles/python-rq">RQ for background tasks</a>.</p>
</li>
<li>
<p><ahref="http://www.linuxforu.com/2013/12/introducing-celery-pythondjango/">Introducing Celery for Python+Django</a>
provides an introduction to the Celery task queue.</p>
</li>
<li>
<p><ahref="https://denibertovic.com/posts/celery-best-practices/">Celery - Best Practices</a>
explains things you should not do with Celery and shows some underused
features for making task queues easier to work with.</p>
</li>
<li>
<p>The "Django in Production" series by
<ahref="https://twitter.com/robgolding63">Rob Golding</a> contains a post
specifically on <ahref="http://www.robgolding.com/blog/2011/11/27/django-in-production-part-2---background-tasks/">Background Tasks</a>.</p>
</li>
<li>
<p><ahref="http://blog.thecodepath.com/2012/11/15/asynchronous-processing-in-web-applications-part-1-a-database-is-not-a-queue/">Asynchronous Processing in Web Applications Part One</a>
and <ahref="http://blog.thecodepath.com/2013/01/06/asynchronous-processing-in-web-applications-part-2-developers-need-to-understand-message-queues/">Part Two</a>
are great reads for understanding the difference between a task queue and
why you shouldn't use your database as one.</p>
</li>
<li>
<p><ahref="http://www.caktusgroup.com/blog/2014/09/29/celery-production/">Celery in Production</a>
on the Caktus Group blog contains good practices from their experience
using Celery with RabbitMQ, monitoring tools and other aspects not often
discussed in existing documentation.</p>
</li>
<li>
<p><ahref="https://www.youtube.com/watch?v=68QWZU_gCDA">A 4 Minute Intro to Celery</a> is
when tasks are otherwise sent over unencrypted networks.</p>
</li>
<li>
<p>Miguel Grinberg wrote a nice post on using the
<ahref="http://blog.miguelgrinberg.com/post/using-celery-with-flask">task queue Celery with Flask</a>.
He gives an overview of Celery followed by specific code to set up the task
queue and integrate it with Flask.</p>
</li>
<li>
<p><ahref="http://wiredcraft.com/posts/2015/02/04/3-gotchas-for-celery.html">3 Gotchas for Working with Celery</a>
are things to keep in mind when you're new to the Celery task queue
implementation.</p>
</li>
<li>
<p><ahref="https://godjango.com/63-deferred-tasks-and-scheduled-jobs-with-celery-31-django-17-and-redis/">Deferred Tasks and Scheduled Jobs with Celery 3.1, Django 1.7 and Redis</a>
is a video along with code that shows how to set up Celery with Redis as the
broker in a Django application.</p>
</li>
<li>
<p><ahref="http://michal.karzynski.pl/blog/2014/05/18/setting-up-an-asynchronous-task-queue-for-django-using-celery-redis/">Setting up an asynchronous task queue for Django using Celery and Redis</a>
is a straightforward tutorial for setting up the Celery task queue for
Django web applications using the Redis broker on the back end.</p>
</li>
<li>
<p><ahref="http://django.zone/posts/background-jobs-django-and-celery">Background jobs with Django and Celery</a>
shows the code and a simple explanation of how to use Celery with
<ahref="/django.html">Django</a>.</p>
</li>
<li>
<p><ahref="https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/">Asynchronous Tasks With Django and Celery</a>
shows how to integrate Celery with <ahref="/django.html">Django</a> and create Periodic Tasks.</p>
</li>
<li>
<p><ahref="https://library.launchkit.io/three-quick-tips-from-two-years-with-celery-c05ff9d7f9eb">Three quick tips from two years with Celery</a>
provides some solid advice on retry delays, the -Ofair flag and global
task timeouts for Celery.</p>
</li>
</ul>
<h2>Task queue learning checklist</h2>
<ol>
<li>
<p>Pick a slow function in your project that is called during an HTTP
request.</p>
</li>
<li>
<p>Determine if you can precompute the results on a fixed interval instead
of during the HTTP request. If so, create a separate function you can call
from elsewhere then store the precomputed value in the database.</p>
</li>
<li>
<p>Read the Celery documentation and the links in the resources section below
to understand how the project works.</p>
</li>
<li>
<p>Install a message broker such as RabbitMQ or Redis and then add Celery to
your project. Configure Celery to work with the installed message broker.</p>
</li>
<li>
<p>Use Celery to invoke the function from step one on a regular basis.</p>
</li>
<li>
<p>Have the HTTP request function use the precomputed value instead of the
<ahref="http://www.deploypython.com/"><imgsrc="//static.fullstackpython.com/fsp-deployment-guide.png" alt="The Full Stack Python Guide to Deployments" width="100%"></a>
<pstyle="font-size: .8em; margin-top: 10px;">Searching for a complete, step-by-step deployment walkthrough? Learn more about <ahref="http://www.deploypython.com/">The Full Stack Python Guide to Deployments book</a>.
</p>
</div>
</div><divclass="panel panel-success">
<divclass="panel-heading">
<h3class="panel-head"><ahref="/table-of-contents.html" style="color: #fff;">Table of Contents</a></h3>
<ahref="/about-author.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>About the Author</a>
</div>
</div>
<divclass="panel panel-success">
<divclass="panel-body">
<ahref="http://www.deploypython.com/"><imgsrc="//static.fullstackpython.com/fsp-deployment-guide.png" alt="The Full Stack Python Guide to Deployments" width="100%"></a>
<pstyle="font-size: .8em; margin-top: 10px;">Searching for a complete, step-by-step deployment walkthrough? Learn more about <ahref="http://www.deploypython.com/">The Full Stack Python Guide to Deployments book</a>.