fullstackpython.com/task-queues.html at reStructuredText · sjl421/fullstackpython.com · GitHub

History

145 lines (138 loc) · 7.66 KB

Raw

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

<!DOCTYPE html>

<title>Full Stack Python</title>

<!--[if lt IE 9]>

<![endif]-->

<style>

html,

body {

font-size: 18px;

color: #222;

background: #fefefe;

}

body {

padding-top: 30px;

}

.footer {

padding: 20px 0 30px 0;

}

a, a:hover {border-bottom: 1px dotted; color: #444;}

a:hover {text-decoration: none; color: #000;}

.logo-title {font-size: 56px; color: #403072; padding-top: 80px;

font-family: "News Cycle", "Arial Narrow Bold", sans-serif;

font-weight: bold; line-height: 30px; margin-left: 5px;}

.logo-title a, .logo-title a:hover {color: #000; text-decoration: none;

border-bottom: none;}

.logo-title a:hover {color: gray;}

.logo-image {vertical-align: top; border: none;}

a.list-group-item.active {background: #444; border: 1px solid #222;}

a.list-group-item.active:hover {background: #444; border: 1px solid #222;}

#sidebar {margin-top: 30px;}

@media (max-width: 600px) {

.logo-header-section {

margin: 20px 32px 0 0;

}

</style>

var _gaq = _gaq || [];

_gaq.push(['_setAccount', 'UA-19910497-7']);

_gaq.push(['_trackPageview']);

(function() {

var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;

ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';

var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);

})();

</script>

</head>

<body>

</a>

<span class="logo-title"><a href="/">Full Stack Python</a></span>

</div>

<h1>Task Queues</h1>

<p>Task queues handle background work that need to be processed outside the

usual HTTP request-response cycle. These tasks are handled asynchronously

because HTTP requests must be responded back to by the server as fast as

possible otherwise the user experience in the web browser will suffer. The

most common types of jobs for task queues include</p>

<li>calculating computationally expensive data analytics</li>

<li>scheduling periodic jobs such as batch processes</li>

<li>spreading out large numbers of independent database inserts over time

instead of all at once</li>

<li>aggregating collected data values on a fixed interval, such as every

15 minutes</li>

</ul>

<h2>Task Queue resources</h2>

<p><a class="reference external" href="http://queues.io/">Queues.io</a> is a collection of task queue systems with

short summaries for each one. The task queues are not all compatible with

Python but ones that work with it are tagged with the "Python" keyword.</p>

<p>The <a class="reference external" href="http://www.celeryproject.org/">Celery</a> distributed task queue is the

most commonly used Python library for handling asynchronous tasks and

scheduling.</p>

<p>The <a class="reference external" href="http://python-rq.org/">RQ (Redis Queue)</a> is a simple Python

library for queueing jobs and processing them in the background with workers.

It is backed by Redis and it is designed to have a low barrier to entry.

It should be integrated in your web stack easily.</p>

</div>

<br/>

Next read the

<a href="/static-content.html">static content</a> section.

</div>

<a href="/introduction.html" class="list-group-item ">Introduction</a>

<a href="/servers.html" class="list-group-item ">Servers</a>

<a href="/operating-systems.html" class="list-group-item ">Operating Systems</a>

<a href="/web-servers.html" class="list-group-item ">Web Servers</a>

<a href="/platform-as-a-service.html" class="list-group-item ">Platform-as-a-service</a>

<a href="/databases.html" class="list-group-item ">Databases</a>

<a href="/wsgi-servers.html" class="list-group-item ">WSGI Servers</a>

<a href="/web-frameworks.html" class="list-group-item ">Web Frameworks</a>

<a href="/task-queues.html" class="list-group-item active">Task Queues</a>

<a href="/static-content.html" class="list-group-item ">Static Content</a>

<a href="/caching.html" class="list-group-item ">Caching</a>

<a href="/web-browsers.html" class="list-group-item ">Web Browsers</a>

<a href="/web-application-security.html" class="list-group-item ">Web Application Security</a>

<a href="/monitoring.html" class="list-group-item ">Monitoring</a>

<a href="/web-analytics.html" class="list-group-item ">Web Analytics</a>

<a href="/api-integration.html" class="list-group-item ">API Integration</a>

<a href="/source-control.html" class="list-group-item ">Source Control</a>

<a href="/configuration-management.html" class="list-group-item ">Configuration Management</a>

<a href="/dependency-management.html" class="list-group-item ">Application Dependencies</a>

<a href="/no-sql-datastore.html" class="list-group-item ">NoSQL Data Stores</a>

<a href="/about-author.html" class="list-group-item ">About</a>

<a href="/change-log.html" class="list-group-item ">Change Log</a>

</div>

</div></div>

<hr/>

<a href="http://www.mattmakai.com/" class="underline">Matt Makai</a> 2014

</div>

</body>

</html>

fullstackpython.com/task-queues.html at reStructuredText · sjl421/fullstackpython.com · GitHub

Expand file tree

Breadcrumbs

task-queues.html

Latest commit

History

Breadcrumbs

task-queues.html

File metadata and controls