fullstackpython.github.com/task-queues.html at gh-pages · csdanping/fullstackpython.github.com · GitHub

History

150 lines (144 loc) · 7.53 KB

Raw

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

<!DOCTYPE html>

<title>Full Stack Python</title>

<!--[if lt IE 9]>

<![endif]-->

<style>

html,

body {

font-size: 18px;

color: #222;

background: #fefefe;

}

body {

padding-top: 30px;

}

.footer {

padding: 20px 0 30px 0;

}

a, a:hover {border-bottom: 1px dotted; color: #444;}

a:hover {text-decoration: none; color: #000;}

.logo-title {font-size: 56px; color: #403072; padding-top: 80px;

font-family: "News Cycle", "Arial Narrow Bold", sans-serif;

font-weight: bold; line-height: 30px; margin-left: 5px;}

.logo-title a, .logo-title a:hover {color: #000; text-decoration: none;

border-bottom: none;}

.logo-title a:hover {color: gray;}

.logo-image {vertical-align: top; border: none;}

a.list-group-item.active {background: #444; border: 1px solid #222;}

a.list-group-item.active:hover {background: #444; border: 1px solid #222;}

#sidebar {margin-top: 30px;}

@media (max-width: 600px) {

.logo-header-section {

margin: 20px 32px 0 0;

}

</style>

var _gaq = _gaq || [];

_gaq.push(['_setAccount', 'UA-19910497-7']);

_gaq.push(['_trackPageview']);

(function() {

var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;

ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';

var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);

})();

</script>

</head>

<body>

</a>

<span class="logo-title"><a href="/">Full Stack Python</a></span>

</div>

<h1>Task queues</h1>

<p>Task queues handle background work that need to be processed outside the

usual HTTP request-response cycle. These tasks are handled asynchronously

because HTTP requests must be responded back to by the server as fast as

possible otherwise the user experience in the web browser will suffer. The

most common types of jobs for task queues include</p>

<ul>

<li>

<p>calculating computationally expensive data analytics</p>

</li>

<li>

<p>scheduling periodic jobs such as batch processes</p>

</li>

<li>

<p>spreading out large numbers of independent database inserts over time

instead of all at once</p>

</li>

<li>

<p>aggregating collected data values on a fixed interval, such as every

15 minutes</p>

</li>

</ul>

<h2>Task Queue resources</h2>

<p>The <a href="http://www.celeryproject.org/">Celery</a> distributed task queue is the

most commonly used Python library for handling asynchronous tasks and

scheduling.</p>

<p><a href="http://queues.io/">Queues.io</a> is a collection of task queue systems with

short summaries for each one. The task queues are not all compatible with

Python but ones that work with it are tagged with the "Python" keyword.</p>

<p>The <a href="http://python-rq.org/">RQ (Redis Queue)</a> is a simple Python

library for queueing jobs and processing them in the background with workers.

It is backed by Redis and it is designed to have a low barrier to entry.

It should be integrated in your web stack easily.</p>

<br/>

Next read the

<a href="/api-integration.html">api integration</a> section.

</div>

<a href="/introduction.html" class="list-group-item ">Introduction</a>

<a href="/servers.html" class="list-group-item ">Servers</a>

<a href="/operating-systems.html" class="list-group-item ">Operating Systems</a>

<a href="/web-servers.html" class="list-group-item ">Web Servers</a>

<a href="/platform-as-a-service.html" class="list-group-item ">Platform-as-a-service</a>

<a href="/databases.html" class="list-group-item ">Databases</a>

<a href="/wsgi-servers.html" class="list-group-item ">WSGI Servers</a>

<a href="/web-frameworks.html" class="list-group-item ">Web Frameworks</a>

<a href="/dependency-management.html" class="list-group-item ">Application Dependencies</a>

<a href="/static-content.html" class="list-group-item ">Static Content</a>

<a href="/source-control.html" class="list-group-item ">Source Control</a>

<a href="/caching.html" class="list-group-item ">Caching</a>

<a href="/task-queues.html" class="list-group-item active">Task Queues</a>

<a href="/api-integration.html" class="list-group-item ">API Integration</a>

<a href="/no-sql-datastore.html" class="list-group-item ">NoSQL Data Stores</a>

<a href="/web-application-security.html" class="list-group-item ">Web Security</a>

<a href="/logging.html" class="list-group-item ">Logging</a>

<a href="/monitoring.html" class="list-group-item ">Monitoring</a>

<a href="/configuration-management.html" class="list-group-item ">Configuration Management</a>

<a href="/web-analytics.html" class="list-group-item ">Web Analytics</a>

<a href="/about-author.html" class="list-group-item ">About the Author</a>

<a href="/change-log.html" class="list-group-item ">Change Log</a>

</div>

</div></div>

<hr/>

<a href="http://www.mattmakai.com/" class="underline">Matt Makai</a> 2014

</div>

</body>

</html>

fullstackpython.github.com/task-queues.html at gh-pages · csdanping/fullstackpython.github.com · GitHub

Expand file tree

Breadcrumbs

task-queues.html

Latest commit

History

Breadcrumbs

task-queues.html

File metadata and controls