fullstackpython.github.com/monitoring.html at gh-pages · Intellifora/fullstackpython.github.com · GitHub

History

357 lines (347 loc) · 22.1 KB

Raw

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

<!DOCTYPE html>

<title>Monitoring - Full Stack Python</title>

<!--[if lt IE 9]>

<![endif]-->

<style>

.toc-indent {padding-left: 25px; padding-bottom: 1px;}

.toc-more-indent {padding-left: 37px; padding-bottom: 1px;}

</style>

(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){

(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),

m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)

})(window,document,'script','//www.google-analytics.com/analytics.js','ga');

ga('create', 'UA-19910497-7', 'auto');

ga('send', 'pageview');

</script> </head>

<body>

<span class="logo-title"><a href="/">Full Stack Python</a></span>

</div>

</div><div class="row">

<h1>Monitoring</h1>

<p>Monitoring tools capture, analyze and display information for a web

application's execution. Every application has issues arise throughout

all levels of the web stack. Monitoring tools provide transparency so

developers and operations teams can respond and fix problems.</p>

<h2>Why is monitoring necessary?</h2>

<p>Capturing and analyzing data about your production environment is critical

to proactively deal with stability, performance, and errors in a web

application.</p>

<h2>Difference between monitoring and logging</h2>

<p>Monitoring and logging are very similar in their purpose of helping to

diagnose issues with an application and aid the debugging process. One way

to think about the difference is that logging happens based on explicit events

while monitoring is a passive background collection of data. </p>

<p>For example, when an error occurs, that event is explicitly logged through

code in an exception handler. Meanwhile, a monitoring agent instruments the

code and gathers data not only about the logged exception but also the

performance of the functions.</p>

<p>This distinction between logging and monitoring is vague and not necessarily

the only way to look at it. Pragmatically, both are useful for maintaining a

production web application.</p>

<h2>Monitoring layers</h2>

<p>There are several important resources to monitor on the operating system

and network level of a web stack.</p>

<ol>

<li>CPU utilization</li>

<li>Memory utilization</li>

<li>Persistence storage consumed versus free</li>

<li>Network bandwidth and latency</li>

</ol>

<p>Application level monitoring encompasses several aspects. The amount of time

and resources dedicated to each aspect will vary based on whether an

application is read-heavy, write-heavy, or subject to rapid swings in traffic.</p>

<ol>

<li>Application warnings and errors (500-level HTTP errors)</li>

<li>Application code performance</li>

<li>Template rendering time</li>

<li>Browser rendering time for the application</li>

<li>Database querying performance</li>

</ol>

<h2>Open source monitoring projects</h2>

<ul>

<li>

<p><a href="https://github.com/etsy/statsd/">statsd</a> is a node.js network daemon that

listens for metrics and aggregates them for transfer into another service

such as Graphite.</p>

</li>

<li>

<p><a href="https://graphite.readthedocs.org/en/latest/overview.html">Graphite</a> stores

time-series data and displays them in graphs through a Django web application.</p>

</li>

<li>

<p><a href="http://github.hubspot.com/bucky/">Bucky</a> measures the performance of a

web application from end user's browsers and sends that data back to the

server for collection.</p>

</li>

<li>

<p><a href="http://sensuapp.org/">Sensu</a> is an open source monitoring framework

written in Ruby but applicable to any programming language web application.</p>

</li>

<li>

<p><a href="http://vimeo.github.io/graph-explorer/">Graph Explorer</a> by Vimeo is a

Graphite-based dashboard with added features and a slick design.</p>

</li>

<li>

<p><a href="http://packetbeat.com/">PacketBeat</a> sniffs protocol packets. Elasticsearch

then allows developers to search the collected data and visualize what's

happening inside their web application using the Kibana user interface.</p>

</li>

<li>

<p><a href="http://munin-monitoring.org/">Munin</a> is a client plugin-based monitoring

system that sends monitoring traffic to the Munin node where the data can

be analyzed and visualized. Note this project is written in Perl so Perl 5

must be installed on the node collecting the data.</p>

</li>

</ul>

<h2>Hosted monitoring services</h2>

<ul>

<li>

<p><a href="http://newrelic.com/">New Relic</a>. Application and database monitoring as

well as plug ins for capturing and analyzing additional data about tools in

your stack.</p>

</li>

<li>

<p><a href="http://copperegg.com/">CopperEgg</a> is lower-level monitoring on server and

infrastructure. It's popular with DevOps shops that are making changes to

their production environments and want immediate feedback on the results

of those modifications.</p>

</li>

<li>

<p><a href="http://status.io/">Status.io</a> focuses on uptime and response metrics

transparency for web applications.</p>

</li>

<li>

<p><a href="https://www.statuspage.io/">StatusPage.io</a> (yes, there's both a Status and

StatusPage.io) provides easy set up status pages for monitoring application

up time.</p>

</li>

<li>

<p><a href="http://www.pagerduty.com/">PagerDuty</a> alerts a designated person or group

if there are stability, performance, or uptime issues with an application.</p>

</li>

<li>

<p><a href="https://appenlight.com/">App Enlight</a> provides performance, exception and

error monitoring and is currently specific to Python web applications.</p>

</li>

</ul>

<h2>Monitoring resources</h2>

<ul>

<li>

<p><a href="http://www.paperplanes.de/2011/1/5/the_virtues_of_monitoring.html">The Virtues of Monitoring</a></p>

</li>

<li>

<p><a href="http://blog.docker.io/2013/07/effortless-monitoring-with-collectd-graphite-and-docker/">Effortless Monitoring with collectd, Graphite, and Docker</a></p>

</li>

<li>

<p><a href="http://matt.aimonetti.net/posts/2013/06/26/practical-guide-to-graphite-monitoring/">Practical Guide to StatsD/Graphite Monitoring</a>

is a detailed guide with code examples for monitoring infrastructure.</p>

</li>

<li>

<p>Bit.ly describes the

"<a href="http://word.bitly.com/post/74839060954/ten-things-to-monitor">10 Things They Forgot to Monitor</a>"

beyond the standard metrics such as disk & memory usage.</p>

</li>

<li>

<p><a href="http://aarvik.dk/four-linux-server-monitoring-and-management-tools/">Four Linux server monitoring tools</a></p>

</li>

<li>

<p><a href="https://blog.serverdensity.com/how-to-design-useful-monitoring-graphs-and-visualizations/">How to design useful monitoring and graphing visualizations</a></p>

</li>

<li>

<p><a href="https://speakerdeck.com/auxesis/5-years-of-metrics-and-monitoring">5 years of metrics and monitoring</a>

is a great presentation highlighting that visualization so humans can

understand measurements is a hard problem. Line graphs are often not

the best solution and they are overused.</p>

</li>

<li>

<p>The Collector Highlight Series has an article on <a href="http://blog.librato.com/posts/statsd">StatsD</a>

that explains how to install it and how it works.</p>

</li>

<li>

<p>This <a href="http://kartar.net/2014/11/monitoring-survey---tools/">survey on monitoring tools</a>

has some nice data and graphs on what developers and operations folks use

in their environments.</p>

</li>

<li>

<p>Ryan Frantz wrote a nice post on

<a href="http://ryanfrantz.com/posts/solving-monitoring/">Solving Monitoring</a>

with a new definition of what monitoring means based on today's complex

systems and how the practice should evolve going forward.</p>

</li>

</ul>

<h2>Monitoring learning checklist</h2>

Review the software-as-a-service and open source monitoring tools below. Third

party services tend to be easier to set up and host the data for you. Open

source projects give you more control but you'll need to have additional

servers ready for the monitoring.</p>

My recommendation is to install <a href="http://newrelic.com/">New Relic</a>'s free

option with the trial period to see how it works with your app. It'll give you

a good idea of the capabilities for application-level monitoring tools.</p>

As your app scales take a look at setting up one of the the open source

monitoring projects such as StatsD with Graphite. The combination of those

two projects will give you fine-grained control over the system metrics

you're collecting and visualizing.</p>

<h3>What topic do you want to learn next?</h3>

How do I learn more about the users of my app with web analytics?

</p>

</div>

</a>

What should I learn about web application security?

</p>

</div>

How do I automate the server configuration I've set up?

</p>

</div>

How should I log events in my application outside monitoring?

</p>

</div>

</div> <div id="mc_embed_signup">

<h4>Sign up here to receive an email with major updates to this site and Python tutorials delivered to your inbox once a month.</h4>

</div>

</div>

</form>

</div>

<h3 class="panel-head">Sponsored By</h3>

</div>

<p style="font-size: .8em; margin-top: 10px;">Want to improve monitoring

in your Python web app? I highly recommend

<a href="https://www.loggly.com/log-management/python-logging/?utm_source=fsp&utm_medium=promo&utm_campaign=march" onclick="trackOutboundLink('https://www.loggly.com/log-management/python-logging/?utm_source=fsp&utm_medium=promo&utm_campaign=march'); return false;">signing up for a free Loggly account</a>.

</p>

</div>

<h3 class="panel-head"><a href="/table-of-contents.html" style="color: #fff;">Table of Contents</a></h3>

</div>

<a href="/introduction.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Introduction</a>

<a href="/why-use-python.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Why Use Python?</a>

<a href="/best-python-resources.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Best Python Resources</a>

<a href="/best-python-videos.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Best Python Videos</a>

<a href="/development-environments.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Development Environments</a>

<a href="/emacs.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Emacs</a>

<a href="/generators.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Generators</a>

<a href="/comprehensions.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Comprehensions</a>

<a href="/web-frameworks.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Web Frameworks</a>

<a href="/django.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Django</a>

<a href="/flask.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Flask</a>

<a href="/bottle.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Bottle</a>

<a href="/pyramid.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Pyramid</a>

<a href="/morepath.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Morepath</a>

<a href="/other-web-frameworks.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Other Web Frameworks</a>

<a href="/web-design.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Web Design</a>

<a href="/cascading-style-sheets.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Cascading Style Sheets</a>

<a href="/javascript.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>JavaScript</a>

<a href="/websockets.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>WebSockets</a>

<a href="/web-application-security.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Web Application Security</a>

<a href="/databases.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Databases</a>

<a href="/no-sql-datastore.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>NoSQL Data Stores</a>

<a href="/application-programming-interfaces.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Application Programming Interfaces</a>

<a href="/api-integration.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>API Integration</a>

<a href="/api-creation.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>API Creation</a>

<a href="/deployment.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Deployment</a>

<a href="/servers.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Servers</a>

<a href="/platform-as-a-service.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Platform-as-a-service</a>

<a href="/operating-systems.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Operating Systems</a>

<a href="/web-servers.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Web Servers</a>

<a href="/wsgi-servers.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>WSGI Servers</a>

<a href="/source-control.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Source Control</a>

<a href="/application-dependencies.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Application Dependencies</a>

<a href="/static-content.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Static Content</a>

<a href="/task-queues.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Task Queues</a>

<a href="/configuration-management.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Configuration Management</a>

<a href="/continuous-integration.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Continuous Integration</a>

<a href="/logging.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Logging</a>

<a href="/monitoring.html" class="list-group-item smaller-item active" style='font-family: "Helvetica Neue",sans-serif;'>Monitoring</a>

<a href="/web-analytics.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Web Analytics</a>

<a href="/docker.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Docker</a>

<a href="/caching.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Caching</a>

<a href="/code-metrics.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Code Metrics</a>

<a href="/what-full-stack-means.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>What "Full Stack" Means</a>

<a href="/change-log.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Change Log</a>

<a href="/future-directions.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Future Directions</a>

<a href="/about-author.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>About the Author</a>

</div>

<div class="panel-heading"><h3 class="panel-head">Monitoring</h3></div>

Major updates are tweeted via

<a href="https://twitter.com/fullstackpython">@fullstackpython</a>.

<hr/>

Need more detailed tutorials than you see here?

<a href="/email.html">Sign up to receive an email when that content is created.</a>

</div>

</div></div>

<hr/>

<a href="http://www.mattmakai.com/" class="underline">Matt Makai</a>

2015

</div>

var trackOutboundLink = function(url) { ga('send', 'event', 'outbound', 'click', url, {'hitCallback': function () { document.location = url; } }); }

</script>

</body>

</html>

fullstackpython.github.com/monitoring.html at gh-pages · Intellifora/fullstackpython.github.com · GitHub

Expand file tree

Breadcrumbs

monitoring.html

Latest commit

History

Breadcrumbs

monitoring.html

File metadata and controls