Your SlideShare is downloading. ×
Monitoring and observability
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Monitoring and observability


Published on

Published in: Technology, Design

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Monitoring & ObservabilityGetting off the starting blocks.Wednesday, June 19, 13
  • 4. AgendaDefine stuff.Set some tenets.Discuss and implement some tenets.Answer a lot of questions.Wednesday, June 19, 13
  • 5. Monitoring... what it is.We’ll get to that.Wednesday, June 19, 13
  • 6. ObservabilityBeing able to measure “things” orwitness state changes.Not useful if doing so alters behavior (significantly).Wednesday, June 19, 13
  • 7. Development & ProductionFor the rest of this talk...There is only production.Wednesday, June 19, 13
  • 8. Data & Information TermsMeasurement: a single measurement of somethinga value on which numerical operations make sense:1, -110, 1.234123, 9.886-19, 0, null“200”, “304”, “v1.234”, “happy”, nullWednesday, June 19, 13
  • 9. Data & Information TermsMetric: something that you are measuringThe version of deployed codeTotal cost on Amazon servicestotal bugs filed, bug backlogTotal queries executedWednesday, June 19, 13
  • 10. Notice no ratesDO NOT STORE RATES.Wednesday, June 19, 13
  • 11. Measurement VelocityThe rate at which new measurements are taken.Wednesday, June 19, 13
  • 12. PerspectiveSometimes perspective matterspage load times, DNS queries,consider RUM (real user monitoring)Usually it does nottotal requests made against a web serverWednesday, June 19, 13
  • 13. VisualizationThe assimilation ofmultiple measurements intoa visual representation.Wednesday, June 19, 13
  • 14. TrendingUnderstanding the“direction” of series of measurements on a metric.Here direction is loose and means “pattern within.”Wednesday, June 19, 13
  • 15. AlertingTo bring something to one’s attention.Wednesday, June 19, 13
  • 16. Anomaly DetectionThe determination that aspecific measurement isnot within reason.Wednesday, June 19, 13
  • 17. Monitoring... what it is.All of that.Wednesday, June 19, 13
  • 18. ReviewMeasurementMeasurement VelocityMetricPerspectiveVisualizationTrendingAlertingAnomaly DetectionObservabilityMonitoringWednesday, June 19, 13
  • 19. Some TenetsMost people suck at monitoring.They monitor all the wrong things (somewhat bad)The don’t monitor the important things (awful)Wednesday, June 19, 13
  • 20. Do not collect rates of thingsRates are like trees making sounds falling in the forest.Direct measurement of rates leads to data lossand ultimately ignorance.Wednesday, June 19, 13
  • 21. Prefer high level telemetry1. Business drivers via KPIs,2. Team KPIs,3. Staff KPIs,4. ... then telemetry from everything else.Wednesday, June 19, 13
  • 22. ImplementationHerein it gets tricky.Wednesday, June 19, 13
  • 23. Only because of the tools.I could show you how to use tool X, or Y or Z.But I wrote Reconnoiter and founded Circonusbecause X, Y and Z didn’t meet my needs.Reconnoiter is open.Circonus is a service.Wednesday, June 19, 13
  • 24. MethodologyI’m going to focus on methodologythat can be applied across whatever toolset you have.Wednesday, June 19, 13
  • 25. Pull vs. PushAnyone who says one is better than the other is...WRONG.They both have their uses.Wednesday, June 19, 13
  • 26. Reasons for pull1. Synthesized observation is desirable.2. Observable activity is infrequent.3. Alterations in observation frequency are useful.Wednesday, June 19, 13
  • 27. Reasons for pushDirect observation is desirable.Discrete observed actions are useful.Discrete observed actions are frequent.Wednesday, June 19, 13
  • 28. False reasons.Polling doesn’t scale.Wednesday, June 19, 13
  • 29. Protocol SoupThe great thing about standards is...there are so many to choose from.Wednesday, June 19, 13
  • 30. Protocol SoupSNMP(v1,v2,v3) both push(trap) and pull(query)collectd(v4,v5) push onlystatsd push onlyJMX, JDBC, ICMP, DHCP, NTP, SSH, TCP, UDP, barf.Wednesday, June 19, 13
  • 31. Color me RESTyUse JSON.HTTP(s) PUT/POST somewhere for pushHTTP(s) GET something for pullWednesday, June 19, 13
  • 32. High-volume DataOccasionally, data velocity is beyond what’s reasonablefor individual HTTP PUT/POST for each observation.1. You can fall back to UDP (try statsd)2. I prefer to batch them and continue to use RESTWednesday, June 19, 13
  • 33. nadnad is great. use nad. of it like an SNMP that’sactually SimpleMonitoring not Managementand trivial extended to suit your needsWednesday, June 19, 13
  • 34. nad online exampleTo the Internet ➥Wednesday, June 19, 13
  • 35. But wait...nad isn’t’s technology.Wednesday, June 19, 13
  • 36. Correct...Back to the topic.I talked about nad briefly to provide asuper simple tool to erase the question:“but how?”Wednesday, June 19, 13
  • 37. The real question is: “what?”What should I be monitoring?This is the best question you can ask yourself.Before you start.While you’re implementing.After you’re done.Wednesday, June 19, 13
  • 38. The industry answer:MONITOR ALL THE THINGS!I’ll tell you this too, in fact.But we have put the cart ahead of the horse.Wednesday, June 19, 13
  • 39. Question?If I could monitor one thing, what would it be?hint: CPU utilization on your web server ain’t it.Wednesday, June 19, 13
  • 40. Answer:It depends on your business.If you don’t know the answer to this,I suggest you stop worrying about monitoringand start worrying about WTF your company does.Wednesday, June 19, 13
  • 41. Here, we can’t continue.Unless I make stuff up...So, here I go makin’ stuff up.Wednesday, June 19, 13
  • 42. Let us assumewe run a web site where customers buy productsWednesday, June 19, 13
  • 43. Monitoring purchases.So, we should monitor how many purchases weremade and ensure it is within acceptable levels.Not so fast.Wednesday, June 19, 13
  • 44. Actually.We want to make sure customerscan purchase from the site andare purchasing from the site.This semantic different is critically important.And choosing which comes down to velocity.Wednesday, June 19, 13
  • 45. What is this velocity thing?Displacement / time(i.e. purchases/second or $/second)BUT WAIT! You said:“Do not collect rates of things.”Correct...collect the displacement,visualize and alert on the rate.Wednesday, June 19, 13
  • 46. So which?High velocity w/ predictably smooth trends:velocity is more importantLow velocity or uneven arrival rates:measuring capability is more importantWednesday, June 19, 13
  • 47. To rephraseIf you have sufficient real data,observing that data works best;otherwise, you mustsynthesize data and monitor that.Wednesday, June 19, 13
  • 48. As a tenet.Always synthesize.additionally observe real data when possibleWednesday, June 19, 13
  • 49. More demonstrable(in a short session)I’ve got a web site that my customers need to visit.The business understands that we need to servecustomers with at least a basic level of QoS:no page loads over 4sWednesday, June 19, 13
  • 50. Active checks.Wednesday, June 19, 13
  • 51. A first attemptcurl the HTTP response codeif 200, we’re super good!Admittedly not so good.Wednesday, June 19, 13
  • 52. A wealth of data.Synthesizing an HTTPS GET could provide:SSL Subject, validity, expirationHTTP code, Headers and ContentTimings on TCP connection, first byte, full payloadWednesday, June 19, 13
  • 53. Still, this is highly imperfect.Don’t get me wrong, they are useful.We use them all over the place... they are cheap.But, ideally, you want to load the page closer to theway a user does (all assets, javascript, etc.)Enter phantomjsWednesday, June 19, 13
  • 54. var page = require(webpage).create();page.viewportSize = { width: 1024, height: 768 };page.onError = function(err) { stats.errors++; };page.onInitialized =function() { start = new Date(); };page.onLoadStarted =function() { stats.load_started = new Date() - start; };page.onLoadFinished =function() { stats.load_finished = new Date() - start; };page.onResourceRequested = function() { stats.res++; };page.onResourceError = function(err) { stats.res_errors++; };page.onUrlChanged = function() { stats.url_redirects++; };, function(status) {stats.status = status;stats.duration = new Date() - start;console.log(JSON.stringify(stats));phantom.exit();});Wednesday, June 19, 13
  • 55. var start, stats = {status: null, errors: 0, load_started: null, load_finished: null, resources: 0, resource_errors: 0, url_redirects: 0};Wednesday, June 19, 13
  • 56. Passive checks.Wednesday, June 19, 13
  • 57. Now for the passive stuffSome examples are Google Analytics, Omniture, etc.Statsd (out-of-the-box) and Metricsare mediocre approach.If we have a lot of observable data N,N̅ isn’t so useful,𝜎, |N|, q(0.5), q(0.95), q(0.99), q(0), q(1), add a lot.Wednesday, June 19, 13
  • 58. Still... we can do better.N̅, 𝜎, |N|, q(0,0.5,0.95,0.99,1) is 8 statistical aggregatesLet’s look at API latencies...say we do 1000/s,that’s 60k/minute.Over a minute of time, 60k points to 8 represents...a lot of information loss.Wednesday, June 19, 13
  • 59. First 60k/minute, how?statsdhttp putslogsetc.Wednesday, June 19, 13
  • 60. HistogramsWednesday, June 19, 13
  • 61. A line graph of data.Wednesday, June 19, 13
  • 62. A heatmap of data.Wednesday, June 19, 13
  • 63. Zoomed in on a heatmap.Wednesday, June 19, 13
  • 64. Unfolding to a histogram.Wednesday, June 19, 13
  • 65. ObservabilityI don’t want to launch into a tutorial on DTracedespite the fact that you can simple spin up anOmniOS AMI in Amazon and have it now.Instead let’s talk about what shouldn’t happen.Wednesday, June 19, 13
  • 66. The production questions:I wonder if that queue is backed up...Performance like that should only happen if our binarytree is badly imbalanced (replace with countless otherpathologically bad precipitates of failure); I wonder if itis...It’s almost like some requests are super slow; I wonderif they are.STOP WONDERING.Wednesday, June 19, 13
  • 67. Instrument your softwareInstrument your software and systemsand stop the wonderDo it for the kidsThis is simple with DTrace & a bit more work otherwiseAvoiding work is not an excuse for ignoranceWednesday, June 19, 13
  • 68. A tour through our SaunaWe have this software that stores data...happens to store all data visualized in Circonus.We have to get data into the system.We have to get data out of the system.I don’t wonder... here’s why.Wednesday, June 19, 13
  • 69. A small backgroundwith metrics intermingled.To the Internet ➥Wednesday, June 19, 13
  • 70. Bad habits.While monitoring all things is a good approach,alerting on things that do not have specific remediationrequirements is horribly damaging.Wednesday, June 19, 13
  • 71. Data tenet.Do not collect data twice.That which you collect for visualizationshould be the same data on which you alert.Wednesday, June 19, 13
  • 72. Alerting tenet.A ruleset against metrics in the system should neverproduce an alert without documetation:the failure condition in plain English,the business impact of the failure condition,a concise and repeatable remediation procedure,an escalation path up the chain.Wednesday, June 19, 13
  • 73. Alerting post mortemsTry this out:for each alert, run a post mortem exerciseunderstand why it alerted, what was done to fixrehash who the stakeholders arehave them in the meetinghave the stakeholder speak to the business impactWednesday, June 19, 13