It's all about telemetry

It’s all about telemetry Monitoring what matters in a useful way.Tuesday, June 26, 12

Theo Schlossnagle @postwait I write software I write books I give talks I participate in the industry I speak frankly about industry issuesTuesday, June 26, 12

Data, data, everywhere. A billion pageviews / month. 100k database queries / second. 1MM memcache queries / second. 500k MQ messages / second. 10MM I/O operations / second.Tuesday, June 26, 12

Big Data Most new big data problems are solvableTuesday, June 26, 12

Big Data Most new big data problems are created by our solutions, and thus solvable despite their ROITuesday, June 26, 12

That’s a whole lot of data Think in terms of logs (too many do) About 26 trillion log lines / month @ 40 bytes compressed: 1PB / month Just because it is possible does not mean it will return on investment (and does not mean it won’t)Tuesday, June 26, 12

It’s all “useful”; which data? Think in terms of cost/beneﬁt. Sure the data is useful, but it costs money to store Does it cost you more to have it or not to have it? Maybe the right approach is to keep that level of detail for a few days?Tuesday, June 26, 12

Double-edged sword. Eroding granularity over time keeps storage under controlTuesday, June 26, 12

Double-edged sword. K E TA Eroding granularity over time S keeps storage under control M ITuesday, June 26, 12

1 year at a glanceTuesday, June 26, 12

1 week looks normalishTuesday, June 26, 12

1 day conﬁdence of normalcy increasesTuesday, June 26, 12

1 week that looks differentTuesday, June 26, 12

1 day yup, that’s not at all like that other weekTuesday, June 26, 12

Other methods What do you store? How do you store it? Why is it useful? Winning the cost benefit game by reducing costs more significantly than reducing benefitsTuesday, June 26, 12

0 0.5 1 1.5 2 2.5 3 1 eﬁt Ben o st C 0.75 0.5 0.25 monitoring activity ➠ Positive Value Be in the green.Tuesday, June 26, 12

0 1 2 3 4 5 6 7 8 9 10 10 7.5 5 o st C 2.5 Beneﬁt monitoring activity ➠ There’s a bigger picture It’s not as easy as you think.Tuesday, June 26, 12

0 0.5 1 1.5 2 2.5 3 1 eﬁt Ben o st C 0.75 0.5 0.25 monitoring activity ➠ Value is difference, not area Green can be misleadingTuesday, June 26, 12

0.5 1 1.5 2 2.5 3 0.5 0.25 -0.25 -0.5 monitoring activity ➠ -0.75 Value = Beneﬁt - Cost -1 Green means we have positive returnTuesday, June 26, 12

0.5 1 1.5 2 2.5 3 0.5 0.25 -0.25 -0.5 monitoring activity ➠ -0.75 It’s not about return -1 Well, it’s not only about returnTuesday, June 26, 12

0.5 1 1.5 2 2.5 3 0.5 0.25 -0.25 -0.5 monitoring activity ➠ -0.75 It’s about maximizing return -1 This is a bit like black magicTuesday, June 26, 12

Technique 1: text Store changesTuesday, June 26, 12

Technique 2: numeric Store rollups (i.e. statistical aggregates over ﬁxed windows) over 1 minute store min/max/avg/stddev/covariance/50%/95%/99% lots of information heavy lossy compression of high-frequency data loses population distribution informationTuesday, June 26, 12

Database replication Lag (green) and rate of lag change (purple)Tuesday, June 26, 12

Storage Usage We can see growth. More useful, we can use this to project.Tuesday, June 26, 12

With simple numeric dataTuesday, June 26, 12

With simple numeric data Unknowns can be predictedTuesday, June 26, 12

With simple numeric data In sane ways with conﬁdenceTuesday, June 26, 12

Full Disclosure You see awesome examples of predictive analytics Like the real-world one on the previous slide In practice, almost all data streams predict one thing: they have no fucking clue.Tuesday, June 26, 12

Technique 3: histograms Store histograms over 1 minute store counts of datapoints seen in various buckets retains complete population distribution loss of precisionTuesday, June 26, 12

Histograms 101 This. This is a histogram. It shows the frequency of values within a population. Height represents frequencyTuesday, June 26, 12

Histograms 101 This. This is a histogram. It shows the frequency of values within a population. Now, height and color represents frequencyTuesday, June 26, 12

Histograms 101 This. This is a histogram. It shows the frequency of values within a population. Now, only color represents frequencyTuesday, June 26, 12

Histograms ➠ time series This. This is a histogram. It shows the frequency of values within a population. Now, only color represents frequencyTuesday, June 26, 12

Histograms ➠ time series This. This is a histogram. It shows the frequency of values within a population. Now, only color represents frequency at a single time intervalTuesday, June 26, 12

API Service Times We can see a full population shift of several millisecondsTuesday, June 26, 12

Combining techniques In our system (as a reference point) Arbitrary numbers of numeric data points on a single stream occupy 32 bytes of space for statistical aggregates and occupy about 2k of space for a histogram These means we can store these transforms on numeric data in perpetuityTuesday, June 26, 12

Combining techniques Text is a bit harder You need to be careful Some data sources can be constantly changing Producing gobs of change data You’re doing it wrong Find these and ﬁx themTuesday, June 26, 12

Correlating Events Change Management vs. PerformanceTuesday, June 26, 12

What to monitor? Most people don’t monitor the things that matter mostTuesday, June 26, 12

Monitor the Business Financials: Revenues. Costs. Margins. AR. Account delinquency. Marketing: Web analytics. Campaigns. Costs. Returns. Convergence.Tuesday, June 26, 12

Monitor the Support Customer Service: Problems. Time investment. Customer satisfaction. Resolution time.Tuesday, June 26, 12

Monitor the Engineering Engineering: Deployments. Test coverage. Bug reports. Bug ﬁxes. Effort spent. Operations: Faults. Pages. Escalations. Provisioning time. Equipment defect rates. 3rd party failure rates.Tuesday, June 26, 12

Monitor the Service Systems: Networks. Systems. Storage. Databases: Performance. Error rates. Backups. Middleware: Herein lies the magic and room for awesomenessTuesday, June 26, 12

Monitor the Middleware Your systems are complex Monitor their interactions Messaging, APIs, etc.Tuesday, June 26, 12

Monitor all the things. But, perhaps most importantly...Tuesday, June 26, 12

Monitor all the things. But, perhaps most importantly... USE UNIFIED TOOLINGTuesday, June 26, 12

What we use... reconnoiter SNMP, nad, resmon, statsd, HTTP traps, jdbc, etc. statsd (clients) javascript beaconsTuesday, June 26, 12

Middleware mix API service times, trafﬁc, user signup rates.Tuesday, June 26, 12

Tuesday, June 26, 12

Thank you!Tuesday, June 26, 12

http://cmgbrasil.posterous.com	228
http://lanyrd.com	24
http://eventifier.co	13
http://www.brijj.com	3
http://posterous.com	3
http://cmgbrasil.com	3
https://si0.twimg.com	1

It's all about telemetry

by Theo Schlossnagle on Jun 26, 2012

Accessibility

Categories

Upload Details

Usage Rights

7 Embeds 275

Statistics

It’s all about telemetry Presentation Transcript