1.
Why visibility into your
stack matters
or, Do you see it all?

2.
Mike Fiedler
Operations
Datadog.com
Twitter: @mikefiedler
GitHub: @miketheman
OpsSchool.org
Chef Community
Roller Derby Referee
Skydiver
©Alex Erde

3.
–CEO calling your cellphone at 03:00
“The site is slow.”

4.
What?
• typical monitoring implementation story
• an alternative approach

5.
(CC BY 2.0) http://www.gotcredit.com/ https://flic.kr/p/6439SA

6.
LB
Data
User
Web

7.
(CC BY 2.0) www.futurealpha.com https://flic.kr/p/8PhF4g

8.
(CC BY 2.0) Aristocrats-hat https://flic.kr/p/6qdTC1
–W. Edwards Deming, The Elements of Statistical Learning
“In God we trust; all others bring data.”

9.
You want more?

10.
• graphite
• ganglia
• mongodb
• mysql
• influxdb
• socket.io
• datadog
• …

11.
from bottle import route
import pymongo
import json
db = pymongo.Connection(‘mongodb://...
@route('/insert/:name')
def insert(name):
doc = {'name': name}
db.words.update(
doc, {"$inc":{"count": 1}}, upsert=True
)
return json.dumps(doc, default=default)

12.
from bottle import route
import pymongo
import json
from statsd import statsd
db = pymongo.Connection(‘mongodb://...
@route(‘/insert/:name')
@statsd.increment('wordcount.insert')
def insert(name):
doc = {'name': name}
db.words.update(
doc, {"$inc":{"count": 1}}, upsert=True
)
return json.dumps(doc, default=default)

13.
Time is a Cruel Master

14.
(CC BY-SA 2.0)
https://www.flickr.com/theilr/
https://flic.kr/p/8MC5YM

15.
Have
• systems
• applications
• services
• developers
• operators
• customers

16.
Have
• systems
• applications
• services
• developers
• operators
• customers

17.
Polyglot Platforms

18.
Complex Systems

19.
Disparate Locations

20.
Information Overload

21.
–CEO calling your cellphone at 03:00
“The site is slow.”

22.
(CC BY 2.0) www.futurealpha.com https://flic.kr/p/8PhF4g

23.
Does this matter?

24.
Top-down
• work metrics
• resource metrics
• events

25.
Work Metrics
throughput (rps), success/error, performance (latency)

26.
Resource Metrics
utilization (%busy), saturation (queued), errors, availability

27.
Events
change/build/deploy, alerts, anything notable

28.
Trend resource metrics,
notify on changes

29.
Wake people up when
work metrics go awry

30.
Slice and Dice
exploration and aggregation

31.
Set-and-Forget

32.
Just-In-Time
Information

33.
Does it scale?

34.
Customer Stats
• AdRoll, ~2m transactions/second
• SimpleReach, ~7b measurements/day
• MercadoLibre, ~18k hosts monitored
• AirBnB, 3000+ monitors defined

35.
–CEO calling your cellphone at 03:00
“The site is slow.”

36.
–You
“Thanks. We know, and are already investigating.”

37.
–You, because you never got that call in the first place due to
proactive data collection and alerting.
“[silence]”

38.
Questions?

39.
–M. Fiedler, Twitter: @mikefiedler
“If you don’t measure, you don’t won’t know.”