When running any amount of systems, gaining visibility into what they are doing can be a non-trivial matter. Starting on the path to monitoring can prove bumpy, and if you don’t measure, you don’t know. In this session, Michael Fiedler, Director of TechOps, will speak on personal experience with scalability, deployment, and monitoring challenges prior to using Datadog - and how that changed. He will cover how to get started, and examples of where monitoring the company's platform with Datadog provided the guiding light towards the team solving scalability problems.

1. Why visibility into your
stack matters
or, Do you see it all?

2. Mike Fiedler
Operations
Datadog.com
Twitter: @mikefiedler
GitHub: @miketheman
OpsSchool.org
Chef Community
Roller Derby Referee
Skydiver
©Alex Erde

3. –CEO calling your cellphone at 03:00
“The site is slow.”

5. What?
• typical monitoring implementation story
• an alternative approach

6. (CC BY 2.0) http://www.gotcredit.com/ https://flic.kr/p/6439SA

7. LB
Data
User
Web

9. (CC BY 2.0) www.futurealpha.com https://flic.kr/p/8PhF4g

10. (CC BY 2.0) Aristocrats-hat https://flic.kr/p/6qdTC1
–W. Edwards Deming, The Elements of Statistical Learning
“In God we trust; all others bring data.”

13. You want more?

14. • graphite
• ganglia
• mongodb
• mysql
• influxdb
• socket.io
• datadog
• …

15. Time is a Cruel Master

16. (CC BY-SA 2.0)
https://www.flickr.com/theilr/
https://flic.kr/p/8MC5YM

17. Have
• systems
• applications
• services
• developers
• operators
• customers

18. Have
• systems
• applications
• services
• developers
• operators
• customers

19. Polyglot Platforms

20. Complex Systems

21. Disparate Locations

22. Information Overload

23. –CEO calling your cellphone at 03:00
“The site is slow.”

24. (CC BY 2.0) www.futurealpha.com https://flic.kr/p/8PhF4g

25. What exactly are we
monitoring, anyhow?

26. Top-down
• work metrics
• resource metrics
• events

27. Work Metrics
throughput, success vs error, performance

28. Resource Metrics
utilization, saturation, errors, availability

29. Events
change/build/deploy, alerts, etc

30. Trend resource metrics,
notify on changes

31. Wake people up when
work metrics go awry

34. Slice and Dice

36. Set-and-Forget

38. Just-In-Time
Information

40. Does it scale?

41. Customer Stats
• AdRoll, ~2m transactions/second
• SimpleReach, ~7b measurements/day
• MercadoLibre, ~18k hosts monitored
• AirBnB, 3000+ monitors defined

42. –M. Fiedler
“If you don’t measure, you don’t know.”

43. Questions?

44. Mike Fiedler
Operations
Twitter: @mikefiedler
GitHub: @miketheman
OpsSchool.org
Chef Community
Roller Derby Referee
Skydiver
©Alex Erde