This document discusses the shift from a bottom-up to a top-down approach in system monitoring. A top-down approach focuses on a few key performance indicators (KPIs) that directly measure overall system health and user experience, rather than monitoring many individual components. The document provides examples from Netflix, GitHub, and BigPanda of using single, high-level metrics as KPIs to monitor critical business activities and detect problems. While a bottom-up approach is easier initially, a top-down approach using KPIs helps avoid alert fatigue and keeps focus on what is most important to the business.

1. Top-Down Approach to Monitoring
July 30, 2015

2. 1996
2
Tivoli Software
acquired by IBM
Patrol Software
acquired by BMC
Ethan Galstad creates a simple
MS-DOS application designed to
"ping" Novell Netware servers
“HOW to monitor?” is the primary question

3. 2015
3
https://www.bigpanda.io/monitoringscape/

4. Shifting from “How?” to “What?”
4

5. 5

6. Bottom-Up Approach
6
Network Servers Apps
Overall System Health

7. Problem #1: Inflation of Tools
7

8. Problem #2: Inflation of “Whats”
8

9. Problem #3: Inflation of Alerts
9

10. 10

11. 11
We’re trying to answer a simple question:
Is our system in a healthy state?

12. 12
No Alerts
Many Alerts Unhealthy System≠
≠ Healthy System

13. 13
Healthy System =
A system that continuously
generates value for its users
under a well known set of KPIs

14. Top-Down Approach
14
KPIs UX
Overall System Health

15. 15
KPIs UX
Overall System Health Network Servers Apps
Overall System Health
• Selective
• Proactive
• Exhaustive
• Reactive
vs
Bottom-UpTop-Down

16. A key performance indicator (KPI)
is a business metric used to
evaluate factors that are crucial to
the success of an organization.
KPIs differ per organization;
Definition of KPI
16

17. Let’s play a game!
17
CPU Utilization # Clicks on
a button
TemperatureThis is Sam
What does Sam’s company do?

18. We sought out a single indicator that closely approximated our most
important activity: viewing. We discovered that a server-side metric
related to playback starts (the act of “clicking play”) had both a
predictable pattern and fluctuated significantly when UI/device/server
problems were happening. The Netflix streaming pulse was created.
The Pulse of Netflix
18
http://techblog.netflix.com/2015/02/sps-pulse-of-netflix-streaming.html
We named it “SPS” for “starts per second”.

19. Healthy SPS Pattern
19
http://techblog.netflix.com/2015/02/sps-pulse-of-netflix-streaming.html

20. Unhealthy SPS Pattern
20
http://techblog.netflix.com/2015/02/sps-pulse-of-netflix-streaming.html

21. What’s so special about SPS?
21
• SPS is easy to understand by all stakeholders
• One metric that covers different point of failure: server
problems, device problems, etc.
• Most important: it’s a clear KPI that indicates when user
experience is compromised

22. But what about root cause analysis?
22
KPIs UX
Overall System Health
Network Servers Apps

23. Github: need for speed
23
https://github.com/blog/1252-how-we-keep-github-fast
The most important factor in web application
design is responsiveness. And the first step
toward responsiveness is speed. But speed
within a web application is complicated.

24. Start from the Top:
Response Times Dashboard
24
https://github.com/blog/1252-how-we-keep-github-fast
• Each row represented a different major
component
• Clicking one of the rows allows you to dive in
and see the mean, 98th percentile, and 99.9th
percentile response times

25. Digging Deeper:
Mission Control Bar
25
https://github.com/blog/1252-how-we-keep-github-fast
Total Time Render Time Cache & Database JS & CSS Size

26. And Deeper
26
https://github.com/blog/1252-how-we-keep-github-fast
Render Breakdown
SQL Query Viewer

27. 27
Why talk about BigPanda?
Because Pandas
are awesome!

28. BigPanda
28
Because..
• We’re not Netflix or Github: growing startup (7 devs, 1 full-time Ops)
• We feel the pain!
• Our KPIs are easy to describe and understand (especially if you’re an
Ops person)

29. BigPanda
29
As a unified dashboard on top of all your
monitoring systems, and eventually a single
point of truth for production incidents, our data
pipeline has to be reliable and fast.
KPI: Low data pipeline latency

30. Pipeline Latency Metric
30
• Metric are sent from within the apps
• Stored in Graphite
• Sum of all the average latencies of
all alerts that went through the
pipeline
• Monitored by Nagios

31. • Very good indicator of possible service outage
• Must have for detection of SLA violation
• Very good indicator of performance
bottlenecks (can be broken down to sub-
pipelines / specific organizations etc)
• Simple and high-level: easy to explain to non-
technical stakeholders (e.g. sales)
Pipeline Latency Metric
31

32. • Bottom-up approach (“monitor all the things”) is easier to start with, but soon enough
leads to alert fatigue and disorientation.
• Top-down approach requires thought and custom instrumentation, but keeps you
focused on what’s important.
• High level metrics can be complemented by low level metrics. Trying to deduce the
former from the latter is futile.
• Take advantage of the rich monitoring landscape, but as means to an end. Don’t let the
tools dictate to you what you need to measure.
• Monitoring is - first of all - about your business.
TL;DR
32

33. 33
Questions?

34. 34
Thanks!