Monitoring is
Never “Done”
@melaniemj

Responsibilities @ Yardi
Implementation and administration of monitoring,
alerting, and log aggregation/analysis tools.
o 15,000+ Devices
o 9 Datacenters
o 5000+ Customer Installations
o We monitor windows envs with linux envs

This was me in 2008 @ Point2

How code is delivered

How code operates in production

A good problem to have
Everyone wants “the monitoring” so they can say
“it’s monitored”

Communicating Work
o Classify
o Quantify
o Qualify

Words....
o Logging
o Alerting
o Dashboards
o Reports
o 4-9s
o 24x7x365 this shit can’t go down

Can it be this simple?
Let’s talk about “the monitoring” for X
Be awesome
X is monitored

DCVA (OODA)

1. Definition
I can hit this one page so it’s up right?
No thanks, let’s redefine status

1. Definition
o What questions are you trying to answer?
o What information do you need when a failure
occurs?
o What are the most common failures?
o Who is the audience for the information?

2. Checks & Collections
o Environment & Code
o Data points
o Detailed logs
o Current state

3. Visualization
o Analysis
o Dashboards
o Correlations

4. Action
o Fault detection
o Alerting
o RCA

Cycle
(What to collect)
(Inform on failure) (How to collect)
(Make collections pretty)

Team Time Distribution

Time Distribution (Desired)

Is “X” monitored?
When “X” goes into some degraded state
o The right people know.
o They have enough information to find the
problem, recover, and later to do RCA.
o If they don’t they will revisit definition.

How does your team
o Classify
o Quantify
o Qualify

Monitoring is Never “Done”
Melanie Cey
@melaniemj
Senior Systems Analyst
Systems Reliability Engineering
@ Yardi