This document discusses holistic approaches to monitoring systems and applications. It emphasizes the importance of monitoring business metrics, system performance, and failure metrics. It recommends defining metrics to monitor, collecting code-level metrics using tools like StatsD and Graphite, and collecting environment metrics from operating systems and databases. It also stresses the importance of visualization through different types of dashboards and anomaly detection. Action items include creating useful alerts and dashboards, adding anomaly detection, and exploiting failures to improve monitoring.

1. Thank you to our Sponsors
A Holistic Approach to Monitoring
Melanie Cey – Yardi Systems Inc.
Media Sponsor:

2. @melaniemj
Systems Analyst in DevOps (Web Operations) @ Yardi
• 5 years Programming
• 3.5 years Team Lead/Project Manager
• 4 years Systems Administration/Analysis

4. Because
• Customers should not alert you to failure
• Business metrics matter
• When something fails you need enough info to know why
• Agile teams release frequently
• No one can afford to be reactive

5. When you release code…

6. Monitoring Cycles

7. Definition: What to measure
• Business Metrics & Events
- Login/logout
- Sign up, buy something
- Sent email
• System Events, Performance and Utilization Metrics
- Web Service Call details (counter / time taken)
- Deployments
- Cache system (e.g. Redis or other) hits / misses
- Environment performance
• Failure Metrics
- Exceptions, segregated by type / app / server of origin
- Number and type of errors that reached customers

8. Code Collection

9. Code Collection – Add / Refine Stats
• Developer Friendly Platform
- Stats need to be able to be added ‘without permission’
- Create own dashboards
- Tools with APIs
- Build client library for sending stats

10. Code Collection – Graphite
• Using Graphite
- (Etsy 2011) StatsD UDP Node.js daemon collects and
aggregates
- Sends stats (as strings) to Graphite where they are stored in
Whisper (like RRD) files
- Graphite has a web interface, url api (with a json output option)
and built in ability to create dashboards
- Can receive stats from anything and is easy to setup
- Open source with lots of industry use
- Plenty of built in functions to help analyze and visualize data

11. Code Collection – Graphite

12. Code Collection – Add / Refine Stats

13. Code Collection – Graphite Samples

14. Code Collection – Logging
• Metrics – what and when
• Logging – how and why

15. Code Collection – Add / Refine Logging
• Why Log and what to log?
- Log when you record a statistic
• Logging Best Practices
- Log locally
- Don’t log to your production database server
- Don’t fail if you can’t log
- Log in GMT
- Keep your logs, ship them to a central location
- Aggregate recent data in real time if you can
- Log more than you think you need to
- Use a parse friendly format

16. Environment Collection

17. Environment Collection
• Operating Systems
- CPU, Free Memory, Paging, I/O ms speeds, network utilization
• Database Management Systems
- Transactions, blocks
• Application Containers
- Memory utilization, IIS requests current & queued, restarts,
cache statistics etc.

18. Visualization

19. Visualization
• Types of Dashboards
- Feature based
- Resource based (server or container)
- Performance
- Anomaly detection
- Correlation
- Root Cause Analysis
- “Overview”

20. Visualization – Tasseo
• https://github.com/obfuscurity/tasseo

21. Visualization – Cubism
• https://github.com/square/cubism

22. Visualization – Cubism
• https://github.com/square/cubism

23. Action: Putting inside knowledge to work

24. Action
• Useful dashboards help create useful alerts
• Add / refine anomaly detection & alerting
• Know your own boundaries
• A fuzzy threshold is better than no threshold
• Attach graphs to alerts
• Exploit failures
- Add an alerts after RCA
- Theorize other possible causes or conditions

25. Monitoring Cycles

26. More?
• http://graphite.readthedocs.org/en/latest/
• http://codeascraft.com/
• http://vimeo.com/monitorama
• Twitter #devops #monitoringlove
• https://github.com/monitoringsucks
• http://www.opsschool.org/en/latest/