This document discusses monitoring for online services and microservices. It notes that companies are moving to public/private clouds and microservices while also adopting DevOps practices. Traditional monitoring stacks are discussed, as well as how Netflix and Expedia monitor usage trends rather than individual alerts. Data from 60 companies is presented on how monitoring needs change as servers increase. The key is allowing development teams to own monitoring for the services they create and support. Dataloop aims to provide simple, self-service metric collection and dashboards so monitoring is decoupled from release cycles and more data-driven.

1. www.dataloop.io | @dataloopio | info@dataloop.io
Monitoring for Online Services

2. Obligatory about me slide..
• Born a SysAdmin
• Always worked in software companies
• Head of Ops a bunch of times
• Helped launch quite a few SaaS products
• Founded Dataloop.IO end of 2013
• Worked solely on monitoring ever since

3. Context of talk
Companies running online services at scale who are are on public /
private cloud, moving to micro-services and ‘doing’ devops

4. The World As We Know It
Application
MySQL
Database
OpsView
(Nagios)
Logstash ElasticSearch Kibana
AppDynamicsPingdom
GraphiteCollectD
PagerDuty
Amazon AWS
Alfresco JVM
SOLR
Transformations
Browser
Google
Analytics
Custom Scripts
Reporting
System
SQL DB’s
Mixpanel GoSquared
Geckoboard

5. Netflix’s “Monitoring System”
“We aggregate the call volume from all of our offices and when that changes
significantly we investigate. It then might be a case of letting Microsoft know
that Xbox is down”

6. Expedia’s “Monitoring System”
“We have a very consistent shape to our bookings graph. If that changes by
more than a few % throughout the day we know something is up even if
nothing else is alerting”

7. Our Sample

8. The Results
http://blog.dataloop.io/2014/01/30/what-we-learnt-talking-to-60-companies-about-monitoring/

9. How Your Tools Change with No. of Servers

10. Your Typical Monitoring Stack
Is my site up or down?
(External)
What happened?
(Logs)
How is my application performing?
(APM)
What’s my app actually doing?
(Custom Metrics)
Is everything working as expected?
(Service)
Dashing
(Custom Dashboards)

11. Your Automation Today
• Tied to releases
• Often a bit of a mess
• Ops centric
• High friction
• Bus factor

13. Attempts to improve things
• Educate, educate, educate! (and then encourage devs to learn cm)
• Automation and ‘just drop the script here’ in the repo
• AppDynamics and New Relic
• Sensu, Check_MK and custom portals / wiki pages etc

14. Why?

15. Making Software Is Comparable to Manufacturing

16. Books!

17. How Software Is Developed

18. The Solution
Technology only gets you so far. To be really successful
you need to devolve monitoring down to the teams that
own their part of the service.

19. The key to good monitoring

20. Why You Need An Operational Historian
http://en.wikipedia.org/wiki/Operational_historian

21. Developers Must Own Your Monitoring
• Devs outnumber Ops 7 – 1 in micro-services
• The best person to write the check script
• Responsible for the service not just code
• Production data makes everyone cleverer
• Bridges the gap between Dev + Ops
• Data driven decisions!

22. Our focus at Dataloop
Simple setup by Ops team (agent install and tags)
Self service metric collection via ‘create in the browser’ script
editor
Decoupled script deployment from cm via drag and drop
Real-time dashboards that anyone can create
Simple alerting rules that teams can manage themselves

23. The infinite loop

24. Operational Intelligence != Analytics

25. www.dataloop.io
@dataloopio
info@dataloop.io

26. Red Stops The floor

27. Public Status

28. Dataloop Crown Jewells

29. Deployment Radiator

30. Visually Defeating GHOST

31. Q&A