This document discusses monitoring democracy and how monitoring systems can be improved. It advocates for monitoring as code, scalability, modularity, and allowing anything to be monitored. The monitoring system discussed uses Prometheus for metrics, Elastic Search for logs, and is open source with many contributors. The goal is to provide better visibility while tightly coupling monitoring with code.

1. Monitoria -
A Monitoring Democracy
@yaronidan @SolutoEng

3. Does monitoring != production?
Let’s think about it
=

4. Can alerts suck less?
Let’s think about it

5. Who carries the pager?
Let’s think about it

6. One person

7. Rotating on-call duty

8. Rotating on-call duty per team

9. ● Self service
● Scalable
● Open source with a wide community
Technology Enables Culture

10. Data
Logic
Alerting

11. Data

12. Where do we keep our monitoring data?
Metrics – Prometheus

13. Where do we keep our monitoring data?
Metrics – Prometheus
Logs - Elastic Search

14. Where do we keep our monitoring data?
Metrics – Prometheus
Logs - Elastic Search
Any other data source

15. Data
Logic

16. Data
Logic

19. ● Monitoring-as-code
● Scalable
● Modular
● We can monitor anything we can think of
Let’s take a look at the engine...

20. Data
Logic
Alerting

22. Self service
Scalable
Open source with a wide community
Did we accomplish what we set out to achieve?

23. 5133 commits in 913 days
Everybody contributes
(commit statistics for production Apr 7, 2016 - Oct 7, 2018)

24. 5133 commits in 913 days
5.6 commits per day
Everybody contributes
(commit statistics for production Apr 7, 2016 - Oct 7, 2018)

25. 5133 commits in 913 days
5.6 commits per day
Contributed by 73authors
Everybody contributes
(commit statistics for production Apr 7, 2016 - Oct 7, 2018)

28. Better visibility
What’s next?
Tighter coupling
with code

29. Back to where we started...
Does
monitoring != production?
Can alerts
suck less?
Who carries
the pager?
=

30. Questions?

31. @yaronidan @SolutoEng
Thank You!

32. Sources
Monitoring blog post at Soluto’s engineering blog –
https://blog.solutotlv.com/distributed-monitoring-for-devops-te
ams-using-icinga-and-puppet/
https://github.com/Soluto/nagios-plugins
https://github.com/Icinga/puppet-icinga2

33. Tips and Tricks
Don’t let alert fatigue get you! Adjust alerts to only fire when disaster strikes
Learn from outages - make sure alerts were firing, and if they don’t - make‘em!
Choose a solution that allows versioning, preferable using monitoring-as-code
Use templating for apps that share the same monitoring patterns
Share the joy of holding a pager with your fellow developers
Focus your monitoring efforts on metrics that can harm your business