This document summarizes Soluto's transition to a new monitoring system. It discusses moving from a self-hosted Nagios installation to using Prometheus and ELK/Logz.io for time-series and log data. The new system provides self-service monitoring, scalability, and is open source. It monitors using a monitoring-as-code approach across technologies. Soluto also uses Mixpanel for business metrics. Next steps include more Prometheus adoption, distributing Icinga, and tighter coupling of monitoring with code.

1. Monitor all the thingz
February 2018
@yaronidan
@SolutoEng

3. Let’s think about it...
● Does monitoring = production?
● Can alerts suck less?
● Who carries the pager?

5. Our old monitoring system -
https://www.ks-soft.net/hostmon.eng/

6. And then we grew…
● Ops became a bottleneck
● No monitoring for new tech
● SLA is now a thing

8. We realized we need a better solution
 Self service
 Scalable
 Open source with a wide community

11. Where do we keep our monitoring data?
● TSDB – Graphite

12. Where do we keep our monitoring data?
● TSDB – Graphite

13. Where do we keep our monitoring data?
● TSDB – Graphite/Prometheus
● ELK - powered by logz.io

14. Where do we keep our monitoring data?
● TSDB – Graphite/Prometheus
● ELK - powered by logz.io
● Any other data source

17. Let’s take a look at the engine
● Monitoring-as-code
● Scalable
● Modularity
● We can monitor anything we can think of

20. Questions so far?

24. Who do we alert?

25. One person

26. Rotating on-call duty

27. Rotating on-call duty per team

28. Escalation

29. Did we fulfill what we set out to achieve?
 Self-service
 Scalable
 Open source with a wide community

30. Everybody contributes (Commit statistics for production Feb 20 - Jan 29)
2000commits in 343days

31. Everybody contributes (Commit statistics for production Feb 20 - Jan 29)
2000commits in 343days
5.8commits per day

32. Everybody contributes (Commit statistics for Feb 20 - Jan 29)
2000commits in 343days
5.8commits per day
Contributed by73authors

33. Monitoring business
● We’re also using mixpanel for monitoring business metrics
● Different than traditional monitoring
● Gives us a broad view of the products condition

37. Back to where we started...
● Does monitroing = production?
● Can alerts suck less?
● Who carries the pager?

38. What’s next?
● More prometheus
● Distribute Icinga
● Tighter coupling with code

39. Questions?

40. @yaronidan
@SolutoEng
#MeetupAtSoluto
Thank You!

41. @yaronidan
@SolutoEng
#MeetupAtSoluto
Obviously, we’re hiring.

42. Sources
Monitoring blog post at Soluto’s engineering blog
https://github.com/Soluto/nagios-plugins
https://github.com/Soluto/check-redis-plugin
https://github.com/Icinga/puppet-icinga2

43. Tips and Tricks
Don’t let alert fatigue get you! Adjust alerts to only fire when disaster strikes
Learn from outages - make sure alerts were firing, and if they don’t - make‘em!
Choose a solution that allows versioning, preferable using monitoring-as-code
Use templating for apps that share the same monitoring patterns
Share the joy of holding a pager with your fellow developers
Focus your monitoring efforts on metrics that can harm your business