Im Juni 2011 wetterte John Vincent auf Twitter über Monitoring mit dem Hashtag #monitoringsucks und für viele von uns, traf er damit genau ins Schwarze. Bei den #devopsdays im November 2012 in Rom verkündete Ulf Mansson seine neu gefundene Liebe zu Monitoring und der Hashtag #monitoringlove wurde ins Leben gerufen. Aufgrund einer neuen Ära von Open Source Monitoring Tools, entwickelt er wieder eine Leidenschaft für Monitoring und viele von uns taten es ihm gleich. Während der letzten 5 Jahre veröffentlichte die Community eine enorme Anzahl an neuen Tools und Mustern, die manchmal mit #devops getaggt wurden und fast alle Open Source waren. Haben Sie noch einen Überblick darüber, welches Tool man für was verwendet und was die Unterschiede sind? Dieser Vortrag wird Ihnen einen Überblick über die Open-Source- Monitoring-Landschaft verschaffen und aufzeigen, welche Tools man am besten nutzt oder aber Ihnen die Entscheidung sogar noch schwerer machen:)

1. From #MonitoringSucks to
#MonitoringLove
(and back)
@KrisBuytaert
OSMC 2014 , Nuremberg, Germany

2. Kris Buytaert
●I used to be a Dev,
●Then Became an Op
●Chief Trolling Officer and Open Source Consultant @inuits.eu
●Everything is an effing DNS Problem
●Building Clouds since before the bookstore
●Organising Conferences
●Evangelizing devops

3. An opinionated talk about the Open Source Monitoring tooling landscape
In which I hope to learn from YOU

4. #devops=~C(L)AMS
● Culture
● (Lean)
● Automation
● Monitoring and Measurement
● Sharing
● Damon Edwards and John Willis
Gene Kim

5. Monitoring is usually an aftertought
ENOBUDGET, ENOTIME

6. An 2008 OLS Paper
● We have bloated Java tools
● Some open Core stuff
● DYI folks want traditional Nagios
● DBA Required

7. #monitoringsucks
● John Vincent (@lusis), june 2011
● A sub #devops movement
● https://github.com/monitoringsucks/

8. Why #monitoringsucks
● Manual config (gui)
● Not in sync with reality
● Hosts only
● Services sometimes
● Aplication never
● Chaos or out of sync with reality
● Alert Fatigue

9. Let's forget about
● Tools with no (stable) API
● Tools with strong focus on GUI
● Unless you are an SME with < 100 nodes
● Zenoss, Hyperic, GroundWork, ....
● P.S. : don't even mention proprietary software to me

10. What we want
● Small , well suited components
• Collect
• Transport / Mangle
• Store
• Analyse
• Act / Alert
• Visualize

11. #monitoringlove
•Ulf Mansson #devopsdays Rome 2011
•A new era of tooling
•#monitoringlove hacksessions @inuits
•#monitorama

15. Icinga
•2009 Fork
•I consider Nagios dead
•Vibrant Community (or they stalk me)
•Throw great parties in Nurnberg
•Nobody can pronounce it anyhow
•https://github.com/Inuits/puppet-icinga/

16. Stored Configs

17. #monitoringlove
But the love was about :

18. Sensu
● Awesome for non static environments
● Scaling a clustered RabbitMQ ?
● This is Europe, U no do cloud

19. Automation of
#monitoring brought back
the #love

20. ●Autodetection
●Multiplexing
●Trend Forecasting
I love CheckMK

21. •Autodetection ?
•Service,
•Business Functionalities
•eg. vhosts etc
•Single Source of Truth
I hate CheckMK

22. Monitoring a service
vs
Monitoring a Service

23. definition of done:
monitored and in production

24. A software project is not done untill your last end user
is dead

25. Culture,
Automation,
Measurement :
measure all the things
Sharing

26. Deploy Statistics
● Time To Deploy
● Deploy Frequency
● Lifecycle frequency
● Map to other metrics

27. CollectD all the metrics,
at high intervals

28. Oldschool graphite

29. Self Service
Gdash based pipelines
Puppetized Templates (wip)

30. Gdash

31. Grafana

32. Graphite++
● Dashboards
• Grafana
● Engines :
• InfluxDB
• Cyanite

33. Triggers on Graphs
● Export Java Metrics
● JMXTrans
● Export JMXConfigs
● Configure NRPE Check
● Export NagiosCheck
● Collect JMX Exports on
JMXTransNode
● Graph Em
● Collect Icinga Configs on Icinga

34. Aggregation
● Alert on streams
● Alert on aggregated metrics

37. Riemann
● I still don't get it ?
● Distributed Top
● Do you like Clojure ?
● Riemann Health plugin ?
● s/riemann-health/collectd/g;
● Output to graphite

38. Graphs to Knowledge
Skyline
•Oculus
•Creating Information out of this data
•Big data
•Machine Learning

39. But I have log files..

43. Logs and Metrics
● Graylog2
● ELSA (Enterprise Log Search and Archive)
● ELK Stack

44. ● Collect from anywhere
● Filter
● Send anywhere

46. Black on White ?

47. APM
But what about my apps ?
Half the world cheers about SAAS tools :(

48. Packetbeat
● Traffic Flow through network
● Transactions causing errros
● SQL per HTTP
● API call usage

49. PacketBeat

50. This new “D” hype

51. Containers are the new black
● 1 process per container
● Metric collection ?
● Service health ?

52. So you want service registration of your healthy (containerized) applications ?

53. Enter Consul.io
● Service discovery
● Failure detection
● Using Gossip build on top of Serf
● Random node 2 node communication
● A HashiCorp project

54. Consul
● Uses monitoring_plugins for health
● Creates unhealthy dns setups
● Sensu alike
● Key-Value store
● Consul_template => fills your templates

55. Everything is a
freaking dns problem

56. Self Healing
● Pacemaker Corosync (ocf resource that monitors your service)
● Mesos
● Kubernetes
● Scale changes, Consensus Models change

57. So your DC fails
Whom to alert when ?

59. 'New' kids on the block
● Flapjack
● flapjack.io
● monitoring notification routing + event processing system
● OpenDuty
● github.com/szechuen/OpenDuty
● Duty management

60. My Alerting Strategy
Is still in beta

61. And back :(
In 2014 I`m still running the same check for
- service registration (consul)
- high availability (pacemaker/corosync)
- monitoring (icinga)

62. But I love where Monitoring is heading
We have much less false positives
And we have a Maintainable Monitoring Infra
Kinda

63. Your next trip to Gent !
CfgMgmtcamp.eu
February 2 and 3, 2015
CFP is Open !

64. Contact
Kris.Buytaert@inuits.eu
Further Reading
@krisbuytaert
http://www.krisbuytaert.be/blog/
http://www.inuits.eu/
Inuits
Duboistraat 50
2060 Antwerpen
Belgium
891.514.231
+32 475 961221