A talk about various Open Source tools for availability and functional monitoring, time series and metrics as well log and event management.

1. 11th May 2017
State of the Open Source
Monitoring landscape

2. Introduction

3. • Open Source since “2007”
• CEO NETWAYS and Icinga
• Core-Organizer DevOpsDays
• Contact me via @gethash
Introduction Bernd Erk

4. • Founded 1995
• German based service company
• Open Source since 1997
• Open Source Datacenter Solutions
Introduction NETWAYS

5. 13th of May – Icinga Camp Bangalore
www.icinga.com

6. Monitoring

9. Availability | Function

10. Availability | Function
Metrics &
Time Series

11. Availability | Function
Metrics &
Time Series
Logs & Events

12. User experience
Availability | Function
Metrics &
Time Series
Logs & Events

13. What to monitor?

15. Focus on your business

16. Top down approach

17. Business logic

18. Business logic
Applications

19. Business logic
Applications
Services

20. Business logic
Applications
Services
Infrastructure

21. How to monitor?

23. Push and Pull

24. No Auto-Discovery

25. Infrastructure as Code

26. Provide monitoring as a service

27. Let‘s talk about
the tools

29. Availability and
Functional Monitoring

30. kartar.net/2015/08/monitoring-survey-2015---tools/
0
50
100
150
200
250
300
350
400
450
Monitoring Survey - James Turnbull

31. • Nagios
• Icinga
• Sensu
• Zabbix
• Riemann
• OpenNMS
Availability and functional monitoring tools

32. nagios.org

33. • Rewritten from scratch
• Built in integrations
• Command-line interface
• Application based cluster stack
• Setup could be complex
• REST API
Icinga 2
icinga.com

34. • Similar scope to Nagios and Icinga
• Standalone and Subscription Checks
• Several external dependencies
• Lack of historical data
• Cool stuff is enterprise only
Sensu
sensuapp.org

35. • Full featured open source solution
• Agend based approach
• Integrated Logging and Graphing
• Hard to orchestrate and automate
• Scale out limitations
Zabbix
zabbix.com

36. • Streaming processor
• Real time monitoring
• Based on Clojure
• Riemann tools provide the data
• Stateless
• There is not so much going on
Riemann
riemann.io

37. • Full featured open source solution
• Based on Java
• Great of homogenous environments
• Native SNMP support
• Built in Auto-Discovery
OpenNMS
opennms.org

38. Metrics & Time Series

40. • Storage engine similar to RRDtool
• Started the metrics revolution
• Advantage
• Flexible architecture
• Disadvantage
• Flexible architecture
Graphite
graphiteapp.org

41. • Based on Hadoop and Hbase
• Overall complexity is huge
• You can keep the raw data forever
• Easy to scale (if you can make it)
OpenTSDB
opentsdb.org

42. • Highly dimensional data model
• Powerful query language
• Designed for web services
• OS Metrics require node exporter
• Rule based alerting
Prometheus
prometheus.io

43. • Similar scope to Graphite
• Easier to install
• SQL-like query language
• Scale out requires enterprise
• Full featured solution – TICK Stack
InfluxDB (InfluxData)
influxdata.com

44. • Very powerful analytics
• Timelion as Kibana extension
• Beats provide metrics
• Statsd could be used
• Different model approach
Elastic (Stack)
elastic.co

45. Visualization

47. Grafana
grafana.com

48. Grafana - Annotations

49. Logs & Events

50. Flow of unstructured data with timestamp and message
May 4 16:57:24 web sshd[25828]: Received disconnect from 10.10.0.31: 11: disconnected by user
Logs

51. Flow of structured data with identifiable attributes
Event {
Time: May 4 16:57:24
Process: sshd
State: Received disconnect from 10.10.0.31
Client: 10.10.0.31
}
Events

52. Log > Event > Analyze > Action

53. • Kind of the standard in logging
• Powered by Apache Lucene
• Highly integrated solution
• Extensible using X-Pack
• Logstash API is very powerful
• By far the largest community
Elastic Stack
elastic.co

54. Graylog
• Based on Elasticsearch
• Powerful graphical configuration
• Additional MongoDB to maintain
• Build in authentication an
authorization
• Could be used with Logstash
graylog.org

55. • Unified log layer
• Based on other storage engines
• An alternative to Logstash
• Built-in Reliability
• Alternative for custom architecture
Fluentd
fluentd.org

56. User experience

58. • Both standards in their field
…. a long time ago
Webinject and AutoIT
webinject.org and autoitscript.com

59. • Combination of Sahi und Sikulix
• Mainly for Nagios compatible systems
• Prebuild Docker containers available
• Cool features in Shahi are enterprise
only
Sakuli
github.com/ConSol/sakuli

60. • End-to End Monitoring
• Based on a Python module
• Text and object recognition
• IDE to create test cases
• Audit trail and notification system
Alyvix
alyvix.com

61. Conclusion

63. I am sorry, but there is no best
tool for everyone 

64. • Monolithic
• If all-in-one gets the job
done, then perfect
• Good for smaller scale and
non-tech-focused companies
• Modular
• Agile and DevOps requires
flexibility and innovation
• Good for tech driven and
ops-focused companies
Monolithic vs. Modular
Michael Merideth - @victorops

65. I prefer the modular approach
based on a tool chain

66. Setup a real life use case
Play (with integrations)
Choose your favorite
Flip a coin

67. netways
netways
netways
www.netways.de
blog.netways.de
git.netways.de
Thank You