In any data center, one will have a lot of machines, and even more applications, plenty of them legacy applications with little to no built-in monitoring capabilities. But even when monitoring is built in, quite often, it just provides basic building blocks. In this talk, it will be shown how to tie a syslog-ng based logging solution to the Riemann monitoring system, and use Kibana to make sense of both logging and monitoring data. The presentation will suggest solutions for extracting data from various applications, ways to transform them into useful metrics, and will - of course - also touch the subject of what exactly useful metrics are to begin with. A live demo of all things discussed will be shown at the end.

1. Monitoring with syslog-ng, Riemann and Kibana
@algernoone @balabit

2. syslog-ng

3. syslog-ng
● Open source event processor and swiss army knife

4. syslog-ng
● Open source event processor and swiss army knife
● Developed since 1998, LGPL + GPL

5. syslog-ng
● Open source event processor and swiss army knife
● Developed since 1998, LGPL + GPL
– (Commercial offering since 2007)

6. syslog-ng
● Open source event processor and swiss army knife
● Developed since 1998, LGPL + GPL
– (Commercial offering since 2007)
● Collects, parses, filters, transforms, transfers events

7. syslog-ng
● Open source event processor and swiss army knife
● Developed since 1998, LGPL + GPL
– (Commercial offering since 2007)
● Collects, parses, filters, transforms, transfers events
● Wide variety of plugins

8. syslog-ng
● Open source event processor and swiss army knife
● Developed since 1998, LGPL + GPL
– (Commercial offering since 2007)
● Collects, parses, filters, transforms, transfers events
● Wide variety of plugins
● A sizable, helpful and very inclusive community

9. Riemann

10. Riemann
● Riemann monitors distributed systems

11. Riemann
● Riemann monitors distributed systems
● Event aggregator with a powerful stream processing
language

12. Riemann
● Riemann monitors distributed systems
● Event aggregator with a powerful stream processing
language
● Provides a low-latency, transient shared state

13. Kibana

14. Kibana
● Visualize logs and time-stamped data

15. Kibana
● Visualize logs and time-stamped data
● Powerful search syntax

16. Kibana
● Visualize logs and time-stamped data
● Powerful search syntax
● Flexible, powerful, yet intuitive interface

17. Kibana

18. Monitoring

19. Monitoring
What

20. Monitoring
● System state
What

21. Monitoring
● System state
● Application state
What

22. Monitoring
● System state
● Application state
● Exceptions
What

23. Monitoring
● System state
● Application state
● Exceptions
● Activity
What

24. Monitoring
Tools

25. Monitoring
● Nagios
● Collectd
● Munin
Tools

26. Monitoring
● Nagios
● Collectd
● Munin
● Riemann
Tools

27. Logging

28. Logging

29. Logging
● Persisting application state

30. Logging
● Persisting application state
● Format is usually application specific

31. Logging
● Persisting application state
● Format is usually application specific
● Structured vs unstructured

32. Logging
● Persisting application state
● Format is usually application specific
● Structured vs unstructured
● Great source for monitoring too!

33. Monitoring + Logging

34. Monitoring + Logging
● What do we already have?

35. Monitoring + Logging
● What do we already have?
● What can we add?

36. Monitoring + Logging
● What do we already have?
● What can we add?
● How will we benefit from all of this?

37. Q & A

38. Thanks!
● http://www.syslog-ng.org/
● https://github.com/balabit/syslog-ng-incubator
● https://talien.blogs.balabit.com/
● https://algernon.blogs.balabit.com/