My PuppetCamp Ghent 2013 talk on building monitoring solutions with Puppet

1. Monitoring in the Cloud
with Puppet
PuppetCamp Ghent 2013
Kris Buytaert

2. Kris Buytaert
● I used to be a Dev,
● Then Became an Op
● Chief Trolling Officer and Open Source
Consultant @inuits.eu
● Everything is an effing DNS Problem
● Building Clouds since before the bookstore
● Some books, some papers, some blogs
● Evangelizing devops

3. Do you know what your children do at 5 am
in the morning ?
● Sleeping ?
● Crashing a Party ?
● Why are the cops at your front door ?
● What happened ?
● How long have they been gone ?

4. devops
● Culture
● (Lean)
● Automation
● Measurement
● Sharing
Damon Edwards and John Willis
Gene Kim

5. Monitoring is usually an
aftertought
ENOBUDGET, ENOTIME

6. A sub movement
● #monitoringsucks trending
● https://github.com/monitoringsucks/
● 2008 Study :Nagios + Friends
● 2011 Conclusion : Nagios/Icinga are
the only automatable alternatives

7. #monitoringsucks
Monitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love:
● Having my hands tied with the model of host and service bindings.
● Having to set up "fake" hosts just to group arbitrary metrics together
● Having to either collect metrics twice - once for alerting and another for trending
● Only being able to see my metrics in 5 minute intervals
● Having to chose between shitty interface but great monitoring or shitty monitoring
but great interface
● Dealing with a monitoring system that thinks IT is the system of truth for my
environment
● Not actually having any real choices
John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-
monitoring-sucks.html

9. Let's forget about
● GroundWork ● Cacti
● Zenoss ● Munin
● Zabbix ● BigBrother
● Hyperic

10. What's different in the cloud ?
● Scale
● Velocity
● Change

14. Challenges
● Reproducability
● Speed
● Auditing
● Keeping stuff in sync

15. Thank god Luke for Puppet

16. And Mark for Cfengine

17. }
#monitoringsucks 2013

18. What we want
● Small , wel suited components
• Collect
• Transport / Mangle
• Analyse / Act
• Visualize

20. Collectd
● Collects
● Zillion Plugins
• Nginx,apache, mysql, disk
● Graphite Carbon Plugin
● Send metrics to graphite
● https://github.com/KrisBuytaert/pupp
et-collectd

21. ● Collect from
anywhere
● Filter
● Send anywhere

22. Puppet and Logstash
● https://github.com/KrisBuytaert/vagrant-puppet-logstash
● Includes Logstash / Kibana / Grok / ES modules as
submodules !
git clone
● git submodule init
● git submodule update
● vagrant up

23. Graphite
● Graphing at Scale
● Graphing at Ease
● Any metric is a graph
● echo "somestring $somevalue $timestamp" |
nc <%= graphitehost %> 2003

24. Graphite Composer
x

25. Graphite API

26. Gdash In action

27. Puppet and Graphite
● https://github.com/KrisBuytaert/vagrant-graphite/
● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd /
Statsd / Tattle and more modules as submodules !
● git clone
● git submodule init
● git submodule update
● vagrant up

28. Collectd & Graphite

29. Configuring a Service
● Package
● Config
● Service
<- we pray status is correct

30. Apache Example:

31. Definition of Done:
Done means : being monitored

32. Icinga ?
• Isn't nagios dead ?
• Vibrant Community
• Throw great parties in Nurnberg
• Nobody can pronounce it anyhow
• https://github.com/Inuits/puppet-icinga/

33. Monitoring Baseline
● Deploy a host,
● Add it to the monitoring
● Add collection tools
● Add check definitions
● Update the monitoring tool config

34. Stored Configs

35. Collection and Export
Export : Collect:
@@resource { Resource <<|
query |>>
... }
Clean out nodes that dissapear
puppet node clean

36. Exporting and Collecting

37. Monitoring a Vhost

38. It is not in production if it is not
monitored

39. Triggers on Graphs
● Export Java Metrics ● Collect JMX Exports on
JMXTransNode
● JMXTrans
● Graph Em
● Export JMXConfigs
Collect Nagios Configs
● Configure NRPE Check on Nagios Server
● Export NagiosCheck

40. Triggers on Graphs

41. Triggers on Graphs

42. Conclusion:
● Reproducable monitoring setup
● Dynamically generated monitoring
config
● Code is available at
github.com/KrisBuytaert/snippets

43. But,
(pronounced in Ozzy dialect)

44. http://www.kartar.net/2013/01/monitoring-sucks---a-rant/

45. #monitoringlove

46. Homework
Sensu Riemann
Extrememon Esper
...

47. Contact
Kris.Buytaert@inuits.eu
Further Reading
@krisbuytaert
http://www.krisbuytaert.be/blog/
http://www.inuits.eu/
Inuits
Duboistraat 50
2060 Antwerpen
Belgium
891.514.231
+32 475 961221

48. http://loadays.org