Monitoring in the cloud with Puppet


Published on

My PuppetCamp Ghent 2013 talk on building monitoring solutions with Puppet

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Monitoring in the cloud with Puppet

  1. Monitoring in the Cloud with Puppet PuppetCamp Ghent 2013 Kris Buytaert
  2. Kris Buytaert● I used to be a Dev,● Then Became an Op● Chief Trolling Officer and Open Source Consultant● Everything is an effing DNS Problem● Building Clouds since before the bookstore● Some books, some papers, some blogs● Evangelizing devops
  3. Do you know what your children do at 5 am in the morning ?● Sleeping ?● Crashing a Party ?● Why are the cops at your front door ?● What happened ?● How long have they been gone ?
  4. devops● Culture● (Lean)● Automation● Measurement● Sharing Damon Edwards and John Willis Gene Kim
  5. Monitoring is usually anaftertought ENOBUDGET, ENOTIME
  6. A sub movement● #monitoringsucks trending●● 2008 Study :Nagios + Friends● 2011 Conclusion : Nagios/Icinga are the only automatable alternatives
  7. #monitoringsucks Monitoring is AWESOME. Metrics are AWESOME. I love it. Heres what I dont love:● Having my hands tied with the model of host and service bindings.● Having to set up "fake" hosts just to group arbitrary metrics together● Having to either collect metrics twice - once for alerting and another for trending● Only being able to see my metrics in 5 minute intervals● Having to chose between shitty interface but great monitoring or shitty monitoring but great interface● Dealing with a monitoring system that thinks IT is the system of truth for my environment● Not actually having any real choices John Vincent (@lusis) on his blog monitoring-sucks.html
  8. Lets forget about● GroundWork ● Cacti● Zenoss ● Munin● Zabbix ● BigBrother● Hyperic
  9. Whats different in the cloud ?● Scale● Velocity● Change
  10. Challenges● Reproducability● Speed● Auditing● Keeping stuff in sync
  11. Thank god Luke for Puppet
  12. And Mark for Cfengine
  13. }#monitoringsucks 2013
  14. What we want● Small , wel suited components • Collect • Transport / Mangle • Analyse / Act • Visualize
  15. Collectd● Collects● Zillion Plugins • Nginx,apache, mysql, disk● Graphite Carbon Plugin● Send metrics to graphite● et-collectd
  16. ● Collect from anywhere● Filter● Send anywhere
  17. Puppet and Logstash●● Includes Logstash / Kibana / Grok / ES modules as submodules ! git clone● git submodule init● git submodule update● vagrant up
  18. Graphite● Graphing at Scale● Graphing at Ease● Any metric is a graph● echo "somestring $somevalue $timestamp" | nc <%= graphitehost %> 2003
  19. Graphite Composer x
  20. Graphite API
  21. Gdash In action
  22. Puppet and Graphite●● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules !● git clone● git submodule init● git submodule update● vagrant up
  23. Collectd & Graphite
  24. Configuring a Service● Package● Config● Service <- we pray status is correct
  25. Apache Example:
  26. Definition of Done: Done means : being monitored
  27. Icinga ?• Isnt nagios dead ?• Vibrant Community• Throw great parties in Nurnberg• Nobody can pronounce it anyhow•
  28. Monitoring Baseline● Deploy a host,● Add it to the monitoring● Add collection tools● Add check definitions● Update the monitoring tool config
  29. Stored Configs
  30. Collection and Export Export : Collect: @@resource { Resource <<| query |>> ... }Clean out nodes that dissapearpuppet node clean
  31. Exporting and Collecting
  32. Monitoring a Vhost
  33. It is not in production if it is not monitored
  34. Triggers on Graphs● Export Java Metrics ● Collect JMX Exports on JMXTransNode● JMXTrans ● Graph Em● Export JMXConfigs Collect Nagios Configs● Configure NRPE Check on Nagios Server● Export NagiosCheck
  35. Triggers on Graphs
  36. Triggers on Graphs
  37. Conclusion:● Reproducable monitoring setup● Dynamically generated monitoring config● Code is available at
  38. But,(pronounced in Ozzy dialect)
  40. #monitoringlove
  41. HomeworkSensu RiemannExtrememon Esper...
  42. ContactKris.Buytaert@inuits.euFurther Reading@krisbuytaert Inuits Duboistraat 50 2060 Antwerpen Belgium 891.514.231 +32 475 961221