Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Monitoring in the cloud with Puppet

My PuppetCamp Ghent 2013 talk on building monitoring solutions with Puppet

  • Login to see the comments

Monitoring in the cloud with Puppet

  1. Monitoring in the Cloud with Puppet PuppetCamp Ghent 2013 Kris Buytaert
  2. Kris Buytaert● I used to be a Dev,● Then Became an Op● Chief Trolling Officer and Open Source Consultant● Everything is an effing DNS Problem● Building Clouds since before the bookstore● Some books, some papers, some blogs● Evangelizing devops
  3. Do you know what your children do at 5 am in the morning ?● Sleeping ?● Crashing a Party ?● Why are the cops at your front door ?● What happened ?● How long have they been gone ?
  4. devops● Culture● (Lean)● Automation● Measurement● Sharing Damon Edwards and John Willis Gene Kim
  5. Monitoring is usually anaftertought ENOBUDGET, ENOTIME
  6. A sub movement● #monitoringsucks trending●● 2008 Study :Nagios + Friends● 2011 Conclusion : Nagios/Icinga are the only automatable alternatives
  7. #monitoringsucks Monitoring is AWESOME. Metrics are AWESOME. I love it. Heres what I dont love:● Having my hands tied with the model of host and service bindings.● Having to set up "fake" hosts just to group arbitrary metrics together● Having to either collect metrics twice - once for alerting and another for trending● Only being able to see my metrics in 5 minute intervals● Having to chose between shitty interface but great monitoring or shitty monitoring but great interface● Dealing with a monitoring system that thinks IT is the system of truth for my environment● Not actually having any real choices John Vincent (@lusis) on his blog monitoring-sucks.html
  8. Lets forget about● GroundWork ● Cacti● Zenoss ● Munin● Zabbix ● BigBrother● Hyperic
  9. Whats different in the cloud ?● Scale● Velocity● Change
  10. Challenges● Reproducability● Speed● Auditing● Keeping stuff in sync
  11. Thank god Luke for Puppet
  12. And Mark for Cfengine
  13. }#monitoringsucks 2013
  14. What we want● Small , wel suited components • Collect • Transport / Mangle • Analyse / Act • Visualize
  15. Collectd● Collects● Zillion Plugins • Nginx,apache, mysql, disk● Graphite Carbon Plugin● Send metrics to graphite● et-collectd
  16. ● Collect from anywhere● Filter● Send anywhere
  17. Puppet and Logstash●● Includes Logstash / Kibana / Grok / ES modules as submodules ! git clone● git submodule init● git submodule update● vagrant up
  18. Graphite● Graphing at Scale● Graphing at Ease● Any metric is a graph● echo "somestring $somevalue $timestamp" | nc <%= graphitehost %> 2003
  19. Graphite Composer x
  20. Graphite API
  21. Gdash In action
  22. Puppet and Graphite●● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules !● git clone● git submodule init● git submodule update● vagrant up
  23. Collectd & Graphite
  24. Configuring a Service● Package● Config● Service <- we pray status is correct
  25. Apache Example:
  26. Definition of Done: Done means : being monitored
  27. Icinga ?• Isnt nagios dead ?• Vibrant Community• Throw great parties in Nurnberg• Nobody can pronounce it anyhow•
  28. Monitoring Baseline● Deploy a host,● Add it to the monitoring● Add collection tools● Add check definitions● Update the monitoring tool config
  29. Stored Configs
  30. Collection and Export Export : Collect: @@resource { Resource <<| query |>> ... }Clean out nodes that dissapearpuppet node clean
  31. Exporting and Collecting
  32. Monitoring a Vhost
  33. It is not in production if it is not monitored
  34. Triggers on Graphs● Export Java Metrics ● Collect JMX Exports on JMXTransNode● JMXTrans ● Graph Em● Export JMXConfigs Collect Nagios Configs● Configure NRPE Check on Nagios Server● Export NagiosCheck
  35. Triggers on Graphs
  36. Triggers on Graphs
  37. Conclusion:● Reproducable monitoring setup● Dynamically generated monitoring config● Code is available at
  38. But,(pronounced in Ozzy dialect)
  40. #monitoringlove
  41. HomeworkSensu RiemannExtrememon Esper...
  42. ContactKris.Buytaert@inuits.euFurther Reading@krisbuytaert Inuits Duboistraat 50 2060 Antwerpen Belgium 891.514.231 +32 475 961221