Monitoring in the cloud with Puppet

Monitoring in the Cloud with Puppet PuppetCamp Ghent 2013 Kris Buytaert

Kris Buytaert● I used to be a Dev,● Then Became an Op● Chief Trolling Officer and Open Source Consultant @inuits.eu● Everything is an effing DNS Problem● Building Clouds since before the bookstore● Some books, some papers, some blogs● Evangelizing devops

Do you know what your children do at 5 am in the morning ?● Sleeping ?● Crashing a Party ?● Why are the cops at your front door ?● What happened ?● How long have they been gone ?

devops● Culture● (Lean)● Automation● Measurement● Sharing Damon Edwards and John Willis Gene Kim

Monitoring is usually anaftertought ENOBUDGET, ENOTIME

A sub movement● #monitoringsucks trending● https://github.com/monitoringsucks/● 2008 Study :Nagios + Friends● 2011 Conclusion : Nagios/Icinga are the only automatable alternatives

#monitoringsucks Monitoring is AWESOME. Metrics are AWESOME. I love it. Heres what I dont love:● Having my hands tied with the model of host and service bindings.● Having to set up "fake" hosts just to group arbitrary metrics together● Having to either collect metrics twice - once for alerting and another for trending● Only being able to see my metrics in 5 minute intervals● Having to chose between shitty interface but great monitoring or shitty monitoring but great interface● Dealing with a monitoring system that thinks IT is the system of truth for my environment● Not actually having any real choices John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why- monitoring-sucks.html

Lets forget about● GroundWork ● Cacti● Zenoss ● Munin● Zabbix ● BigBrother● Hyperic

Whats different in the cloud ?● Scale● Velocity● Change

Challenges● Reproducability● Speed● Auditing● Keeping stuff in sync

Thank god Luke for Puppet

And Mark for Cfengine

}#monitoringsucks 2013

What we want● Small , wel suited components • Collect • Transport / Mangle • Analyse / Act • Visualize

Collectd● Collects● Zillion Plugins • Nginx,apache, mysql, disk● Graphite Carbon Plugin● Send metrics to graphite● https://github.com/KrisBuytaert/pupp et-collectd

● Collect from anywhere● Filter● Send anywhere

Puppet and Logstash● https://github.com/KrisBuytaert/vagrant-puppet-logstash● Includes Logstash / Kibana / Grok / ES modules as submodules ! git clone● git submodule init● git submodule update● vagrant up

Graphite● Graphing at Scale● Graphing at Ease● Any metric is a graph● echo "somestring $somevalue $timestamp" | nc <%= graphitehost %> 2003

Graphite Composer x

Graphite API

Gdash In action

Puppet and Graphite● https://github.com/KrisBuytaert/vagrant-graphite/● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules !● git clone● git submodule init● git submodule update● vagrant up

Collectd & Graphite

Configuring a Service● Package● Config● Service <- we pray status is correct

Apache Example:

Definition of Done: Done means : being monitored

Icinga ?• Isnt nagios dead ?• Vibrant Community• Throw great parties in Nurnberg• Nobody can pronounce it anyhow• https://github.com/Inuits/puppet-icinga/

Monitoring Baseline● Deploy a host,● Add it to the monitoring● Add collection tools● Add check definitions● Update the monitoring tool config

Stored Configs

Collection and Export Export : Collect: @@resource { Resource <<| query |>> ... }Clean out nodes that dissapearpuppet node clean

Exporting and Collecting

Monitoring a Vhost

It is not in production if it is not monitored

Triggers on Graphs● Export Java Metrics ● Collect JMX Exports on JMXTransNode● JMXTrans ● Graph Em● Export JMXConfigs Collect Nagios Configs● Configure NRPE Check on Nagios Server● Export NagiosCheck

Triggers on Graphs

Conclusion:● Reproducable monitoring setup● Dynamically generated monitoring config● Code is available at github.com/KrisBuytaert/snippets

But,(pronounced in Ozzy dialect)

http://www.kartar.net/2013/01/monitoring-sucks---a-rant/

#monitoringlove

HomeworkSensu RiemannExtrememon Esper...

ContactKris.Buytaert@inuits.euFurther Reading@krisbuytaerthttp://www.krisbuytaert.be/blog/http://www.inuits.eu/ Inuits Duboistraat 50 2060 Antwerpen Belgium 891.514.231 +32 475 961221

http://loadays.org