Monitoring in the Cloud with
Puppet PuppetCamp Ghent 2013 Kris Buytaert

Kris Buytaert ● I used
to be a Dev, ● Then Became an Op ● Chief Trolling Officer and Open Source Consultant @inuits.eu ● Everything is an effing DNS Problem ● Building Clouds since before the bookstore ● Some books, some papers, some blogs ● Evangelizing devops

Do you know what your
children do at 5 am in the morning ? ● Sleeping ? ● Crashing a Party ? ● Why are the cops at your front door ? ● What happened ? ● How long have they been gone ?

devops ● Culture ● (Lean)
● Automation ● Measurement ● Sharing Damon Edwards and John Willis Gene Kim

Monitoring is usually an aftertought
ENOBUDGET, ENOTIME

A sub movement ● #monitoringsucks
trending ● https://github.com/monitoringsucks/ ● 2008 Study :Nagios + Friends ● 2011 Conclusion : Nagios/Icinga are the only automatable alternatives

#monitoringsucks Monitoring is AWESOME. Metrics
are AWESOME. I love it. Here's what I don't love: ● Having my hands tied with the model of host and service bindings. ● Having to set up "fake" hosts just to group arbitrary metrics together ● Having to either collect metrics twice - once for alerting and another for trending ● Only being able to see my metrics in 5 minute intervals ● Having to chose between shitty interface but great monitoring or shitty monitoring but great interface ● Dealing with a monitoring system that thinks IT is the system of truth for my environment ● Not actually having any real choices John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why- monitoring-sucks.html

Let's forget about ● GroundWork
● Cacti ● Zenoss ● Munin ● Zabbix ● BigBrother ● Hyperic

What's different in the cloud
? ● Scale ● Velocity ● Change

Challenges ● Reproducability ● Speed
● Auditing ● Keeping stuff in sync

Thank god Luke for Puppet

And Mark for Cfengine

}
#monitoringsucks 2013

What we want ● Small
, wel suited components • Collect • Transport / Mangle • Analyse / Act • Visualize

Collectd ● Collects ● Zillion
Plugins • Nginx,apache, mysql, disk ● Graphite Carbon Plugin ● Send metrics to graphite ● https://github.com/KrisBuytaert/pupp et-collectd

● Collect from anywhere ●
Filter ● Send anywhere

Puppet and Logstash ● https://github.com/KrisBuytaert/vagrant-puppet-logstash
● Includes Logstash / Kibana / Grok / ES modules as submodules ! git clone ● git submodule init ● git submodule update ● vagrant up

Graphite ● Graphing at Scale
● Graphing at Ease ● Any metric is a graph ● echo "somestring $somevalue $timestamp" | nc <%= graphitehost %> 2003

Graphite Composer
x

Graphite API

Gdash In action

Puppet and Graphite ● https://github.com/KrisBuytaert/vagrant-graphite/
● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules ! ● git clone ● git submodule init ● git submodule update ● vagrant up

Collectd & Graphite

Configuring a Service ● Package
● Config ● Service <- we pray status is correct

Apache Example:

Definition of Done: Done means
: being monitored

Icinga ? • Isn't nagios
dead ? • Vibrant Community • Throw great parties in Nurnberg • Nobody can pronounce it anyhow • https://github.com/Inuits/puppet-icinga/

Monitoring Baseline ● Deploy a
host, ● Add it to the monitoring ● Add collection tools ● Add check definitions ● Update the monitoring tool config

Stored Configs

Collection and Export Export :
Collect: @@resource { Resource <<| query |>> ... } Clean out nodes that dissapear puppet node clean

Exporting and Collecting

Monitoring a Vhost

It is not in production
if it is not monitored

Triggers on Graphs ● Export
Java Metrics ● Collect JMX Exports on JMXTransNode ● JMXTrans ● Graph Em ● Export JMXConfigs Collect Nagios Configs ● Configure NRPE Check on Nagios Server ● Export NagiosCheck

Triggers on Graphs

Triggers on Graphs

Conclusion: ● Reproducable monitoring setup
● Dynamically generated monitoring config ● Code is available at github.com/KrisBuytaert/snippets

But,
(pronounced in Ozzy dialect)

http://www.kartar.net/2013/01/monitoring-sucks---a-rant/

#monitoringlove

Homework Sensu Riemann Extrememon Esper
...

Contact Kris.Buytaert@inuits.eu Further Reading @krisbuytaert
http://www.krisbuytaert.be/blog/ http://www.inuits.eu/ Inuits Duboistraat 50 2060 Antwerpen Belgium 891.514.231 +32 475 961221

http://loadays.org