This document discusses using Puppet to configure and manage cloud monitoring infrastructure at scale. Puppet allows defining reusable modules and parameterized classes to configure tools like Graphite, Collectd, Nagios, and JMXTrans in a repeatable way across environments. Metrics can be collected and exported to Graphite for long-term storage and visualization of trends. Triggers can be created on graphs to detect anomalies and configure monitoring checks. The full Puppet code used is available online for reference.

1. Beyond VM deployment
Monitoring your VM's at scale
Kris Buytaert

2. Kris Buytaert
● I used to be a Dev,
● Then Became an Op
● Chief Trolling Officer and Open Source
Consultant @inuits.eu
● Everything is an effing DNS Problem
● Building Clouds since before the bookstore
● Some books, some papers, some blogs
● Evangelizing devops
● But mostly, trying to be good at my job

3. What's different in the cloud ?
● Scale
● Velocity
● Change

4. Challenges
● Reproducability
● Speed
● Auditing
● Keeping stuff in sync
• Monitoring
• Security

5. Case :
Using a configuration management
tool to configure, update and keep
your cloudscale monitoring and metric
infrastructure sane and manageable.

6. Tools
● Puppet ● Chef Cfengine
● Ganglia
● Collectd
● Sensu
● Graphite
● Nagios / Icinga

7. Not quite a Muppet.
● Puppet is...
● OSS
● A DSL language
● Written in Ruby
● Client/server oriented
● Contains abstraction layers
● Repeatable processes

8. Master of Puppets
● Puppet master
• CA authority
• Hosts Modules
• Hosts Node descriptions
• Compare, compile, apply
● Master is not a requirement !

9. Puppet Clients
● daemon
● Cron jobs
● External orchestration:
• for i in $hosts; do ssh $i “puppetd --test”; done
• mCollective, Func, …
● Get catalogs, play them,
● reporting

10. Puppet Environments
● Different code bases on 1 master
● Dev, Uat, Prod
● Only break one environment at once :)
● What about testing your Puppetmaster ?

11. Node definitions
● Nodes.pp
class defaults {
$search = "inuits.be"
$nameservers = ['208.67.220.220', '208.67.222.222']
include dns::resolv
include ssh::keys
include ssh::server
}
node "ns1.dev.inuits.be" {
include defaults
include dns::powerdns::server
include dns::powerdns::resolver
}
node “web1.dev.inuits.be” {
include defaults
include apache2
include mysql
}

12. External Node Classifier
● Fixed hostname ?
● How many nodes
● Naming schemas solve some issues
● External script that sends back yaml class descriptions
• Custom writtten
• Foreman
• ...

13. Classes vs Modules
● Module :
●
Abstract definition on configuring a service
●
Reusable
● Class :
●
Specific implementation of your use case of
such a module
•e.g usernames / passwords / hosts do not belong in
modules

14. Modules
● Files
● Templates
● Manifests
• DSL
• Classes
• Elements

15. Parametrized
Classes

16. Stored Configs

17. Use Cases:
● Ssh keys
● Reverse proxy configs
● Monitoring resources
● Measuring resources

18. Collection and Export
Export : Collect:
@@resource { Resource <<|
query |>>
... }
Clean out nodes that dissapear
puppet node clean

19. Defining a Service
● Local class that :
• Configures service using a standard
module call with hiera based parameters
• Configures Backup
• Configures logrotation
• Configures logshipping
• Exports Monitoring Needs
● Abuse modules for git ease

20. Apache Example:

21. #monitoringsucks
Monitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love:
● Having my hands tied with the model of host and service bindings.
● Having to set up "fake" hosts just to group arbitrary metrics together
● Having to either collect metrics twice - once for alerting and another for trending
● Only being able to see my metrics in 5 minute intervals
● Having to chose between shitty interface but great monitoring or shitty monitoring but
great interface
● Dealing with a monitoring system that thinks IT is the system of truth for my
environment
● Not actually having any real choices
John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-

23. #monitoringlove
● Puppet
● Nagios (Icinga)
● Graphite
● Collectd
● Logstash

24. Graphite
● Graphing at Scale
● Graphing at Ease
● Any metric is a graph
● echo "somestring $somevalue
$timestamp" | nc <%= graphitehost
%> 2003

25. Graphite Composer
x

26. Graphite API

27. Gdash In action

28. Puppet and Graphite
● https://github.com/KrisBuytaert/vagrant-graphite/
● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd /
Statsd / Tattle and more modules as submodules !
● git clone
● git submodule init
● git submodule update
● vagrant up

29. Collectd
● Collects
● Zillion Plugins
• Nginx,apache, mysql, disk
● Graphite Carbon Plugin
● Send metrics to graphite

30. Collectd & Graphite

31. Exporting and Collecting

32. Triggers on Graphs
● Export Java Metrics ● Collect JMX Exports
on JMXTransNode
● JMXTrans
● Graph Em
● Export JMXConfigs
Collect Nagios
● Configure NRPE Configs on Nagios
Check Server
● Export NagiosCheck

33. Triggers on Graphs

34. Triggers on Graphs

35. Conclusion:
● Reproducable monitoring setup
● Dynamically generated monitoring
config
● Code is available at
github.com/KrisBuytaert/snippets

36. Contact
Kris Buytaert
Kris.Buytaert@inuits.be
Further Reading
@krisbuytaert
http://www.krisbuytaert.be/blog/
http://www.inuits.be/
Inuits
Duboistraat 50
2060 Antwerpen
Belgium
891.514.231
+32 475 961221