This document discusses using Puppet to configure and manage cloud monitoring infrastructure at scale. Puppet allows defining reusable modules and parameterized classes to configure tools like Graphite, Collectd, Nagios, and JMXTrans in a repeatable way across environments. Metrics can be collected and exported to Graphite for long-term storage and visualization of trends. Triggers can be created on graphs to detect anomalies and configure monitoring checks. The full Puppet code used is available online for reference.
2. Kris Buytaert
● I used to be a Dev,
● Then Became an Op
● Chief Trolling Officer and Open Source
Consultant @inuits.eu
● Everything is an effing DNS Problem
● Building Clouds since before the bookstore
● Some books, some papers, some blogs
● Evangelizing devops
● But mostly, trying to be good at my job
5. Case :
Using a configuration management
tool to configure, update and keep
your cloudscale monitoring and metric
infrastructure sane and manageable.
6. Tools
● Puppet ● Chef Cfengine
● Ganglia
● Collectd
● Sensu
● Graphite
● Nagios / Icinga
7. Not quite a Muppet.
● Puppet is...
● OSS
● A DSL language
● Written in Ruby
● Client/server oriented
● Contains abstraction layers
● Repeatable processes
8. Master of Puppets
● Puppet master
• CA authority
• Hosts Modules
• Hosts Node descriptions
• Compare, compile, apply
● Master is not a requirement !
9. Puppet Clients
● daemon
● Cron jobs
● External orchestration:
• for i in $hosts; do ssh $i “puppetd --test”; done
• mCollective, Func, …
● Get catalogs, play them,
● reporting
10. Puppet Environments
● Different code bases on 1 master
● Dev, Uat, Prod
● Only break one environment at once :)
● What about testing your Puppetmaster ?
11. Node definitions
● Nodes.pp
class defaults {
$search = "inuits.be"
$nameservers = ['208.67.220.220', '208.67.222.222']
include dns::resolv
include ssh::keys
include ssh::server
}
node "ns1.dev.inuits.be" {
include defaults
include dns::powerdns::server
include dns::powerdns::resolver
}
node “web1.dev.inuits.be” {
include defaults
include apache2
include mysql
}
12. External Node Classifier
● Fixed hostname ?
● How many nodes
● Naming schemas solve some issues
● External script that sends back yaml class descriptions
• Custom writtten
• Foreman
• ...
13. Classes vs Modules
● Module :
●
Abstract definition on configuring a service
●
Reusable
● Class :
●
Specific implementation of your use case of
such a module
•e.g usernames / passwords / hosts do not belong in
modules
18. Collection and Export
Export : Collect:
@@resource { Resource <<|
query |>>
... }
Clean out nodes that dissapear
puppet node clean
19. Defining a Service
● Local class that :
• Configures service using a standard
module call with hiera based parameters
• Configures Backup
• Configures logrotation
• Configures logshipping
• Exports Monitoring Needs
● Abuse modules for git ease
21. #monitoringsucks
Monitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love:
● Having my hands tied with the model of host and service bindings.
● Having to set up "fake" hosts just to group arbitrary metrics together
● Having to either collect metrics twice - once for alerting and another for trending
● Only being able to see my metrics in 5 minute intervals
● Having to chose between shitty interface but great monitoring or shitty monitoring but
great interface
● Dealing with a monitoring system that thinks IT is the system of truth for my
environment
● Not actually having any real choices
John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-