1. Beyond VM deployment
Monitoring your VM's at scale
Kris Buytaert

2. Kris Buytaert
● I used to be a Dev,
● Then Became an Op
● Chief Trolling Officer and Open Source
Consultant @inuits.eu
● Everything is an effing DNS Problem
● Building Clouds since before the bookstore
● Some books, some papers, some blogs
● Evangelizing devops
● But mostly, trying to be good at my job

3. What's different in the cloud ?
● Scale
● Velocity
● Change

4. Challenges
● Reproducability
● Speed
● Auditing
● Keeping stuff in sync
• Monitoring
• Security

5. Case :
Using a configuration management
tool to configure, update and keep
your cloudscale monitoring and metric
infrastructure sane and manageable.

6. Tools
● Puppet ● Chef Cfengine
● Ganglia
● Collectd
● Sensu
● Graphite
● Nagios / Icinga

7. Not quite a Muppet.
● Puppet is...
● OSS
● A DSL language
● Written in Ruby
● Client/server oriented
● Contains abstraction layers
● Repeatable processes

8. Master of Puppets
● Puppet master
• CA authority
• Hosts Modules
• Hosts Node descriptions
• Compare, compile, apply
● Master is not a requirement !

9. Puppet Clients
● daemon
● Cron jobs
● External orchestration:
• for i in $hosts; do ssh $i “puppetd --test”; done
• mCollective, Func, …
● Get catalogs, play them,
● reporting

10. Puppet Environments
● Different code bases on 1 master
● Dev, Uat, Prod
● Only break one environment at once :)
● What about testing your Puppetmaster ?

11. Node definitions
● Nodes.pp
class defaults {
$search = "inuits.be"
$nameservers = ['208.67.220.220', '208.67.222.222']
include dns::resolv
include ssh::keys
include ssh::server
}
node "ns1.dev.inuits.be" {
include defaults
include dns::powerdns::server
include dns::powerdns::resolver
}
node “web1.dev.inuits.be” {
include defaults
include apache2
include mysql
}

12. External Node Classifier
● Fixed hostname ?
● How many nodes
● Naming schemas solve some issues
● External script that sends back yaml class descriptions
• Custom writtten
• Foreman
• ...

13. Classes vs Modules
● Module :
●
Abstract definition on configuring a service
●
Reusable
● Class :
●
Specific implementation of your use case of
such a module
•e.g usernames / passwords / hosts do not belong in
modules

14. Modules
● Files
● Templates
● Manifests
• DSL
• Classes
• Elements

15. Parametrized
Classes

16. Stored Configs

17. Use Cases:
● Ssh keys
● Reverse proxy configs
● Monitoring resources
● Measuring resources

18. Collection and Export
Export : Collect:
@@resource { Resource <<|
query |>>
... }
Clean out nodes that dissapear
puppet node clean

19. Defining a Service
● Local class that :
• Configures service using a standard
module call with hiera based parameters
• Configures Backup
• Configures logrotation
• Configures logshipping
• Exports Monitoring Needs
● Abuse modules for git ease

20. Apache Example:

21. #monitoringsucks
Monitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love:
● Having my hands tied with the model of host and service bindings.
● Having to set up "fake" hosts just to group arbitrary metrics together
● Having to either collect metrics twice - once for alerting and another for trending
● Only being able to see my metrics in 5 minute intervals
● Having to chose between shitty interface but great monitoring or shitty monitoring but
great interface
● Dealing with a monitoring system that thinks IT is the system of truth for my
environment
● Not actually having any real choices
John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-

23. #monitoringlove
● Puppet
● Nagios (Icinga)
● Graphite
● Collectd
● Logstash

24. Graphite
● Graphing at Scale
● Graphing at Ease
● Any metric is a graph
● echo "somestring $somevalue
$timestamp" | nc <%= graphitehost
%> 2003

25. Graphite Composer
x

26. Graphite API

27. Gdash In action

28. Puppet and Graphite
● https://github.com/KrisBuytaert/vagrant-graphite/
● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd /
Statsd / Tattle and more modules as submodules !
● git clone
● git submodule init
● git submodule update
● vagrant up

29. Collectd
● Collects
● Zillion Plugins
• Nginx,apache, mysql, disk
● Graphite Carbon Plugin
● Send metrics to graphite

30. Collectd & Graphite

31. Exporting and Collecting

32. Triggers on Graphs
● Export Java Metrics ● Collect JMX Exports
on JMXTransNode
● JMXTrans
● Graph Em
● Export JMXConfigs
Collect Nagios
● Configure NRPE Configs on Nagios
Check Server
● Export NagiosCheck

33. Triggers on Graphs

34. Triggers on Graphs

35. Conclusion:
● Reproducable monitoring setup
● Dynamically generated monitoring
config
● Code is available at
github.com/KrisBuytaert/snippets

36. Contact
Kris Buytaert
Kris.Buytaert@inuits.be
Further Reading
@krisbuytaert
http://www.krisbuytaert.be/blog/
http://www.inuits.be/
Inuits
Duboistraat 50
2060 Antwerpen
Belgium
891.514.231
+32 475 961221