Successfully reported this slideshow.

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Monitoring your VM's at Scale

  1. Beyond VM deployment Monitoring your VM's at scale Kris Buytaert
  2. Kris Buytaert ● I used to be a Dev, ● Then Became an Op ● Chief Trolling Officer and Open Source Consultant @inuits.eu ● Everything is an effing DNS Problem ● Building Clouds since before the bookstore ● Some books, some papers, some blogs ● Evangelizing devops ● But mostly, trying to be good at my job
  3. What's different in the cloud ? ● Scale ● Velocity ● Change
  4. Challenges ● Reproducability ● Speed ● Auditing ● Keeping stuff in sync • Monitoring • Security
  5. Case : Using a configuration management tool to configure, update and keep your cloudscale monitoring and metric infrastructure sane and manageable.
  6. Tools ● Puppet ● Chef Cfengine ● Ganglia ● Collectd ● Sensu ● Graphite ● Nagios / Icinga
  7. Not quite a Muppet. ● Puppet is... ● OSS ● A DSL language ● Written in Ruby ● Client/server oriented ● Contains abstraction layers ● Repeatable processes
  8. Master of Puppets ● Puppet master • CA authority • Hosts Modules • Hosts Node descriptions • Compare, compile, apply ● Master is not a requirement !
  9. Puppet Clients ● daemon ● Cron jobs ● External orchestration: • for i in $hosts; do ssh $i “puppetd --test”; done • mCollective, Func, … ● Get catalogs, play them, ● reporting
  10. Puppet Environments ● Different code bases on 1 master ● Dev, Uat, Prod ● Only break one environment at once :) ● What about testing your Puppetmaster ?
  11. Node definitions ● Nodes.pp class defaults { $search = "inuits.be" $nameservers = ['208.67.220.220', '208.67.222.222'] include dns::resolv include ssh::keys include ssh::server } node "ns1.dev.inuits.be" { include defaults include dns::powerdns::server include dns::powerdns::resolver } node “web1.dev.inuits.be” { include defaults include apache2 include mysql }
  12. External Node Classifier ● Fixed hostname ? ● How many nodes ● Naming schemas solve some issues ● External script that sends back yaml class descriptions • Custom writtten • Foreman • ...
  13. Classes vs Modules ● Module : ● Abstract definition on configuring a service ● Reusable ● Class : ● Specific implementation of your use case of such a module •e.g usernames / passwords / hosts do not belong in modules
  14. Modules ● Files ● Templates ● Manifests • DSL • Classes • Elements
  15. Parametrized Classes
  16. Stored Configs
  17. Use Cases: ● Ssh keys ● Reverse proxy configs ● Monitoring resources ● Measuring resources
  18. Collection and Export Export : Collect: @@resource { Resource <<| query |>> ... } Clean out nodes that dissapear puppet node clean
  19. Defining a Service ● Local class that : • Configures service using a standard module call with hiera based parameters • Configures Backup • Configures logrotation • Configures logshipping • Exports Monitoring Needs ● Abuse modules for git ease
  20. Apache Example:
  21. #monitoringsucks Monitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love: ● Having my hands tied with the model of host and service bindings. ● Having to set up "fake" hosts just to group arbitrary metrics together ● Having to either collect metrics twice - once for alerting and another for trending ● Only being able to see my metrics in 5 minute intervals ● Having to chose between shitty interface but great monitoring or shitty monitoring but great interface ● Dealing with a monitoring system that thinks IT is the system of truth for my environment ● Not actually having any real choices John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-
  22. #monitoringlove ● Puppet ● Nagios (Icinga) ● Graphite ● Collectd ● Logstash
  23. Graphite ● Graphing at Scale ● Graphing at Ease ● Any metric is a graph ● echo "somestring $somevalue $timestamp" | nc <%= graphitehost %> 2003
  24. Graphite Composer x
  25. Graphite API
  26. Gdash In action
  27. Puppet and Graphite ● https://github.com/KrisBuytaert/vagrant-graphite/ ● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules ! ● git clone ● git submodule init ● git submodule update ● vagrant up
  28. Collectd ● Collects ● Zillion Plugins • Nginx,apache, mysql, disk ● Graphite Carbon Plugin ● Send metrics to graphite
  29. Collectd & Graphite
  30. Exporting and Collecting
  31. Triggers on Graphs ● Export Java Metrics ● Collect JMX Exports on JMXTransNode ● JMXTrans ● Graph Em ● Export JMXConfigs Collect Nagios ● Configure NRPE Configs on Nagios Check Server ● Export NagiosCheck
  32. Triggers on Graphs
  33. Triggers on Graphs
  34. Conclusion: ● Reproducable monitoring setup ● Dynamically generated monitoring config ● Code is available at github.com/KrisBuytaert/snippets
  35. Contact Kris Buytaert Kris.Buytaert@inuits.be Further Reading @krisbuytaert http://www.krisbuytaert.be/blog/ http://www.inuits.be/ Inuits Duboistraat 50 2060 Antwerpen Belgium 891.514.231 +32 475 961221

×