Monitoring your VM's at Scale
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Monitoring your VM's at Scale

on

  • 5,493 views

Slides from the

Slides from the

Statistics

Views

Total Views
5,493
Views on SlideShare
5,292
Embed Views
201

Actions

Likes
10
Downloads
91
Comments
0

7 Embeds 201

https://twitter.com 155
http://canndlesavhh.blogspot.mx 39
http://www.scoop.it 2
https://www.linkedin.com 2
http://paulmccaughensportfolio.wordpress.com 1
https://si0.twimg.com 1
http://campamocha777.blogspot.mx 1
More...

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Monitoring your VM's at Scale Presentation Transcript

  • 1. Beyond VM deployment Monitoring your VMs at scale Kris Buytaert
  • 2. Kris Buytaert● I used to be a Dev,● Then Became an Op● Chief Trolling Officer and Open Source Consultant @inuits.eu● Everything is an effing DNS Problem● Building Clouds since before the bookstore● Some books, some papers, some blogs● Evangelizing devops● But mostly, trying to be good at my job
  • 3. Whats different in the cloud ?● Scale● Velocity● Change
  • 4. Challenges● Reproducability● Speed● Auditing● Keeping stuff in sync • Monitoring • Security
  • 5. Case :Using a configuration managementtool to configure, update and keepyour cloudscale monitoring and metricinfrastructure sane and manageable.
  • 6. Tools● Puppet ● Chef Cfengine ● Ganglia● Collectd ● Sensu● Graphite● Nagios / Icinga
  • 7. Not quite a Muppet.● Puppet is...● OSS● A DSL language● Written in Ruby● Client/server oriented● Contains abstraction layers● Repeatable processes
  • 8. Master of Puppets● Puppet master • CA authority • Hosts Modules • Hosts Node descriptions • Compare, compile, apply● Master is not a requirement !
  • 9. Puppet Clients● daemon● Cron jobs● External orchestration: • for i in $hosts; do ssh $i “puppetd --test”; done • mCollective, Func, …● Get catalogs, play them,● reporting
  • 10. Puppet Environments● Different code bases on 1 master● Dev, Uat, Prod● Only break one environment at once :)● What about testing your Puppetmaster ?
  • 11. Node definitions● Nodes.pp class defaults { $search = "inuits.be" $nameservers = [208.67.220.220, 208.67.222.222] include dns::resolv include ssh::keys include ssh::server } node "ns1.dev.inuits.be" { include defaults include dns::powerdns::server include dns::powerdns::resolver } node “web1.dev.inuits.be” { include defaults include apache2 include mysql }
  • 12. External Node Classifier● Fixed hostname ?● How many nodes● Naming schemas solve some issues● External script that sends back yaml class descriptions • Custom writtten • Foreman • ...
  • 13. Classes vs Modules● Module : ● Abstract definition on configuring a service ● Reusable● Class : ● Specific implementation of your use case of such a module•e.g usernames / passwords / hosts do not belong inmodules
  • 14. Modules● Files● Templates● Manifests • DSL • Classes • Elements
  • 15. Parametrized Classes
  • 16. Stored Configs
  • 17. Use Cases:● Ssh keys● Reverse proxy configs● Monitoring resources● Measuring resources
  • 18. Collection and Export Export : Collect: @@resource { Resource <<| query |>> ... }Clean out nodes that dissapearpuppet node clean
  • 19. Defining a Service● Local class that : • Configures service using a standard module call with hiera based parameters • Configures Backup • Configures logrotation • Configures logshipping • Exports Monitoring Needs● Abuse modules for git ease
  • 20. Apache Example:
  • 21. #monitoringsucks Monitoring is AWESOME. Metrics are AWESOME. I love it. Heres what I dont love:● Having my hands tied with the model of host and service bindings.● Having to set up "fake" hosts just to group arbitrary metrics together● Having to either collect metrics twice - once for alerting and another for trending● Only being able to see my metrics in 5 minute intervals● Having to chose between shitty interface but great monitoring or shitty monitoring but great interface● Dealing with a monitoring system that thinks IT is the system of truth for my environment● Not actually having any real choices John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-
  • 22. #monitoringlove● Puppet● Nagios (Icinga)● Graphite● Collectd● Logstash
  • 23. Graphite● Graphing at Scale● Graphing at Ease● Any metric is a graph● echo "somestring $somevalue $timestamp" | nc <%= graphitehost %> 2003
  • 24. Graphite Composer x
  • 25. Graphite API
  • 26. Gdash In action
  • 27. Puppet and Graphite● https://github.com/KrisBuytaert/vagrant-graphite/● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules !● git clone● git submodule init● git submodule update● vagrant up
  • 28. Collectd● Collects● Zillion Plugins • Nginx,apache, mysql, disk● Graphite Carbon Plugin● Send metrics to graphite
  • 29. Collectd & Graphite
  • 30. Exporting and Collecting
  • 31. Triggers on Graphs● Export Java Metrics ● Collect JMX Exports on JMXTransNode● JMXTrans ● Graph Em● Export JMXConfigs Collect Nagios● Configure NRPE Configs on Nagios Check Server● Export NagiosCheck
  • 32. Triggers on Graphs
  • 33. Triggers on Graphs
  • 34. Conclusion:● Reproducable monitoring setup● Dynamically generated monitoring config● Code is available at github.com/KrisBuytaert/snippets
  • 35. ContactKris BuytaertKris.Buytaert@inuits.beFurther Reading@krisbuytaerthttp://www.krisbuytaert.be/blog/http://www.inuits.be/ Inuits Duboistraat 50 2060 Antwerpen Belgium 891.514.231 +32 475 961221