Beyond VM deployment   Monitoring your VMs at scale           Kris Buytaert
Kris Buytaert●   I used to be a Dev,●   Then Became an Op●   Chief Trolling Officer and Open Source    Consultant @inuits....
Whats different in the cloud ?●   Scale●   Velocity●   Change
Challenges●   Reproducability●   Speed●   Auditing●   Keeping stuff in sync    •   Monitoring    •   Security
Case :Using a configuration managementtool to configure, update and keepyour cloudscale monitoring and metricinfrastructur...
Tools●   Puppet            ●   Chef Cfengine                      ●   Ganglia●   Collectd                      ●   Sensu● ...
Not quite a Muppet.●   Puppet is...●   OSS●   A DSL language●   Written in Ruby●   Client/server oriented●   Contains abst...
Master of Puppets●   Puppet master    •   CA authority    •   Hosts Modules    •   Hosts Node descriptions    •   Compare,...
Puppet Clients●   daemon●   Cron jobs●   External orchestration:    •   for i in $hosts; do ssh $i “puppetd --test”; done ...
Puppet Environments●   Different code bases on 1 master●   Dev, Uat, Prod●   Only break one environment at once :)●   What...
Node definitions●   Nodes.pp    class defaults {         $search = "inuits.be"         $nameservers = [208.67.220.220, 208...
External Node Classifier●   Fixed hostname ?●   How many nodes●   Naming schemas solve some issues●   External script that...
Classes vs Modules●   Module :          ●              Abstract definition on configuring a service          ●            ...
Modules●   Files●   Templates●   Manifests    •   DSL    •   Classes    •   Elements
Parametrized  Classes
Stored Configs
Use Cases:●   Ssh keys●   Reverse proxy configs●   Monitoring resources●   Measuring resources
Collection and Export Export :           Collect: @@resource {       Resource <<|                    query |>> ... }Clean ...
Defining a Service●   Local class that :    •   Configures service using a standard        module call with hiera based pa...
Apache Example:
#monitoringsucks    Monitoring is AWESOME. Metrics are AWESOME. I love it. Heres what I dont love:●   Having my hands tied...
#monitoringlove●   Puppet●   Nagios (Icinga)●   Graphite●   Collectd●   Logstash
Graphite●   Graphing at Scale●   Graphing at Ease●   Any metric is a graph●   echo "somestring $somevalue    $timestamp" |...
Graphite Composer    x
Graphite API
Gdash In action
Puppet and Graphite●   https://github.com/KrisBuytaert/vagrant-graphite/●   Includes Graphite / Gdash / Jmxtrans / Logster...
Collectd●   Collects●   Zillion Plugins    •   Nginx,apache, mysql, disk●   Graphite Carbon Plugin●   Send metrics to grap...
Collectd & Graphite
Exporting and Collecting
Triggers on Graphs●   Export Java Metrics   ●   Collect JMX Exports                              on JMXTransNode●   JMXTra...
Triggers on Graphs
Triggers on Graphs
Conclusion:●   Reproducable monitoring setup●   Dynamically generated monitoring    config●   Code is available at    gith...
ContactKris BuytaertKris.Buytaert@inuits.beFurther Reading@krisbuytaerthttp://www.krisbuytaert.be/blog/http://www.inuits.b...
Monitoring your VM's at Scale
Upcoming SlideShare
Loading in...5
×

Monitoring your VM's at Scale

4,832

Published on

Slides from the

Published in: Technology
0 Comments
11 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,832
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
98
Comments
0
Likes
11
Embeds 0
No embeds

No notes for slide

Monitoring your VM's at Scale

  1. 1. Beyond VM deployment Monitoring your VMs at scale Kris Buytaert
  2. 2. Kris Buytaert● I used to be a Dev,● Then Became an Op● Chief Trolling Officer and Open Source Consultant @inuits.eu● Everything is an effing DNS Problem● Building Clouds since before the bookstore● Some books, some papers, some blogs● Evangelizing devops● But mostly, trying to be good at my job
  3. 3. Whats different in the cloud ?● Scale● Velocity● Change
  4. 4. Challenges● Reproducability● Speed● Auditing● Keeping stuff in sync • Monitoring • Security
  5. 5. Case :Using a configuration managementtool to configure, update and keepyour cloudscale monitoring and metricinfrastructure sane and manageable.
  6. 6. Tools● Puppet ● Chef Cfengine ● Ganglia● Collectd ● Sensu● Graphite● Nagios / Icinga
  7. 7. Not quite a Muppet.● Puppet is...● OSS● A DSL language● Written in Ruby● Client/server oriented● Contains abstraction layers● Repeatable processes
  8. 8. Master of Puppets● Puppet master • CA authority • Hosts Modules • Hosts Node descriptions • Compare, compile, apply● Master is not a requirement !
  9. 9. Puppet Clients● daemon● Cron jobs● External orchestration: • for i in $hosts; do ssh $i “puppetd --test”; done • mCollective, Func, …● Get catalogs, play them,● reporting
  10. 10. Puppet Environments● Different code bases on 1 master● Dev, Uat, Prod● Only break one environment at once :)● What about testing your Puppetmaster ?
  11. 11. Node definitions● Nodes.pp class defaults { $search = "inuits.be" $nameservers = [208.67.220.220, 208.67.222.222] include dns::resolv include ssh::keys include ssh::server } node "ns1.dev.inuits.be" { include defaults include dns::powerdns::server include dns::powerdns::resolver } node “web1.dev.inuits.be” { include defaults include apache2 include mysql }
  12. 12. External Node Classifier● Fixed hostname ?● How many nodes● Naming schemas solve some issues● External script that sends back yaml class descriptions • Custom writtten • Foreman • ...
  13. 13. Classes vs Modules● Module : ● Abstract definition on configuring a service ● Reusable● Class : ● Specific implementation of your use case of such a module•e.g usernames / passwords / hosts do not belong inmodules
  14. 14. Modules● Files● Templates● Manifests • DSL • Classes • Elements
  15. 15. Parametrized Classes
  16. 16. Stored Configs
  17. 17. Use Cases:● Ssh keys● Reverse proxy configs● Monitoring resources● Measuring resources
  18. 18. Collection and Export Export : Collect: @@resource { Resource <<| query |>> ... }Clean out nodes that dissapearpuppet node clean
  19. 19. Defining a Service● Local class that : • Configures service using a standard module call with hiera based parameters • Configures Backup • Configures logrotation • Configures logshipping • Exports Monitoring Needs● Abuse modules for git ease
  20. 20. Apache Example:
  21. 21. #monitoringsucks Monitoring is AWESOME. Metrics are AWESOME. I love it. Heres what I dont love:● Having my hands tied with the model of host and service bindings.● Having to set up "fake" hosts just to group arbitrary metrics together● Having to either collect metrics twice - once for alerting and another for trending● Only being able to see my metrics in 5 minute intervals● Having to chose between shitty interface but great monitoring or shitty monitoring but great interface● Dealing with a monitoring system that thinks IT is the system of truth for my environment● Not actually having any real choices John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-
  22. 22. #monitoringlove● Puppet● Nagios (Icinga)● Graphite● Collectd● Logstash
  23. 23. Graphite● Graphing at Scale● Graphing at Ease● Any metric is a graph● echo "somestring $somevalue $timestamp" | nc <%= graphitehost %> 2003
  24. 24. Graphite Composer x
  25. 25. Graphite API
  26. 26. Gdash In action
  27. 27. Puppet and Graphite● https://github.com/KrisBuytaert/vagrant-graphite/● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules !● git clone● git submodule init● git submodule update● vagrant up
  28. 28. Collectd● Collects● Zillion Plugins • Nginx,apache, mysql, disk● Graphite Carbon Plugin● Send metrics to graphite
  29. 29. Collectd & Graphite
  30. 30. Exporting and Collecting
  31. 31. Triggers on Graphs● Export Java Metrics ● Collect JMX Exports on JMXTransNode● JMXTrans ● Graph Em● Export JMXConfigs Collect Nagios● Configure NRPE Configs on Nagios Check Server● Export NagiosCheck
  32. 32. Triggers on Graphs
  33. 33. Triggers on Graphs
  34. 34. Conclusion:● Reproducable monitoring setup● Dynamically generated monitoring config● Code is available at github.com/KrisBuytaert/snippets
  35. 35. ContactKris BuytaertKris.Buytaert@inuits.beFurther Reading@krisbuytaerthttp://www.krisbuytaert.be/blog/http://www.inuits.be/ Inuits Duboistraat 50 2060 Antwerpen Belgium 891.514.231 +32 475 961221
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×