• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Monitoring your VM's at Scale
 

Monitoring your VM's at Scale

on

  • 5,129 views

Slides from the

Slides from the

Statistics

Views

Total Views
5,129
Views on SlideShare
4,930
Embed Views
199

Actions

Likes
10
Downloads
86
Comments
0

7 Embeds 199

https://twitter.com 153
http://canndlesavhh.blogspot.mx 39
http://www.scoop.it 2
https://www.linkedin.com 2
http://paulmccaughensportfolio.wordpress.com 1
https://si0.twimg.com 1
http://campamocha777.blogspot.mx 1
More...

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Monitoring your VM's at Scale Monitoring your VM's at Scale Presentation Transcript

    • Beyond VM deployment Monitoring your VMs at scale Kris Buytaert
    • Kris Buytaert● I used to be a Dev,● Then Became an Op● Chief Trolling Officer and Open Source Consultant @inuits.eu● Everything is an effing DNS Problem● Building Clouds since before the bookstore● Some books, some papers, some blogs● Evangelizing devops● But mostly, trying to be good at my job
    • Whats different in the cloud ?● Scale● Velocity● Change
    • Challenges● Reproducability● Speed● Auditing● Keeping stuff in sync • Monitoring • Security
    • Case :Using a configuration managementtool to configure, update and keepyour cloudscale monitoring and metricinfrastructure sane and manageable.
    • Tools● Puppet ● Chef Cfengine ● Ganglia● Collectd ● Sensu● Graphite● Nagios / Icinga
    • Not quite a Muppet.● Puppet is...● OSS● A DSL language● Written in Ruby● Client/server oriented● Contains abstraction layers● Repeatable processes
    • Master of Puppets● Puppet master • CA authority • Hosts Modules • Hosts Node descriptions • Compare, compile, apply● Master is not a requirement !
    • Puppet Clients● daemon● Cron jobs● External orchestration: • for i in $hosts; do ssh $i “puppetd --test”; done • mCollective, Func, …● Get catalogs, play them,● reporting
    • Puppet Environments● Different code bases on 1 master● Dev, Uat, Prod● Only break one environment at once :)● What about testing your Puppetmaster ?
    • Node definitions● Nodes.pp class defaults { $search = "inuits.be" $nameservers = [208.67.220.220, 208.67.222.222] include dns::resolv include ssh::keys include ssh::server } node "ns1.dev.inuits.be" { include defaults include dns::powerdns::server include dns::powerdns::resolver } node “web1.dev.inuits.be” { include defaults include apache2 include mysql }
    • External Node Classifier● Fixed hostname ?● How many nodes● Naming schemas solve some issues● External script that sends back yaml class descriptions • Custom writtten • Foreman • ...
    • Classes vs Modules● Module : ● Abstract definition on configuring a service ● Reusable● Class : ● Specific implementation of your use case of such a module•e.g usernames / passwords / hosts do not belong inmodules
    • Modules● Files● Templates● Manifests • DSL • Classes • Elements
    • Parametrized Classes
    • Stored Configs
    • Use Cases:● Ssh keys● Reverse proxy configs● Monitoring resources● Measuring resources
    • Collection and Export Export : Collect: @@resource { Resource <<| query |>> ... }Clean out nodes that dissapearpuppet node clean
    • Defining a Service● Local class that : • Configures service using a standard module call with hiera based parameters • Configures Backup • Configures logrotation • Configures logshipping • Exports Monitoring Needs● Abuse modules for git ease
    • Apache Example:
    • #monitoringsucks Monitoring is AWESOME. Metrics are AWESOME. I love it. Heres what I dont love:● Having my hands tied with the model of host and service bindings.● Having to set up "fake" hosts just to group arbitrary metrics together● Having to either collect metrics twice - once for alerting and another for trending● Only being able to see my metrics in 5 minute intervals● Having to chose between shitty interface but great monitoring or shitty monitoring but great interface● Dealing with a monitoring system that thinks IT is the system of truth for my environment● Not actually having any real choices John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-
    • #monitoringlove● Puppet● Nagios (Icinga)● Graphite● Collectd● Logstash
    • Graphite● Graphing at Scale● Graphing at Ease● Any metric is a graph● echo "somestring $somevalue $timestamp" | nc <%= graphitehost %> 2003
    • Graphite Composer x
    • Graphite API
    • Gdash In action
    • Puppet and Graphite● https://github.com/KrisBuytaert/vagrant-graphite/● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules !● git clone● git submodule init● git submodule update● vagrant up
    • Collectd● Collects● Zillion Plugins • Nginx,apache, mysql, disk● Graphite Carbon Plugin● Send metrics to graphite
    • Collectd & Graphite
    • Exporting and Collecting
    • Triggers on Graphs● Export Java Metrics ● Collect JMX Exports on JMXTransNode● JMXTrans ● Graph Em● Export JMXConfigs Collect Nagios● Configure NRPE Configs on Nagios Check Server● Export NagiosCheck
    • Triggers on Graphs
    • Triggers on Graphs
    • Conclusion:● Reproducable monitoring setup● Dynamically generated monitoring config● Code is available at github.com/KrisBuytaert/snippets
    • ContactKris BuytaertKris.Buytaert@inuits.beFurther Reading@krisbuytaerthttp://www.krisbuytaert.be/blog/http://www.inuits.be/ Inuits Duboistraat 50 2060 Antwerpen Belgium 891.514.231 +32 475 961221