Your SlideShare is downloading. ×
0
11 Tools for your devops          stack        Kris Buytaert       @krisbuytaert
Kris Buytaert•   I used to be a Dev,•   Then Became an Op•   Chief Trolling Officer and Open Source    Consultant @inuits....
Whats this devops thing       anyhow ?
World , 200X-2009 Patrick Debois, Gildas Le Nadan, Andrew Clay Shafer, Kris Buytaert, JezzHumble, Lindsay Holmwood, John W...
●   Devops is a growing professional and cultural    movement●   We dont have all the answers yet●   We are reaching out t...
Devops, a definition:
●   Adopt the new philosophy. We are in a new economic age. Western management    must awaken to the challenge, must learn...
William Edwards          Deming1986, Out of the Crisis.  http://en.wikipedia.org/wiki/W._Edwards_Deming
CAMS●   Culture●   Automation●   Measurement●   Sharing              Damon Edwards and John Willis
Culture,   automation,  Measturement,     sharing
Devs vs Ops
Breaking the SilosDevs    Ops    Getting Along
What tool?
Culture,Automation,    Measurement,      Sharing
“Solved problems”●   Deployment automation :         ●             Fai , kickstart, preseed, cobbler,●   Configuration Man...
“challenges”•    Build & Release Management•    Package & Repository management•    Orchestration
Jenkins●   Open Source Continuous Integration Server●   A zillion plugins (400)●   Have developers build stable and deploy...
Jenkins
fpm    #packagingsucks             ●                 Missing upstream             ●                 Ancient upstream      ...
fpm
fpmfpm -t rpm -s dir -n hornetq -v 2.2.5 hornetqExecuting(%prep): /bin/sh -e /var/tmp/rpm-tmp.nNkVwh+ umask 022+ cd /usr/l...
fpm in action●   https://github.com/Inuits/build-gems●   Fork, pull●   Jenkins pulls , builds , pushes to repo●   (variant...
The Marionette Collective●   Distributed ssh ++●   What version of ssh do I have installed on my    servers ?●   On what s...
mc-servicemc-service httpd status -W /uat/ * [ ============================================================> ] 80 /80 bleh...
mc-packagemc-package -W /dev/ status jdk * [ ============================================================> ] 33 / 33servic...
Culture,     Automation,Measurement,       Sharing
#monitoringsucks    Monitoring is AWESOME. Metrics are AWESOME. I love it. Heres what I dont love:●   Having my hands tied...
A sub movement●   #monitoringsucks trending●   https://github.com/monitoringsucks/●   2008 Study :Nagios + Friends●   2011...
Logstash●   Not your average centralized logging tool●   Elasticsearch backed●   Shipper●   Indexer●   Web
Logstash    ●   Collect from        anywhere    ●   Filter    ●   Send anywhere    ●   Queuing support
Logstash in Action
Kibana in Action
Puppet and Logstash●   https://github.com/KrisBuytaert/vagrant-    puppet-logstash●   Includes Logstash / Kibana / Grok / ...
Graphite●   Graphing at Scale●   Graphing at Ease●   Any metric is a graph●   echo "somestring $somevalue $timestamp" |   ...
Graphite Composer   x
Graphite API
Friends of Graphite●   Collection :          ●              Statsd,          ●              Collectd + Carbonplugin       ...
Gdash In action
Puppet and Graphite●   https://github.com/KrisBuytaert/vagrant-    graphite/●   Includes Graphite / Gdash / Jmxtrans /    ...
Culture,  Automation,  Measurement,Sharing
●   Open Source●   Github●   Talk about Experiences●   Open Spaces
Sharing environments●   Build identical environments●   Share code●   Shared ownership of content , code and    configurat...
Vagrant●   Abstraction layer for VirtualBox●   Integrates well with Puppet/Chef●   Project =          ●              Vagra...
VagrantfileVagrant::Config.run do |config| # All Vagrant configuration is done here. The most common configuration # optio...
Vagrant Rocks●   Vagrant init●   Vagrant up●   Vagrant provision●   Vagrant down●   Vagrant destroy
Sushi
Quiz Time :Which tool did I forget ?
You
Linkshttp://puppetlabs.com/mcollective/introduction/https://github.com/jordansissel/fpmhttp://logstash.net/https://github....
ContactKris BuytaertKris.Buytaert@inuits.beFurther Reading@krisbuytaerthttp://www.krisbuytaert.be/blog/http://www.inuits.b...
11 Tools for your Open Source devops stack
11 Tools for your Open Source devops stack
11 Tools for your Open Source devops stack
Upcoming SlideShare
Loading in...5
×

11 Tools for your Open Source devops stack

12,638

Published on

11 Tools for your Open Source devops stack as presented at Linuxcon 2012 Barcelona, and T-Dose , Fall 2012

Published in: Technology
1 Comment
32 Likes
Statistics
Notes
No Downloads
Views
Total Views
12,638
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
324
Comments
1
Likes
32
Embeds 0
No embeds

No notes for slide

Transcript of "11 Tools for your Open Source devops stack "

  1. 1. 11 Tools for your devops stack Kris Buytaert @krisbuytaert
  2. 2. Kris Buytaert• I used to be a Dev,• Then Became an Op• Chief Trolling Officer and Open Source Consultant @inuits.eu• Everything is an effing DNS Problem
  3. 3. Whats this devops thing anyhow ?
  4. 4. World , 200X-2009 Patrick Debois, Gildas Le Nadan, Andrew Clay Shafer, Kris Buytaert, JezzHumble, Lindsay Holmwood, John Willis, Chris Read, Julian Simpson, Luke Kanies, John Allspaw and lots of others .. Gent , October 2009 Mountain View , June 2010 Hamburg , October 2010 Boston, March 2011 Mountain View, June 2011 Bangalore, Melbourne, Goteborg , October 2011 Rome, earlier this month! ....
  5. 5. ● Devops is a growing professional and cultural movement● We dont have all the answers yet● We are reaching out to different communities● We will point out problems we see..● Only the name is new While we are still working out the solutions
  6. 6. Devops, a definition:
  7. 7. ● Adopt the new philosophy. We are in a new economic age. Western management must awaken to the challenge, must learn their responsibilities, and take on leadership for change.● Cease dependence on inspection to achieve quality. Eliminate the need for massive inspection by building quality into the product in the first place.● Improve constantly and forever the system of production and service, to improve quality and productivity, and thus constantly decrease costs.● Institute training on the job.● Institute leadership The aim of supervision should be to help people and machines and gadgets do a better job.● Drive out fear, so that everyone may work effectively for the company.● Break down barriers between departments. People in research, design, sales, and production must work as a team, in order to foresee problems of production and usage that may be encountered with the product or service.● Eliminate slogans, exhortations, and targets for the work force asking for zero defects and new levels of productivity. Such exhortations only create adversarial relationships, as the bulk of the causes of low quality and low productivity belong to the system and thus lie beyond the power of the work force. ● Eliminate management by objective. Eliminate management by numbers and numerical goals. Instead substitute with leadership. ● Remove barriers that rob the hourly worker of his right to pride of workmanship. The responsibility of supervisors must be changed from sheer numbers to quality. ● Remove barriers that rob people in management and in engineering of their right to pride of workmanship.● Institute a vigorous program of education and self-improvement.● Put everybody in the company to work to accomplish the transformation. The transformation is everybodys job.
  8. 8. William Edwards Deming1986, Out of the Crisis. http://en.wikipedia.org/wiki/W._Edwards_Deming
  9. 9. CAMS● Culture● Automation● Measurement● Sharing Damon Edwards and John Willis
  10. 10. Culture, automation, Measturement, sharing
  11. 11. Devs vs Ops
  12. 12. Breaking the SilosDevs Ops Getting Along
  13. 13. What tool?
  14. 14. Culture,Automation, Measurement, Sharing
  15. 15. “Solved problems”● Deployment automation : ● Fai , kickstart, preseed, cobbler,● Configuration Management ● Puppet, chef, cfengine, ● ....
  16. 16. “challenges”• Build & Release Management• Package & Repository management• Orchestration
  17. 17. Jenkins● Open Source Continuous Integration Server● A zillion plugins (400)● Have developers build stable and deployable code● Test Infra code
  18. 18. Jenkins
  19. 19. fpm #packagingsucks ● Missing upstream ● Ancient upstream ● Unneeded dependencies ● Broken upstream ● Distro Policies● “Maximum RPM” ?● Packaging is needed ● Dependencies, tooling, repositories● Anger driven development
  20. 20. fpm
  21. 21. fpmfpm -t rpm -s dir -n hornetq -v 2.2.5 hornetqExecuting(%prep): /bin/sh -e /var/tmp/rpm-tmp.nNkVwh+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ exit 0Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.yUd4MV+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ tar -zxf /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/data.tar.gz+ exit 0Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.jkpqeA+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ /usr/lib/rpm/brp-compress+ /usr/lib/rpm/brp-strip+ /usr/lib/rpm/brp-strip-static-archive+ /usr/lib/rpm/brp-strip-comment-noteProcessing files: hornetq-2.2.5-1.x86_64Checking for unpackaged file(s): /usr/lib/rpm/check-files /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILDWrote: /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/SRPMS/hornetq-2.2.5-1.src.rpmWrote: /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/RPMS/x86_64/hornetq-2.2.5-1.x86_64.rpmExecuting(%clean): /bin/sh -e /var/tmp/rpm-tmp.z2UL3B+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ rm -rf /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ exit 0Created /usr/local/hornetq-2.2.5.x86_64.rpm
  22. 22. fpm in action● https://github.com/Inuits/build-gems● Fork, pull● Jenkins pulls , builds , pushes to repo● (variants for Nagios Plugins / Jenkins Plugins available)
  23. 23. The Marionette Collective● Distributed ssh ++● What version of ssh do I have installed on my servers ?● On what servers is XYZ running ?● Clean al my ssl certs ?● Restart apache on all servers with fact X
  24. 24. mc-servicemc-service httpd status -W /uat/ * [ ============================================================> ] 80 /80 bleh.poc.com status=stopped xs03.ta.google.com status=stopped elasticsearch02.uat.com status=stopped dbs01-.poc.com status=stopped dino.uat.com status=stopped dsp01.uat.com status=stopped webserver01.uat.com status=running dbs01-old.uat.com status=stopped alexdb02-hdp.poc.com status=stopped postfix.uat.com status=stopped ns01.uat.com status=stopped devtools03.uat.com status=running
  25. 25. mc-packagemc-package -W /dev/ status jdk * [ ============================================================> ] 33 / 33servicesdb01.dev.com version = -absentservices.dev.google.com version = jdk-1.6.0_13-fcsdrbdtest02.dev.google.com version = -absentservices3.dev.google.com version = jdk-1.6.0_20-fcsum.dev.google.com version = jdk-1.5.0_19-fcsdevtools03.uat.com version = jdk-1.6.0_29-fcsalexandria02.dev.google.com version = -absentweblink01.dev.com version = -absentwikitest.dev.google.com version = jdk-1.6.0_24-fcspayment.dev.google.com version = jdk-1.5.0_17-fcstiff2pdf01.dev.com version = -absentdevdoos.dev.com version = jdk-1.6.0_30-fcswiki.dev.google.com version = jdk-1.6.0_24-fcsreporting01.dev.com version = -absentdevtools01-dev.uat.com version = jdk-1.6.0_23-fcsdevtools02.uat.com version = jdk-1.6.0_29-fcsdrbdtest01.dev.google.com version = -absent---- package agent summary ---- Nodes: 33/33 Versions: 1 * 1.5.0_17-fcs, 1 * 1.5.0_19-fcs, 1 * 1.6.0_13-fcs, 1 * 1.6.0_20-fcs, 1 * 1.6.0_23-fcs, 2 * 1.6.0_24-fcs, 2 * 1.6.0_29-fcs Elapsed Time: 1.73 s
  26. 26. Culture, Automation,Measurement, Sharing
  27. 27. #monitoringsucks Monitoring is AWESOME. Metrics are AWESOME. I love it. Heres what I dont love:● Having my hands tied with the model of host and service bindings.● Having to set up "fake" hosts just to group arbitrary metrics together● Having to either collect metrics twice - once for alerting and another for trending● Only being able to see my metrics in 5 minute intervals● Having to chose between shitty interface but great monitoring or shitty monitoring but great interface● Dealing with a monitoring system that thinks IT is the system of truth for my environment● Not actually having any real choices John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why- monitoring-sucks.html
  28. 28. A sub movement● #monitoringsucks trending● https://github.com/monitoringsucks/● 2008 Study :Nagios + Friends● 2011 Conclusion : Nagios/Icinga are the only automatable alternatives● Monitoring and trending at scale , new kids Graphite, Icinga, flapjack, etc● Old Cool Kids, Ganglia
  29. 29. Logstash● Not your average centralized logging tool● Elasticsearch backed● Shipper● Indexer● Web
  30. 30. Logstash ● Collect from anywhere ● Filter ● Send anywhere ● Queuing support
  31. 31. Logstash in Action
  32. 32. Kibana in Action
  33. 33. Puppet and Logstash● https://github.com/KrisBuytaert/vagrant- puppet-logstash● Includes Logstash / Kibana / Grok / ES modules as submodules !● git clone● git submodule init● git submodule update● vagrant up
  34. 34. Graphite● Graphing at Scale● Graphing at Ease● Any metric is a graph● echo "somestring $somevalue $timestamp" | nc <%= graphitehost %> 2003
  35. 35. Graphite Composer x
  36. 36. Graphite API
  37. 37. Friends of Graphite● Collection : ● Statsd, ● Collectd + Carbonplugin ● Jmxtrans ● Logster● Dashboards ● Tattle ● Gdash ● ...
  38. 38. Gdash In action
  39. 39. Puppet and Graphite● https://github.com/KrisBuytaert/vagrant- graphite/● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules !● git clone● git submodule init● git submodule update● vagrant up
  40. 40. Culture, Automation, Measurement,Sharing
  41. 41. ● Open Source● Github● Talk about Experiences● Open Spaces
  42. 42. Sharing environments● Build identical environments● Share code● Shared ownership of content , code and configuration
  43. 43. Vagrant● Abstraction layer for VirtualBox● Integrates well with Puppet/Chef● Project = ● Vagrantfile ● Manifests / Cookbooks● Portable, Small , Versionable● Use veewee to build your boxen
  44. 44. VagrantfileVagrant::Config.run do |config| # All Vagrant configuration is done here. The most common configuration # options are documented and commented below. For a complete reference, # please see the online documentation at vagrantup.com. config.vm.define :mongo1 do |mongo1_config| mongo1_config.ssh.max_tries = 100 mongo1_config.vm.box = "MyCentOS2" mongo1_config.vm.network("192.168.99.101") mongo1_config.vm.host_name = "mongo1" mongo1_config.vm.provision :puppet do |mongo1_puppet| mongo1_puppet.pp_path = "/tmp/vagrant-puppet" mongo1_puppet.manifests_path = "manifests" mongo1_puppet.module_path = "modules" mongo1_puppet.manifest_file = "site.pp" end end config.vm.define :mongo2 do |mongo2_config| mongo2_config.ssh.max_tries = 100 mongo2_config.vm.box = "MyCentOS2" mongo2_config.vm.network("192.168.99.102") mongo2_config.vm.host_name = "mongo2" mongo2_config.vm.provision :puppet do |mongo2_puppet| mongo2_puppet.pp_path = "/tmp/vagrant-puppet" mongo2_puppet.manifests_path = "manifests" mongo2_puppet.module_path = "modules" mongo2_puppet.manifest_file = "site.pp" end end
  45. 45. Vagrant Rocks● Vagrant init● Vagrant up● Vagrant provision● Vagrant down● Vagrant destroy
  46. 46. Sushi
  47. 47. Quiz Time :Which tool did I forget ?
  48. 48. You
  49. 49. Linkshttp://puppetlabs.com/mcollective/introduction/https://github.com/jordansissel/fpmhttp://logstash.net/https://github.com/rashidkpc/Kibanahttp://graphite.wikidot.com/https://github.com/wayfair/Graphite-Tattlehttp://vagrantup.com/https://github.com/KrisBuytaert
  50. 50. ContactKris BuytaertKris.Buytaert@inuits.beFurther Reading@krisbuytaerthttp://www.krisbuytaert.be/blog/http://www.inuits.be/ Inuits Duboistraat 50 2060 Antwerpen Belgium 891.514.231 +32 475 961221
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×