Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Infrastructure as Code - ABUG Session

42,545 views

Published on

An overview of the infrast

Published in: Technology, Self Improvement

Infrastructure as Code - ABUG Session

  1. 1. Infrastructure asPatrick DeboisABUG-Session
  2. 2. Blog: http://jedi.be/blog - Twitter: @patrickdebois - #devopsTechnical ReviewerMonitoring Chapter Co-Author Veewee / Sahara / MccloudOrganizer Since 2009 Europe Organizer 2010EngineerFirst EuropeTraining 2010Libvirt - FogSpeakerFreelance consultant http://github.com/jedi4ever
  3. 3. CONFIGURATIONMANAGEMENT SYSTEMShttp://ansible.cc/http://opscode.com/chef/http://cfengine.com/http://palletops.com/http://puppetlabs.com/
  4. 4. http://www.nerdnirvana.org/wp-content/uploads/2010/12/apology-form.jpgChef & Puppet focusedbased on my experience
  5. 5. SERVICEFILE2FILE1PACKAGEDECLARATIVE LANGUAGE
  6. 6. “KNOWN STATE”STATE X STATEYCODE RUNFILE1SERVICEFILE2FILE1PACKAGEconvergence
  7. 7. “IDEMPOTENCE”STATEY STATEYCODE RUNSERVICEFILE2FILE1PACKAGESERVICEFILE2FILE1PACKAGE
  8. 8. GIVE METHE CODE:“LANGUAGE”
  9. 9. Puppet Chefclass myapache {package “apache2”service “apache2”:ensure => “running”,require => Package[“apache2”]}package “apache2”service “apache2 doaction [:enable , :start]endEXAMPLE SYNTAX
  10. 10. TERMINOLOGYJava Puppet Chef*.javamanifests*.pprecipes*.rbpackage module cookbook
  11. 11. TERMINOLOGY (2)Java Puppet Chefsingletonparametrizedclassesrecipes withattributesclassdefines,classesdefinitions,libraries
  12. 12. SYNTAXPuppet ChefLimited DSL(by design)DSL + full rubyExtended byDefinitionsLight WeightResource Providers /DefinitionsCustom ResourceprovidersProviders, LightWeight ResourceProvidersERB & CustomLibrariesLibraries
  13. 13. LOOPING CONSTRUCTSPuppet Chefpass array of elementserb template for loopscreate_resource(:type, hash)do while , repeat etc.available throughnative ruby
  14. 14. VARIABLESPuppet Chefspecial syntax$bla = “aa”string interpolation$bla_string = “${bla}”native rubybla = “aa”string interpolationbla_string = #{bla}
  15. 15. SINGLE CLASSINHERITANCEPuppet Chefoverride via ‘modulepath’override via‘cookbook’ path+> operator “cookbook” attribute
  16. 16. Puppet Module Chef Cookbookfiles/lib/README.rdocmanifests/templates/attributes/definitions/files/libraries/metadata.rbproviders/README.rdocrecipes/resources/templates/PROJECT STRUCTURE
  17. 17. EXECUTION MODELPuppet ChefCompile catalog(directed graphmodel)Then ExecuteTop Down execution
  18. 18. PLUGINSPuppet ChefFACESKNIFEPLUGINSREPORTSREPORTHANDLERShttp://puppetlabs.com/blog/puppet-faces-what-the-heck-are-faces/http://wiki.opscode.com/display/chef/Knife+Plugins
  19. 19. CODE EDITOR
  20. 20. https://github.com/rodjek/puppet-pygments-lexerhttp://redmine.djagios.org/projects/puppet-lexerLEXERS
  21. 21. VIM SYNTAX HIGHLIGHTIINGhttps://github.com/rodjek/vim-puppethttps://github.com/t9md/vim-chefhttps://github.com/vim-scripts/Cfengine-version-3-syntax
  22. 22. http://cloudsmith.github.com/geppetto/https://github.com/cloudsmith/geppettoGEPPETTO - ECLIPSE IDE
  23. 23. JETBRAINS RUBYMINE (PUPPET)http://youtrack.jetbrains.com/issue/RUBY-10832REACTOR8http://beta.r8network.com/COMING SOONhttps://github.com/janschumann/intellij-lang-puppet
  24. 24. REVERSE ENGINEERINGhttps://github.com/devstructure/blueprintExisting Server -> Manifest/Recipes
  25. 25. CODING STYLE
  26. 26. Puppet Chefhttp://puppet-lint.com/ http://acrmp.github.com/foodcritic/• Must use two-space soft tabs.• Must not use literal tab characters.• Must not contain trailing white space.• Should not exceed an 80 character line widthAn exception has been made for source  =>  puppet://... lines assplitting these over multiple lines decreases the readability of themanifests.• Should align arrows (=>) within blocks of attributes.• ....•• FC001: Use strings in preference to symbols to access node attributes• FC002: Avoid string interpolation where not required• FC003: Check whether you are running with chef server before using server-specific features• FC004: Use a service resource to start and stop services• FC005: Avoid repetition of resource declarations• FC006: Mode should be quoted or fully specified when setting filepermissions• FC007: Ensure recipe dependencies are reflected in cookbook metadata• FC008: Generated cookbook metadata needs updating• FC009: Resource attribute not recognised• FC010: Invalid search syntax• FC011: Missing README in markdown format• FC012: Use Markdown for README rather than RDoc• FC013: Use file_cache_path rather than hard-coding tmp paths•http://docs.puppetlabs.com/guides/style_guide.htmlhttps://github.com/ampledata/cookbook-style-guide
  27. 27. http://www.slideshare.net/PuppetLabs/modern-module-development-ken-barber-2012-edinburgh-puppet-camp
  28. 28. CODEVS CONFIG(THINK “PROFILES”)http://wiki.opscode.com/display/chef/Data+Bagshttp://projects.puppetlabs.com/projects/hierahttps://github.com/crayfishx/hiera-gpg
  29. 29. https://github.com/jedi4ever/stop-the-forkhttp://realityforge.org/code/2012/11/12/reusable-cookbooks-revisited.htmlhttp://devopsanywhere.blogspot.it/2012/11/how-to-write-reusable-chef-cookbooks.html“STOP THEFORK(ING)”
  30. 30. PRE-COMMIT CHECKS
  31. 31. DEPENDENCY MGMTGitSubmodulesSubtreeà la Pom File
  32. 32. UPSTREAM REPOShttp://forge.puppetlabs.com/http://community.opscode.com/
  33. 33. Puppet Chefhttps://github.com/rodjek/librarian-puppethttps://github.com/applicationsonline/librarianhttp://berkshelf.com/site  "http://community.opscode.com/api/v1"cookbook  "ntp"cookbook  "timezone",  "0.0.1"cookbook  "rvm",    :git  =>  "https://github.com/fnichol/chef-­‐rvm",    :ref  =>  "v0.7.1"cookbook  "cloudera",    :path  =>  "vendor/cookbooks/cloudera-­‐cookbook"
  34. 34. CHEF METADATA
  35. 35. http://bitfieldconsulting.com/puppet-dependency-graphsDependency Graphshttps://github.com/miketheman/knife-role-spaghetti
  36. 36. http://wham.cloudsmith.comOnline Service
  37. 37. DEBUGGING
  38. 38. LOGGING WITHDIFFERENT LEVELSInfo only, no exceptionChef::Log.info(Some  useful  info)Fatal, raising exceptionChef::Log.fatal(Something  bad)
  39. 39. REPL - STYLE(READ EVAL PRINT)http://wiki.opscode.com/display/chef/Shefhttps://github.com/pry/pry
  40. 40. DEV ENVIRONMENT
  41. 41. Info & Credits• Mitchell Hashimoto & John Bender• http://vagrantup.com• Based onVirtualbox , Ruby• OS-Installers (not gem based)• Runs on Mac OS, Linux,Windows
  42. 42. http://www.hashicorp.com/
  43. 43. Vagrant 101
  44. 44. Step 1 : installation
  45. 45. The power of CLI$ vagrant box add base http://files.vagrantup.com/lucid32.box$ vagrant init$ vagrant up
  46. 46. Step 2 : base box$ gem install vagrant$ vagrant box add base http://files.vagrantup.com/lucid32.box$ vagrant init$ vagrant upDownloads predefinedVM - f.i. Ubuntu 10.04 (Lucid)and registers it as name ‘base’‘BASE’LUCID
  47. 47. Step 3 : configuration$ gem install vagrant$ vagrant box add base http://files.vagrantup.com/lucid32.box$ vagrant init$ vagrant upCreates a standard configuration file
  48. 48. VagrantfileVagrant::Config.run do |config|# EveryVagrant virtual environment requires a box to build off of.config.vm.box = "base"end‘BASE’LUCIDVagrantfile = Project fileLinks our project with the ‘Base’ box
  49. 49. Step 4 : boot$ gem install vagrant$ vagrant box add base http://files.vagrantup.com/lucid32.box$ vagrant init$ vagrant upExtracts the ‘basebox’Creates a newVM inVirtualbox based on ‘basebox’Boots the newVM
  50. 50. Ready to rock SSHKA-CHING
  51. 51. I ♥ Vagrant
  52. 52. Reduces Setup Time1 2 3 4No-VMSSHReady♥Automated
  53. 53. Simplicity♥$ gem install vagrant$ vagrant box add base http://files.vagrantup.com/lucid32.box$ vagrant init$ vagrant upVBoxManage guestcontrol exec[ute] <vmname>|<uuid><path to program>--username <name> --password <password>[--arguments "<arguments>"][--environment "<NAME>=<VALUE> [<NAME>=<VALUE>]"][--flags <flags>] [--timeout <msec>][--verbose] [--wait-for exit,stdout,stderr||]copyto|cp <vmname>|<uuid><source on host> <destination on guest>--username <name> --password <password>[--dryrun] [--follow] [--recursive] [--verbose]createdir[ectory]|mkdir|md <vmname>|<uuid><directory to create on guest>--username <name> --password <password>[--parents] [--mode <mode>] [--verbose]updateadditions <vmname>|<uuid>[--source <guest additions .ISO>] [--verbose]
  54. 54. Self-Servicing♥Basebox“PULL”
  55. 55. Consistency♥BaseboxDEVELOPER1 DEVELOPER2
  56. 56. Repeatability♥BaseboxUP DESTROY UP DESTROYt1 t2 t3 t4
  57. 57. Contained♥Project 1 Project 2 Project 3Laptop
  58. 58. Vagrant 201
  59. 59. FileSharing/home/patrick/data/dataconfig.vm.share_folder "datastore", "/data", "./data"
  60. 60. Port Forwarding808080# Forward a port from the guest to the host, which allows for outside# computers to access theVM, whereas host only networking does not.config.vm.forward_port "http", 80, 9000config.vm.forward_port "redis", 6379, 6379config.vm.forward_port "mysql", 3306, 3306
  61. 61. Non-Intrusive♥Project 1 Project 2 Project 3LaptopMy Editor My IDE My Browser
  62. 62. MultiVM Stack♥DB DB DBProject 1 Project 2 Project 3APP APP APPWEB WEB WEBconfig.vm.define :web do |web_config|web_config.vm.box = "ubuntu"...endconfig.vm.define :app do |app_config|app_config.vm.box = "ubuntu"...endconfig.vm.define :db do |db_config|db_config.vm.box = "ubuntu"...end
  63. 63. Vagrant 301
  64. 64. BaseboxProvisionJust EnoughOperating SystemProvisionerchef/puppet/script+Cookbooks/ManifestsVM
  65. 65. PackageBaseboxJust EnoughOperating SystemProvisionerchef/puppet/script+Cookbooks/ManifestsVMBasebox
  66. 66. Provision Configconfig.vm.provision :chef_solo do |chef|chef.cookbooks_path = ["chefrepo/cookbooks"]chef.add_recipe("app_rails")chef.json.merge!({:apache => {:listen_ports => [ "8080"]})endChefSoloChefServerPuppetstandalonePuppetServerconfig.vm.provision :puppet do |puppet|puppet.pp_path = "/tmp/vagrant-puppet"puppet.manifests_path = "./manifests"puppet.module_path = "./modules"puppet.manifest_file = "newbox.pp"end
  67. 67. Veeweehttps://github.com/jedi4ever/veeweeBaseboxJust EnoughOperating SystemProvisionerchef/puppet/scriptISOKickstart/Preseed/Autoyast/AutoUnattendUbuntu, Debian, Centos,Fedora, Gentoo,Solaris, Suse,Archlinux,Windows
  68. 68. JEOS CREATIONJust Enough Operating System+ Config ManagementToolYour LaptopvirtualizationVirtualbox , FusionKVM, ParallelsLinux & Windowshttps://github.com/jedi4ever/veewee/Centos, Ubuntu, Debian, Solaris,Gentoo, Suse, Oracle Linux, ....And windows now too!
  69. 69. SERVICEFILE2FILE1PACKAGEVirtual Machine (1)Just Enough Operating System+ Config ManagementToolYour Laptopvirtualizationhttp://vagrantup.com/Virtualbox + Fusion + EC2Linux & Windows
  70. 70. Self-Executable Boxes• java -jar mygreatbox.jarhttp://www.jedi.be/blog/2011/03/31/installable-vagrant-boxes/https://github.com/maestrodev/wanton“proof of concept”
  71. 71. https://github.com/opscode/bentouses #veeweeCREATING BASEBOX‘EN MASS’https://github.com/atkm/seisan-linehttps://github.com/atkm/origami
  72. 72. http://www.vagrantbox.es/http://www.morethanseven.net/2011/05/08/Vagrant-plugin-for-interacting-with-vagrantboxes/http://vagrantfil.es/http://www.nodescription.net/ideation-into-the-void-vagrant-plugin-for-vagSHAREYOUR SETUP
  73. 73. CLIVagrantbox.es>> vagrant vagrantboxes search centos3 centos 5.5 http://dl.dropbox.com/u/15307300/vagrant-0.7-centos-64-base.box6 opscode centos 5 http://opscode-vagrant-boxes.s3.amazonaws.com/centos5-gems.box7 opscode ubuntu 10.04 http://opscode-vagrant-boxes.s3.amazonaws.com/ubuntu10.04-gems.b9 puppet centos 5.5 64 http://puppetlabs.s3.amazonaws.com/pub/centos5_64.box10 puppet centos 4 64 http://puppetlabs.s3.amazonaws.com/pub/centos4_64.box21 centos 5.6 32 http://yum.mnxsolutions.com/vagrant/centos_56_32.boxhttps://github.com/garethr/ruby-vagrantboxes
  74. 74. http://vmg.slynett.com/
  75. 75. http://puphpet.com/
  76. 76. http://vmg.slynett.com/
  77. 77. Version ControlledBaseboxJust EnoughOperating SystemProvisionerchef/puppet/scriptISOKickstart/Preseed/Autoyast/AutoUnattendVM♥Git Repo Git RepoCookbooks/ManifestsVagrantfileGit Repo
  78. 78. Complex SetupsVM♥Cookbooks/ManifestsRails Nodejs FlumeHadoop EsperMysql HDFSPostgres Java
  79. 79. Sharing on GithubKickstart/Preseed/Autoyast/AutoUnattend♥Git Repo Git RepoCookbooks/ManifestsVagrantfileGit RepoVeewee Chef/Puppet Vagrant
  80. 80. Kickstart/Preseed/Autoyast/AutoUnattendGit Repo Git RepoCookbooks/ManifestsVagrantfileGit RepoGit RepoApplication CodeShared “codebase”♥
  81. 81. Kickstart/Preseed/Autoyast/AutoUnattendGit RepoGit RepoCookbooks/ManifestsVagrantfileGit RepoGit RepoApplication CodeCo-Ownership♥DEV & OPSpairing
  82. 82. More Ops into Dev• can we speed up this provisioning?• we need a migration script from vX -> vY• we need start/stop scripts• we need consistent data restore♥
  83. 83. SynchronizedRelease Cycle♥Application vXSystem vXApplication vX+1System vX+1
  84. 84. Tools/Config Consistency♥Cookbooks/ManifestsCookbooks/ManifestsDEV PRODProvisionerchef/puppet/scriptProvisionerchef/puppet/scriptKickstart/Preseed/Autoyast/AutoUnattendKickstart/Preseed/Autoyast/AutoUnattend
  85. 85. Disposable Test Environments♥Cookbooks/ManifestsCookbooks/ManifestsDEV PRODProvisionerchef/puppet/scriptProvisionerchef/puppet/scriptKickstart/Preseed/Autoyast/AutoUnattendKickstart/Preseed/Autoyast/AutoUnattendTEST/QAhttp://paperairoplane.net/?p=240Cookbooks/ManifestsProvisionerchef/puppet/scriptKickstart/Preseed/Autoyast/AutoUnattend
  86. 86. Reuse WorkflowVagranthttps://github.com/geemus/fog♥
  87. 87. https://github.com/BIAINC/vagrant-windowshttps://github.com/dotless-de/vagrant-vbguesthttps://github.com/folken-laeneck/vagrant-bindfshttps://github.com/BerlinVagrant/vagrant-dnshttps://github.com/dwt/vagrant-hostsUSEFULVAGRANT PLUGINS
  88. 88. http://marketplace.eclipse.org/node/519961#.UKISB-Oe-d4VAGRANTECLIPSE INTEGRATION
  89. 89. http://www.jetbrains.com/ruby/webhelp/configuring-remote-interpreters-via-virtual-boxes.html
  90. 90. VAGRANT - CLOJUREhttps://github.com/tbatchelli/vmfest
  91. 91. VAGRANT - JAVAhttps://github.com/guigarage/vagrant-binding
  92. 92. UNITTESTING
  93. 93. https://github.com/nistude/cucumber-puppet
  94. 94. https://github.com/rodjek/rspec-puppethttps://github.com/calavera/rspec-chefhttps://github.com/acrmp/chefspec
  95. 95. https://github.com/sstephenson/bats
  96. 96. DON’TTESTYOUR CONFIGMANAGEMENTTESTYOUR LOGICNOTWas thepackageinstalledYESIf (OS=’X’ ) and (fileexists) ...did it give an error?
  97. 97. FAKETHE FACTS
  98. 98. https://github.com/customink/fauxhai
  99. 99. AUTO-TRIGGERTESTS RUNhttps://github.com/guard/guard-chefhttps://github.com/alister/guard-puppet-linthttps://github.com/johnbintz/guard-puppethttps://github.com/guard/guard-rspecCODEEDITAUTO RUNFEEDBACK
  100. 100. CI INTEGRATION
  101. 101. https://github.com/nicksieger/ci_reporterJunit processor for Ruby
  102. 102. SYNTAX CHECKSCOMPILE CHECKSUNITTESTSSMOKETESTSNO VMNEEDEDREAL VM
  103. 103. SMOKETESTING(ON REAL MACHINE)CodeAppliedCLEANVMSTATE A STATE BCode Run RunTestsCreateVMSNAPSHOT
  104. 104. http://auxesis.github.com/cucumber-nagios/
  105. 105. RE-USABLE CUCUMBER STEPShttps://github.com/hedgehog/cukenhttps://github.com/cucumber/arubahttp://www.cucumber-chef.org/
  106. 106. https://github.com/calavera/minitest-chef-handlerhttps://github.com/jedi4ever/puppet-assertValidate “inside” the provision-run cycle
  107. 107. VM CREATION IS EXPENSIVEhttps://github.com/jedi4ever/saharahttps://github.com/t9md/vagrant-snapCodeAppliedCLEANVMSTATE A STATE BCode Run RunTestsCreateVMSNAPSHOTROLLBACK
  108. 108. https://github.com/opscode/test-kitchen
  109. 109. USEFULVAGRANT PLUGINShttps://github.com/riotgames/vagrant-berkshelfhttps://github.com/schisamo/vagrant-omnibus
  110. 110. https://github.com/rtyler/vagrant-plugin
  111. 111. https://travis-ci.org/Uses vagrant & veewee
  112. 112. TARGETEDTESTINGIMPACT OF CHANGEhttps://github.com/jedi4ever/puppet-cichttps://github.com/jonlives/knife-preflightROLE1ROLE2ROLENMODULEAMODULEBMODULECMODULE...RealVM testingis expensiveOnly test what’s needed
  113. 113. PERFORMANCE/METRICS
  114. 114. https://github.com/rodjek/puppet-profiler
  115. 115. https://github.com/joemiller/puppet-graphite_eventhttps://github.com/imeyer/chef-handler-graphitehttps://github.com/krux/puppet-module-graphite-reportexecution timenumber of classes
  116. 116. “It works on mymachine”♥
  117. 117. For Devs(Test Code)
  118. 118. For Ops(Test Manifests)
  119. 119. For QA(Explore)
  120. 120. For Sales(To show off)
  121. 121. Bridgingfour worldswithout changing!♥http://www.jedi.be/blog/2011/03/28/using-vagrant-as-a-team/
  122. 122. http://foodfightshow.org/https://twitter.com/hangops
  123. 123. QUESTIONS?
  124. 124. OLDER OR RELATED STUFF
  125. 125. https://github.com/tknerr/bills-kitchenWindows
  126. 126. LOCALVMTESTINGhttp://www.cucumber-chef.org/https://github.com/exceedhl/toftFaster testing via LXCspinning up a vm is expensiveunless in containerEC2+LXCVagrant+LXC
  127. 127. http://wiki.opscode.com/display/chef/Whyrun+Testing
  128. 128. https://github.com/lak/puppet-memcached
  129. 129. https://github.com/lak/puppet-static-compiler
  130. 130. https://github.com/rtyler/blimpyhttps://github.com/jedi4ever/mccloud/Vagrant ‘clones’EC2, KVM, FUSION,HOSTS,VagrantEC2https://github.com/neerolyte/vagueanthttps://github.com/lstoll/tenderloin FUSIONLXC
  131. 131. http://puppetlabs.com/blog/what-is-user-experience-in-puppet/
  132. 132. AUDITINGhttp://www.normation.com/en/solutions/rudderhttp://www.fusioninventory.org/
  133. 133. http://comodit.github.com/synapse-agent/
  134. 134. http://saltstack.org/
  135. 135. https://github.com/tobami/littlechef
  136. 136. https://github.com/dwt/vagrant-hosts
  137. 137. https://github.com/opscode/omnibus
  138. 138. https://github.com/puppetlabs/puppet-module-tool
  139. 139. http://wiki.opscode.com/display/chef/Recipes#Recipes-ExceptionsandLogging
  140. 140. http://projects.puppetlabs.com/projects/puppet/wiki/Anchor_Pattern
  141. 141. VENDORING
  142. 142. https://github.com/railsmachine/rump
  143. 143. http://www.example42.com/?q=Puppi_A_Puppet_module_for_Deployment_Automation
  144. 144. http://rundeck.org/
  145. 145. http://www.sonian.com/cloud-monitoring-sensu/
  146. 146. http://docs.puppetlabs.com/guides/parameterized_classes.html
  147. 147. https://github.com/rerun/rerun
  148. 148. AUTHENTICATION
  149. 149. DASHBOARDS
  150. 150. http://puppetlabs.com/puppet/related-projects/dashboard/
  151. 151. http://blog.mornati.net/2012/01/29/kermit-a-webui-for-mcollective/
  152. 152. http://www.opscode.com/hosted-chef/
  153. 153. http://theforeman.org/
  154. 154. https://github.com/chapmanb/cloudbiolinux
  155. 155. https://github.com/aetherical/nimblestratus
  156. 156. https://github.com/fiddyspence/puppet-mconotify
  157. 157. https://github.com/ankurcha/stemcell

×