Oscar: Rapid Iteration with Vagrant and Puppet Enterprise - PuppetConf 2013


Published on

"Oscar: Rapid Iteration with Vagrant and Puppet Enterprise" by Adrien Thebo, Software Engineer, Puppet Labs.

Presentation Overview: When trying to debug software problems it's critical to be able to reproduce the original situation, and Puppet Enterprise is no exception to this. The Puppet Labs support team needs a way to rapidly reproduce customer issues across a wide range of operating systems and various versions of Puppet Enterprise. Oscar is a set of Vagrant plugins that handles machine provisioning and configuration to install Puppet Enterprise. It's designed to make building a Puppet Enterprise as simple as running `vagrant up`. While Oscar was originally built for supporting Puppet Enterprise, it provides a general platform for developing and testing against Puppet Enterprise. This talk will go over the history of Oscar, the current state, how it's used, and where to get it.

Speaker Bio: Adrien Thebo has been in the Operations/Software development field since 2005, starting at small IT shop in Boise, Idaho. He started at Puppet Labs in 2011 on the Operations team, and used Puppet to run the Puppet Labs infrastructure. In 2013 he transferred to the Community platform team, working with contributors to merge their contributions in Puppet Core. He develops and maintains a number of Puppet modules and tools around Puppet, and when he's not writing code for Puppet then he's probably blogging about it.

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Oscar: Rapid Iteration with Vagrant and Puppet Enterprise - PuppetConf 2013

  1. 1. Oscar Rapid Iteration with Vagrant and Puppet Enterprise Adrien Thebo | Puppet Labs @nullfinch | Freenode: finch
  2. 2. Hi.
  3. 3. Introductions n’ stuff On/Off Operations/Development, ~8 years Devops before it was cool
  4. 4. Introductions n’ stuff Puppet Labs, 2.5 years Operations Engineer, 2 years Community Software Engineer, 6 months
  5. 5. Things I do
  6. 6. r10k Better Puppet deployment, powered by robots
  7. 7. Puppet modules puppet-network puppet-portage puppet-filemapper
  8. 8. So how about that Vagrant?
  9. 9. Mandatory audience fun time! Who has… heard of Vagrant? tried Vagrant? used Vagrant regularly? developed on Vagrant or plugins? created Vagrant?
  10. 10. A brief introduction to Vagrant
  11. 11. Good idea, bad idea Good idea: hand crafted, one of a kind artisan furniture Bad idea: hand crafted, one of a kind artisan VMs
  12. 12. Before Vagrant Hand rolled VMs No automation Unreproducible Full of forgotten hacks
  13. 13. Vagrant is… Local VM infrastructure without the pain Easy to configure Reproducible Portable
  14. 14. Vagrant Workflow generate provision use/break discard It’s your very own cloud!
  15. 15. Laying the Groundwork Operations @ Puppet Labs … Support @ Puppet Labs? Infrastructure & client support
  16. 16. The problem space
  17. 17. Supporting PE 2 Need to reproduce customer problems Various supported platforms Various node configurations
  18. 18. Supporting PE 2 Speed/reproducibility paramount Have to meet SLA Also handle other operations work
  19. 19. Support ♥ Vagrant Receive ticket Duplicate client environment Build environment Reproduce & debug Help customer^W^Wprofit!!!
  20. 20. Limitations If this presentation was all roses and butterflies Then it would be very boring.
  21. 21. Choose your own adventure Like any tool, Vagrant makes tradeoffs Some use cases are easier than others
  22. 22. Vagrant excels at… Static/Stable environments Configure VM definition Set up provisioning git commit -m 'Add Vagrantfile'
  23. 23. Vagrant struggles with… Highly mutable environments Complex provisioning steps Configuration reuse Partial configuration Distributable configuration
  24. 24. The Vagrantfile DSL Pro - all the power of Ruby Con - all the power of Ruby
  25. 25. A descent into madness (I’m sorry.) DONT. DO. THIS. #vi:setft=ruby: require'yaml' begin #Loadconfig yaml_config=File.join(File.dirname(__FILE__),"config.yaml") config=nil File.open(yaml_config)do|fd| config=YAML.load(fd.read) end nodes =config["nodes"] profiles =config["profiles"] nodes.eachdo|node| #SetdefaultPEconfiguration,andallownodeoverridingofthesevalues defaults={"pe"=>config['pe']} node.merge!(defaults)do|key,oldval,newval| ifoldval.is_a?Hash newval.mergeoldval else warn"Triedtomergehashvalueswith#{key}=>[#{oldval},#{newval}],butwasnotahash.Using#{oldval}." oldval end end profile =node["profile"] node.merge!profiles[profile] end rescue=>e puts"Malformedormissingconfig.yaml:#{e}" putse.backtrace exit!(1) end #Thisisanextensionofthecommonnodedefinition,asitmakesprovisions #forcustomizingthemasterformoreseamlessinteractionwiththebase #system defprovision_master(config,node,attributes) #Manifestsandmodulesneedtobemountedonthemasterviasharedfolders, #butthedefault/vagrantmounthaspermissionsandownershipthatconflicts #withthemasterandpe-puppet.Wemountthesefoldersseparatelywith #loosenedpermissions. config.vm.share_folder'manifests','/manifests','./manifests',:extra=>'fmode=644,dmode=755,fmask=022,dmask=022' config.vm.share_folder'modules','/modules','./modules', :extra=>'fmode=644,dmode=755,fmask=022,dmask=022' #Updatepuppet.conftoaddthemanifestdirdirectivetopointtothe #/manifestsmount,ifthedirectiveisn'talreadypresent. node.vm.provision:shelldo|shell| shell.inline=<<-EOT sed-i' 2{ /manifest/!i manifestdir=/manifests } '/etc/puppetlabs/puppet/puppet.conf EOT end #Updatepuppet.conftoaddthemodulepathdirectivetopointtothe #/modulemount,ifithasn'talreadybeenset. node.vm.provision:shelldo|shell| shell.inline=<<-EOT sed-i' /modulepath/{ /vagrant/!s,$,:/modules, } '/etc/puppetlabs/puppet/puppet.conf EOT end #Rewritetheoldesite.ppconfigsinceit'snotused,andwarnpeople #aboutthis. node.vm.provision:shelldo|shell| shell.inline=%{echo"#/etc/puppetlabs/puppet/manifestsisnotused;see/manifests.">/etc/puppetlabs/puppet/manifests/site.pp} end #BoostRAMforthemastersothatactivemqdoesn'tasplode node.vm.customize(["modifyvm",:id,"--memory","1024"]) #Enableautosigningonthemaster node.vm.provision:shelldo|shell| shell.inline=%{echo'*'>/etc/puppetlabs/puppet/autosign.conf} end end #Addsthevagrantconfigurationtweaks defconfigure_node(config,node,attributes) node.vm.box=attributes["boxname"] #Applyallspecifiedportforwards attributes["forwards"].eachdo|(src,dest)| node.vm.forward_portsrc,dest endifattributes["forwards"]#<--Iamamonster #Addinoptionalper-nodeconfiguration node.vm.box_url=attributes["boxurl"]ifattributes["boxurl"] node.vm.network:hostonly,attributes["address"]ifattributes["address"] node.vm.boot_mode=attributes[:gui]ifattributes[:gui] end #Provideprovisioningdetailsforthisnode defprovision_node(config,node,attributes) #HackinfauxDNS #PuppetenterpriserequiressomethingresemblingfunctioningDNStobe #installedcorrectly attributes["hosts_entries"].eachdo|entry| node.vm.provision:shelldo|shell| shell.inline=%{grep"#{entry}"/etc/hosts||echo"#{entry}">>/etc/hosts} end end #Setthemachinehostname node.vm.provision:shelldo|shell| shell.inline=%{hostname#{attributes["name"]}} end node.vm.provision:shelldo|shell| shell.inline=%{domainnamepuppetlabs.test} end end definstall_pe(config,node,attributes) #Customizetheanswersfileforeachnode node.vm.provision:shelldo|shell| shell.inline=%{sed-e's/%%CERTNAME%%/#{attributes["name"]}/'</vagrant/answers/#{attributes["role"]}.txt>/tmp/answers.txt} end #IfthePEversionis0.0.0,bypasspuppetinstallation. #Thiscouldeasilymakelaterprovisioningfail. ifattributes['pe']['version'].match%r[^0.0] warn"RequestedPEversionwassetto#{attributes['pe']['version']};willnotautomaticallyinstallPE" return end #Assembletheinstallercommand fragments=[] fragments<<"2>&1" fragments<<attributes['pe']['installer']['executable'] fragments<<'-a/tmp/answers.txt' fragments<<attributes['pe']['installer']['args'].join('') installer_cmd=fragments.join('').gsub(':version',attributes['pe']['version']) #InstallPuppetEnterpriseifithasn'tbeeninstalledyet.Ifthemachine #isrebootedthenthisprovisioningstepwillbeskipped. node.vm.provision:shelldo|shell| shell.inline=<<-EOT if![-f/opt/pe_version];then #{installer_cmd} fi EOT end end Vagrant::Config.rundo|config| #GeneratealistofnodeswithstaticIPaddresses hosts_entries=nodes.select{|h|h["address"]}.map{|h|%{#{h["address"]}#{h["name"]}}} #TweakeachhostforPuppetEnterprise,andtheninstallPEitself. nodes.eachdo|attributes| config.vm.defineattributes["name"]do|node| attributes["hosts_entries"]=hosts_entries configure_node(config,node,attributes) provision_node(config,node,attributes) install_pe(config,node,attributes) ifattributes["role"].match/master/ provision_master(config,node,attributes) end end end end
  26. 26. Vagrant and networking No default internal networking Flaky DHCP DNS resolution
  27. 27. Vagrant and networking Solutions! Run your own infrastructure! DHCPD BIND
  28. 28. The reality Can’t support every configuration Shouldn’t support every configuration Batteries not included
  29. 29. Introducing Oscar
  30. 30. Oscar in a nutshell Set of Vagrant plugins Built for mutable environments
  31. 31. An overview vagrant-hosts vagrant-auto_network vagrant-pe_build vagrant-config_builder oscar
  32. 32. Vagrant hosts DNS record types: starting with A: A, AAAA, AFSDB, APL starting with C: CAA, CERT, CNAME starting with D: DHCID, DLV, DNAME, DNSKEY, DS … We need: name -> ip address
  33. 33. Vagrant hosts Inside of a transitory environment Query private network addresses -> /etc/hosts Go do something you care about Or manage BIND on $platform
  34. 34. Vagrant Auto-network Have to add extra interfaces f$&k it. STATIC IP ADDRESSES FOR ALL config.vm.network :private_network, :auto_network => true
  35. 35. Vagrant PE Build PE configuration optimized for Vagrant Download installers on demand
  36. 36. Vagrant config builder Configuration as data
  37. 37. Before config builder Vagrantfile config.vm.define :puppetmaster do |box| flavor = :centos_6 set_box box, S3_BOXES[:centos_64_nocm] # NOTE: Hostonly _may_ mean no internets if this adapter gets found first!!! # Check /etc/resolv.conf ! box.vm.network :hostonly, "" # NOTE: Share folders, such as Git checkouts of the Puppet source code share_puppet_source box box.vm.provision :shell, :inline => BOOTSTRAPS[flavor] provision_box box, 'server.pp' end
  38. 38. After config builder roles.yaml --- roles: puppetmaster: private_networks: - {auto_network: true} synced_folders: - {host_path: ~/Source/puppetlabs, guest_path: /puppetlabs} provisioners: - {type: hosts} - {type: pe_bootstrap, role: master} puppetagent: private_networks: - {auto_network: true} provisioners: - {type: hosts} - {type: pe_bootstrap}
  39. 39. After config builder vms.yaml --- vms: - name: master box: centos_64_nocom roles: puppetmaster - name: agent box: debian_64_nocm roles: puppetagent
  40. 40. Pick your data source YAML JSON XML interpretive dance
  41. 41. What does Oscar do? Dependencies! Everything is a standalone plugin Mix and match
  42. 42. What does Oscar do? Templates and defaults Sane defaults to get you started
  43. 43. PE stack in a box Configure your development environment like production Develop your modules in complete isolation Simulate app deployments before going live Pre-production in a box! Stable Puppet environment
  44. 44. What Oscar gets you All the perks of Vagrant Minimal user setup Complex config made easy
  45. 45. What Oscar gets you Goal - from zero to PE in vagrant up
  46. 46. Who uses it? Commercial Support Open Source Support Sales Engineering People… and stuff…
  47. 47. Demo time! Basic spin up Sales eng env
  48. 48. Questions?
  49. 49. Try it yourself Code on Github Under active development contributions accepted!
  50. 50. Thanks Tom Linkin! Chris Barker! Charlie Sharpsteen! Hunter Haugen! Adam Crews! The fabulous Puppet and Vagrant community! Exclamation points! Many others!
  51. 51. Credits Layout by Nanoc Presentation by Reveal.js Consciousness by caffeine