Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
Load testing with Blitz
Load testing with Blitz
Loading in …3
×
1 of 191

Islands: Puppet at Bulletproof Networks

1

Share

Download to read offline

Bulletproof Networks provides managed hosting services to some of the largest companies in Australia. Bulletproof implements strong isolation of customer environments, and this can present unique challenges when re-using Puppet code across our customer base. Additionally, the environments range in size from small to very large, and our tools + processes need to be able to handle both uses cases equally well.

In this talk Lindsay + Mick will cover how Bulletproof's approach to these problems has evolved over the last 4 years, and some of the tools Bulletproof has developed and built upon to provide an awesome service to our customers.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Islands: Puppet at Bulletproof Networks

  1. 1. Islands
  2. 2. Who are these guys?
  3. 3. Mick Pollard @aussielunix & Lindsay Holmwood @auxesis
  4. 4. Puppet users for > 5 years
  5. 5. BULLETPROOF
  6. 6. IaaS & Managed Services
  7. 7. Movember Australian Museum Rebel Group Blackmores Angus & Robertson Telstra Perisher BlueScope Steel Woolworths DMG Radio Clive Peters Deloitte Clemenger budget.gov.au Nissan AOC Nova Sydney Airports Whirlpool Smooth Theiss Borders Fosters Country Road Midas Australian Geographic Sensis Tourism Victoria ABS AusPost DET FWA Vodafone
  8. 8. Using Puppet since 2008
  9. 9. Unique challenges
  10. 10. Strong isolation
  11. 11. Network segregation with VLANs
  12. 12. Central Puppetmaster isnʼt an option
  13. 13. Thorough change control
  14. 14. Rapid growth
  15. 15. How do we use Puppet?
  16. 16. Standalone systems (puppetmaster-less)
  17. 17. Internal infrastructure
  18. 18. Full customer environments
  19. 19. Standalone systems (puppetmaster-less)
  20. 20. Campaign driven business
  21. 21. budget.gov.au
  22. 22. movember.com
  23. 23. mamamia.com.au
  24. 24. Reverse proxies
  25. 25. Nginx with customisation
  26. 26. Rump
  27. 27. More detail in John Ferlitoʼs talk at 14.00
  28. 28. Internal infrastructure
  29. 29. Vanilla
  30. 30. One Puppetmaster
  31. 31. Ubuntu (Lucid or Precise)
  32. 32. Full customer environments
  33. 33. Every customer has their own puppetmaster
  34. 34. “Islands of Puppet”
  35. 35. Copypasta
  36. 36. Configuration drift
  37. 37. apache
  38. 38. apache
  39. 39. apache customer-a
  40. 40. apache customer-a
  41. 41. apache customer-b customer-a
  42. 42. apache customer-b customer-a
  43. 43. apache customer-b customer-a customer-c
  44. 44. apache customer-b customer-a customer-c
  45. 45. apache customer-b customer-a customer-d customer-c
  46. 46. Poor code share
  47. 47. What if customers edit the code?
  48. 48. How do we maintain common code?
  49. 49. Commonalities
  50. 50. Mix of Puppet versions
  51. 51. 0.25 (as provided by Ubuntu)
  52. 52. 2.7 (as provided by Puppet Labs)
  53. 53. Mix of Operating Systems
  54. 54. lucid precise 2.7 internal infrastructure some customers 0.25 most customers
  55. 55. Passenger > webrick
  56. 56. --no-daemonize
  57. 57. Default behaviour is orthogonal to change control
  58. 58. We don't want systems to change without control
  59. 59. All changes initiated by an engineer
  60. 60. nodes + roles
  61. 61. node 'stlyqy-lvs02.cust.bulletproof.net' { server { $fqdn: } include snmp::server::lvs include sysctl::lvs include keepalive::lvs include network::conntrack::modules include network::conntrack::hashsize include network::bonding::activebackup include network::type::bonded_vlan include ript }
  62. 62. node 'stlyqy-lvs02.cust.bulletproof.net' { server { $fqdn: } include snmp::server::lvs include sysctl::lvs include keepalive::lvs include network::conntrack::modules include network::conntrack::hashsize include network::bonding::activebackup include network::type::bonded_vlan include ript }
  63. 63. define server($collectd_client_report_to='collectd.bulletproof.net') { include motd include augeas include apt include utils include puppet::client include ssh::server include ssh::authorized_keys include ntp::client include postfix::satellite include ruby::dev include ruby::rubygems include bzr::client include git::common include git::github include snmp::server include vmware::tools include apparmor::disable collectd::client { "${fqdn}": report_to => $collectd_client_report_to } }
  64. 64. app_server database_server file_server management_server memcache_server monitor_server proxy_server puppetmaster_server redis_server single_server sphinx_server static_server
  65. 65. Heira is the future
  66. 66. We use Capistrano
  67. 67. ssh in-a-parallel-for-loop
  68. 68. Why cap and not mcollective?
  69. 69. We deploy everything with cap
  70. 70. Monitoring configuration Firewall configuration Web applications Internal tools
  71. 71. Consistent deployment tool across all projects
  72. 72. Principle of least surprise
  73. 73. Engineers learn 1 tool
  74. 74. Puppet is no different to the rest of our stack
  75. 75. How do we use cap + Puppet?
  76. 76. Puppet changes
  77. 77. cap puppet:go ROLES=lvs options="--noop" cap puppet:go ROLES=lvs
  78. 78. Smoke tests
  79. 79. $ cap puppet:go options="--noop" # ... infmon hosts serves a Nagios page over https Finished in 1.67 seconds 1 example, 0 failures
  80. 80. Works out all roles that hosts within a run belong to
  81. 81. Runs tests tagged with those roles
  82. 82. Fast feedback on change success
  83. 83. Bootstrapping
  84. 84. cap node:bootstrap HOSTFILTER=lvs-08.bp.net
  85. 85. Takes VM in unknown state
  86. 86. Brings into known state for Puppet 2.7 run
  87. 87. Limitations
  88. 88. Singling out hosts is tricky
  89. 89. Re-using data across commands requires... creativity
  90. 90. servers = [] run "mysql -e "SHOW MASTER STATUS;" | tail -n 1" do |channel, type, data| hostname = channel[:host] filename = data.split(/s+/).first position = data.split(/s+/).last servers << { :hostname => hostname, :filename => filename, :position => position } end
  91. 91. ROLES and HOSTFILTER
  92. 92. TLDR;
  93. 93. There are edge cases
  94. 94. It does the job
  95. 95. How have we tried to solve them?
  96. 96. First iteration
  97. 97. Modules
  98. 98. modules/apache/ | manifests/init.pp | files/ | templates/ | lib/ | README.markdown
  99. 99. Stored on GitHub
  100. 100. Drink from the firehose
  101. 101. puppet-module-tool
  102. 102. github.com puppetlabs/puppet-module-tool
  103. 103. gem install puppet-module
  104. 104. Modulefile
  105. 105. puppet-module build
  106. 106. Turns
  107. 107. modules/apache/ | manifests/init.pp | files/ | templates/ | lib/ | README.markdown
  108. 108. into
  109. 109. bulletproofnetworks-apache-1.3.0.tar.gz
  110. 110. Puppet forge
  111. 111. Public by default
  112. 112. Not great if you don't want to open source all your secret sauce
  113. 113. But puppet- module-tool is interesting...
  114. 114. ...can we fake the forge?
  115. 115. Pain points
  116. 116. Arduous release workflow
  117. 117. Better suited for infrequent changes
  118. 118. High barrier of entry for customers to submit patches
  119. 119. Sharing bugfixes & improvements requires significant refactoring
  120. 120. Limited reporting on customer lag
  121. 121. Second iteration
  122. 122. Bundler
  123. 123. Gemfile
  124. 124. #!/usr/bin/env ruby source :rubygems gem 'capistrano', '2.9.0' gem 'capistrano-ext', '1.2.1' gem 'colorize' gem 'puppet', '2.7.13' gem 'puppet-module' group :test do gem 'rspec', '2.8.0' gem 'mechanize' gem 'puppet-lint' end
  125. 125. Can we reuse it?
  126. 126. Tim Sharpe @ GitHub (@rodjek)
  127. 127. Messy working prototype
  128. 128. pre-alpha quality
  129. 129. More research required
  130. 130. Librarian
  131. 131. “A framework for bundlers”
  132. 132. librarian-chef manages Chef repositories
  133. 133. by Jay Feldblum github.com applicationsonline/librarian
  134. 134. librarian-puppet
  135. 135. “You can all stop using git submodules now”
  136. 136. gem install librarian-puppet --pre
  137. 137. github.com rodjek/librarian-puppet
  138. 138. Bundler-like behavior for Puppet
  139. 139. Puppetfile
  140. 140. #!/usr/bin/env ruby forge "http://forge.puppetlabs.com" mod "puppetlabs/razor" mod "puppetlabs/ntp", "0.0.3"
  141. 141. #!/usr/bin/env ruby forge "http://forge.puppetlabs.com" mod "puppetlabs/razor" mod "puppetlabs/ntp", "0.0.3" mod "stdlib", :git => "git://github.com/puppetlabs/puppetlabs-stdlib.git"
  142. 142. #!/usr/bin/env ruby forge "http://forge.puppetlabs.com" mod "puppetlabs/razor" mod "puppetlabs/ntp", "0.0.3" mod "stdlib", :git => "git://github.com/puppetlabs/puppetlabs-stdlib.git" mod "apt", :git => "git://github.com/puppetlabs/puppetlabs-apt.git" :ref => 'feature/master/dans_refactor'
  143. 143. One canonical module
  144. 144. modules/ in .gitignore
  145. 145. librarian-puppet outdated tells you what needs to be updated
  146. 146. librarian-puppet update updates your modules
  147. 147. Demo! http://aussielunix.github.com/jenkins-appliance/
  148. 148. Policy
  149. 149. Always use a module's master
  150. 150. Make changes to modules as usual
  151. 151. git commit && git push the module
  152. 152. Make module changes generic by default
  153. 153. Only branch a module when:
  154. 154. 1. something is super client specific
  155. 155. 2. there is an unmissable deadline
  156. 156. 3. testing ideas
  157. 157. Set a reminder to merge changes into master
  158. 158. Use pull requests on GitHub for dangerous changes
  159. 159. Take aways:
  160. 160. Puppet modules are pretty neat
  161. 161. Keep feedback loops short
  162. 162. Code share is king
  163. 163. Do you have similar problems?
  164. 164. How do you solve them?
  165. 165. Thank you!

×