Islands: Puppet at Bulletproof Networks

1,978 views

Published on

Bulletproof Networks provides managed hosting services to some of the largest companies in Australia. Bulletproof implements strong isolation of customer environments, and this can present unique challenges when re-using Puppet code across our customer base. Additionally, the environments range in size from small to very large, and our tools + processes need to be able to handle both uses cases equally well.

In this talk Lindsay + Mick will cover how Bulletproof's approach to these problems has evolved over the last 4 years, and some of the tools Bulletproof has developed and built upon to provide an awesome service to our customers.

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,978
On SlideShare
0
From Embeds
0
Number of Embeds
14
Actions
Shares
0
Downloads
26
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Islands: Puppet at Bulletproof Networks

  1. 1. Islands
  2. 2. Who arethese guys?
  3. 3. Mick Pollard @aussielunix &Lindsay Holmwood @auxesis
  4. 4. Puppet usersfor > 5 years
  5. 5. BULLETPROOF
  6. 6. IaaS &ManagedServices
  7. 7. Movember Australian Museum Rebel Group Blackmores Angus & Robertson Telstra Perisher BlueScope Steel Woolworths DMG Radio Clive Peters Deloitte Clemenger budget.gov.au Nissan AOC Nova Sydney Airports Whirlpool Smooth Theiss Borders Fosters Country Road Midas Australian Geographic SensisTourism Victoria ABS AusPost DET FWA Vodafone
  8. 8. Using Puppet since 2008
  9. 9. Uniquechallenges
  10. 10. Strong isolation
  11. 11. Networksegregationwith VLANs
  12. 12. CentralPuppetmasterisnʼt an option
  13. 13. Thoroughchange control
  14. 14. Rapid growth
  15. 15. How do weuse Puppet?
  16. 16. Standalone systems(puppetmaster-less)
  17. 17. Internalinfrastructure
  18. 18. Full customerenvironments
  19. 19. Standalone systems(puppetmaster-less)
  20. 20. Campaign drivenbusiness
  21. 21. budget.gov.au
  22. 22. movember.com
  23. 23. mamamia.com.au
  24. 24. Reverseproxies
  25. 25. Nginxwith customisation
  26. 26. Rump
  27. 27. More detail inJohn Ferlitoʼs talk at 14.00
  28. 28. Internalinfrastructure
  29. 29. Vanilla
  30. 30. OnePuppetmaster
  31. 31. Ubuntu(Lucid or Precise)
  32. 32. Full customerenvironments
  33. 33. Every customer has their own puppetmaster
  34. 34. “Islands of Puppet”
  35. 35. Copypasta
  36. 36. Configuration drift
  37. 37. apache
  38. 38. apache
  39. 39. apachecustomer-a
  40. 40. apachecustomer-a
  41. 41. apache customer-bcustomer-a
  42. 42. apache customer-bcustomer-a
  43. 43. apache customer-bcustomer-a customer-c
  44. 44. apache customer-bcustomer-a customer-c
  45. 45. apache customer-bcustomer-a customer-d customer-c
  46. 46. Poor code share
  47. 47. What if customersedit the code?
  48. 48. How do we maintaincommon code?
  49. 49. Commonalities
  50. 50. Mix of Puppetversions
  51. 51. 0.25(as provided by Ubuntu)
  52. 52. 2.7(as provided by Puppet Labs)
  53. 53. Mix ofOperatingSystems
  54. 54. lucid precise2.7 internal infrastructure some customers0.25 most customers
  55. 55. Passenger > webrick
  56. 56. --no-daemonize
  57. 57. Default behaviour is orthogonal tochange control
  58. 58. We dont wantsystems to change without control
  59. 59. All changesinitiated byan engineer
  60. 60. nodes + roles
  61. 61. node stlyqy-lvs02.cust.bulletproof.net { server { $fqdn: } include snmp::server::lvs include sysctl::lvs include keepalive::lvs include network::conntrack::modules include network::conntrack::hashsize include network::bonding::activebackup include network::type::bonded_vlan include ript}
  62. 62. node stlyqy-lvs02.cust.bulletproof.net { server { $fqdn: } include snmp::server::lvs include sysctl::lvs include keepalive::lvs include network::conntrack::modules include network::conntrack::hashsize include network::bonding::activebackup include network::type::bonded_vlan include ript}
  63. 63. define server($collectd_client_report_to=collectd.bulletproof.net) { include motd include augeas include apt include utils include puppet::client include ssh::server include ssh::authorized_keys include ntp::client include postfix::satellite include ruby::dev include ruby::rubygems include bzr::client include git::common include git::github include snmp::server include vmware::tools include apparmor::disable collectd::client { "${fqdn}": report_to => $collectd_client_report_to }}
  64. 64. app_serverdatabase_serverfile_servermanagement_servermemcache_servermonitor_serverproxy_serverpuppetmaster_serverredis_serversingle_serversphinx_serverstatic_server
  65. 65. Heirais the future
  66. 66. We useCapistrano
  67. 67. sshin-a-parallel-for-loop
  68. 68. Why cap andnot mcollective?
  69. 69. We deployeverything with cap
  70. 70. Monitoring configuration Firewall configuration Web applications Internal tools
  71. 71. Consistentdeployment toolacross all projects
  72. 72. Principle ofleast surprise
  73. 73. Engineers learn 1 tool
  74. 74. Puppet is no different to therest of our stack
  75. 75. How do we usecap + Puppet?
  76. 76. Puppet changes
  77. 77. cap puppet:go ROLES=lvs options="--noop"cap puppet:go ROLES=lvs
  78. 78. Smoke tests
  79. 79. $ cap puppet:go options="--noop"# ...infmon hosts serves a Nagios page over httpsFinished in 1.67 seconds1 example, 0 failures
  80. 80. Works out all rolesthat hosts within a run belong to
  81. 81. Runs tests tagged with those roles
  82. 82. Fast feedback onchange success
  83. 83. Bootstrapping
  84. 84. cap node:bootstrap HOSTFILTER=lvs-08.bp.net
  85. 85. Takes VM inunknown state
  86. 86. Brings intoknown state forPuppet 2.7 run
  87. 87. Limitations
  88. 88. Singling outhosts is tricky
  89. 89. Re-using data across commandsrequires... creativity
  90. 90. servers = []run "mysql -e "SHOW MASTER STATUS;" | tail -n 1" do |channel, type, data| hostname = channel[:host] filename = data.split(/s+/).first position = data.split(/s+/).last servers << { :hostname => hostname, :filename => filename, :position => position }end
  91. 91. ROLES andHOSTFILTER
  92. 92. TLDR;
  93. 93. There areedge cases
  94. 94. It does the job
  95. 95. How have wetried to solve them?
  96. 96. First iteration
  97. 97. Modules
  98. 98. modules/apache/ | manifests/init.pp | files/ | templates/ | lib/ | README.markdown
  99. 99. Stored on GitHub
  100. 100. Drink from the firehose
  101. 101. puppet-module-tool
  102. 102. github.compuppetlabs/puppet-module-tool
  103. 103. gem install puppet-module
  104. 104. Modulefile
  105. 105. puppet-module build
  106. 106. Turns
  107. 107. modules/apache/ | manifests/init.pp | files/ | templates/ | lib/ | README.markdown
  108. 108. into
  109. 109. bulletproofnetworks-apache-1.3.0.tar.gz
  110. 110. Puppet forge
  111. 111. Publicby default
  112. 112. Not great if you dontwant to open sourceall your secret sauce
  113. 113. But puppet-module-tool isinteresting...
  114. 114. ...can we fake the forge?
  115. 115. Pain points
  116. 116. Arduous releaseworkflow
  117. 117. Better suitedfor infrequent changes
  118. 118. High barrier ofentry for customers to submit patches
  119. 119. Sharing bugfixes & improvementsrequires significant refactoring
  120. 120. Limitedreporting oncustomer lag
  121. 121. Second iteration
  122. 122. Bundler
  123. 123. Gemfile
  124. 124. #!/usr/bin/env rubysource :rubygemsgem capistrano, 2.9.0gem capistrano-ext, 1.2.1gem colorizegem puppet, 2.7.13gem puppet-modulegroup :test do gem rspec, 2.8.0 gem mechanize gem puppet-lintend
  125. 125. Can wereuse it?
  126. 126. Tim Sharpe @ GitHub(@rodjek)
  127. 127. Messy working prototype
  128. 128. pre-alpha quality
  129. 129. More research required
  130. 130. Librarian
  131. 131. “A framework for bundlers”
  132. 132. librarian-chefmanages Chef repositories
  133. 133. by Jay Feldblum github.comapplicationsonline/librarian
  134. 134. librarian-puppet
  135. 135. “You can all stop using git submodules now”
  136. 136. gem install librarian-puppet --pre
  137. 137. github.comrodjek/librarian-puppet
  138. 138. Bundler-like behavior for Puppet
  139. 139. Puppetfile
  140. 140. #!/usr/bin/env rubyforge "http://forge.puppetlabs.com"mod "puppetlabs/razor"mod "puppetlabs/ntp", "0.0.3"
  141. 141. #!/usr/bin/env rubyforge "http://forge.puppetlabs.com"mod "puppetlabs/razor"mod "puppetlabs/ntp", "0.0.3"mod "stdlib", :git => "git://github.com/puppetlabs/puppetlabs-stdlib.git"
  142. 142. #!/usr/bin/env rubyforge "http://forge.puppetlabs.com"mod "puppetlabs/razor"mod "puppetlabs/ntp", "0.0.3"mod "stdlib", :git => "git://github.com/puppetlabs/puppetlabs-stdlib.git"mod "apt", :git => "git://github.com/puppetlabs/puppetlabs-apt.git" :ref => feature/master/dans_refactor
  143. 143. One canonical module
  144. 144. modules/ in.gitignore
  145. 145. librarian-puppet outdated tells you what needs to be updated
  146. 146. librarian-puppet updateupdates your modules
  147. 147. Demo!http://aussielunix.github.com/jenkins-appliance/
  148. 148. Policy
  149. 149. Always use a modules master
  150. 150. Make changesto modules as usual
  151. 151. git commit && git push the module
  152. 152. Make modulechanges generic by default
  153. 153. Only brancha module when:
  154. 154. 1. something is super client specific
  155. 155. 2. there is anunmissable deadline
  156. 156. 3. testing ideas
  157. 157. Set a reminder tomerge changes into master
  158. 158. Use pull requests on GitHub fordangerous changes
  159. 159. Take aways:
  160. 160. Puppet modules are pretty neat
  161. 161. Keep feedback loops short
  162. 162. Code share is king
  163. 163. Do you have similar problems?
  164. 164. How do yousolve them?
  165. 165. Thank you!

×