Puppet at GitHub - PuppetConf 2013

3,849 views

Published on

"Puppet at GitHub" by Will Farrington Junior Executive Assistant to the Assistant VP of Operations, GitHub, Inc.

Presentation Overview: At GitHub, we've got a pretty large Puppet code base to manage all of the GitHub.com infrastructure. It's also pretty old; a little over four and a half years of history lives in our Puppet repo. We've also hired a lot more people over the past few years, going from around 70 people a year ago to more than double that now. This talk focuses on how we continue to embrace the GitHub principles of constantly shipping, iterative improvement, constant experimenting, and no managers to manage our rather large Puppet infrastructure with success.

Speaker Bio: Will is a developer and operations engineer. Will works on system operations at GitHub, where he spends most of his time slinging Puppet to manage GitHub's extensive production environment where he spends a great deal of time on automating, homogenizing, and improving GitHub's development environments. Prior to GitHub, he spent the previous 3 years using Ruby and Puppet to build and scale massive web services with Highgroove Studios and Rails Machine.

0 Comments
11 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,849
On SlideShare
0
From Embeds
0
Number of Embeds
1,034
Actions
Shares
0
Downloads
64
Comments
0
Likes
11
Embeds 0
No embeds

No notes for slide

Puppet at GitHub - PuppetConf 2013

  1. 1. Puppet at GitHub
  2. 2. @wfarr GitHub Operations Known Aliases: King of Kebabs The Chairman Mr. Caremad !
  3. 3. The State of Puppet at GitHub "
  4. 4. " The State of Puppet at GitHub github/boxen
  5. 5. " The State of Puppet at GitHub ~1.5 years old
  6. 6. " The State of Puppet at GitHub open-sourced ~7 months ago
  7. 7. " The State of Puppet at GitHub ~240 open-source puppet modules
  8. 8. " The State of Puppet at GitHub puppet 3.latest
  9. 9. " The State of Puppet at GitHub supports ruby 1.8.7, 1.9.3, 2.0.0
  10. 10. " The State of Puppet at GitHub hiera
  11. 11. " The State of Puppet at GitHub OS X
  12. 12. " The State of Puppet at GitHub Linux support in-progress
  13. 13. " The State of Puppet at GitHub # tomorrow @ 2:20pm
  14. 14. " The State of Puppet at GitHub github/puppet
  15. 15. " The State of Puppet at GitHub $
  16. 16. " The State of Puppet at GitHub ~5 years old
  17. 17. " The State of Puppet at GitHub 0.24.x — 2.7.x
  18. 18. " The State of Puppet at GitHub 121 modules
  19. 19. " The State of Puppet at GitHub ~280k lines of code
  20. 20. " The State of Puppet at GitHub every single employee has commit access
  21. 21. " The State of Puppet at GitHub 15.5k commits to master past 12 months
  22. 22. " The State of Puppet at GitHub by 86 contributors
  23. 23. " The State of Puppet at GitHub % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % %
  24. 24. " The State of Puppet at GitHub % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % 42.5%
  25. 25. " The State of Puppet at GitHub % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % now with ops taken out
  26. 26. " The State of Puppet at GitHub 0 100 200 300 400 2012-08-25 2012-10-20 2012-12-15 2013-02-09 2013-04-06 2013-06-01 2013-07-27 commits on master per week, last 12 months
  27. 27. " The State of Puppet at GitHub 0 1000 2000 3000 4000 total commits by author past year
  28. 28. " The State of Puppet at GitHub 0 175 350 525 700 total commits by author past year, except ops
  29. 29. " The State of Puppet at GitHub 0 175 350 525 700 total commits by authors with >10 commits, past year, except ops
  30. 30. " The State of Puppet at GitHub single puppetmaster
  31. 31. " The State of Puppet at GitHub rubygems
  32. 32. " The State of Puppet at GitHub ruby 1.8.7
  33. 33. " The State of Puppet at GitHub unicorn
  34. 34. " The State of Puppet at GitHub puppet 2.7.latest
  35. 35. " The State of Puppet at GitHub ~600 nodes
  36. 36. " The State of Puppet at GitHub run hourly via crond
  37. 37. " The State of Puppet at GitHub puppetdb
  38. 38. " The State of Puppet at GitHub nagiosdb
  39. 39. " The State of Puppet at GitHub "yo puppetdb, gimme all your nagios::object::* resources so I can update this nagios config"
  40. 40. " The State of Puppet at GitHub filtergendb
  41. 41. " The State of Puppet at GitHub "yo puppetdb, gimme all the filtergen::rule resources I would realize so I can update this filtergen config"
  42. 42. " The State of Puppet at GitHub ⚡ puppetdb ⚡
  43. 43. " The State of Puppet at GitHub as it turns out, an api call is faster than running puppet on a host
  44. 44. " The State of Puppet at GitHub gpanel
  45. 45. " The State of Puppet at GitHub "Imagine Puppet Dashboard meets Razor and went on a weekend trip to the beach with the Heroku API and drank epic amounts of blue drink"
  46. 46. " The State of Puppet at GitHub aka we reinvented our own, smaller version of Foreman
  47. 47. " The State of Puppet at GitHub inventory
  48. 48. " The State of Puppet at GitHub
  49. 49. " The State of Puppet at GitHub
  50. 50. " The State of Puppet at GitHub app configuration
  51. 51. " The State of Puppet at GitHub
  52. 52. " The State of Puppet at GitHub versioning of configuration values
  53. 53. " The State of Puppet at GitHub Create nil => 1 Update 1 => 2 Delete 2 => nil
  54. 54. " The State of Puppet at GitHub let's make credential rolling less awful
  55. 55. " The State of Puppet at GitHub provisioning
  56. 56. " The State of Puppet at GitHub provisioning is typically awful
  57. 57. " The State of Puppet at GitHub we sprinkled in some ChatOps
  58. 58. " The State of Puppet at GitHub
  59. 59. " The State of Puppet at GitHub a little bit later...
  60. 60. " The State of Puppet at GitHub <MACHINE>
  61. 61. " The State of Puppet at GitHub enc
  62. 62. " The State of Puppet at GitHub we have the dumbest ENC out there
  63. 63. " The State of Puppet at GitHub ⋆
  64. 64. " The State of Puppet at GitHub # /usr/local/sbin/fetch_gpanel_enc fe1.rs.github.com --- parameters: gpanel_cabinet: D20-13 gpanel_enabled: true gpanel_monitored: false
  65. 65. " The State of Puppet at GitHub we never specify classes via the ENC
  66. 66. " The State of Puppet at GitHub any variables we pass through are prefixed with gpanel_
  67. 67. How GitHub writes Puppet (
  68. 68. ( How GitHub Writes Puppet rodjek/puppet-lint
  69. 69. ( How GitHub Writes Puppet
  70. 70. ( How GitHub Writes Puppet if you aren't using puppet-lint to audit your puppet codebase, you are doing it wrong
  71. 71. ( How GitHub Writes Puppet puppet-lint enforces the Puppet Labs style guide
  72. 72. ( How GitHub Writes Puppet puppet-lint can even fix a ton of linter errors for you
  73. 73. ( How GitHub Writes Puppet put it in a pre-commit hook
  74. 74. ( How GitHub Writes Puppet $ git commit -am "can't lint this" modules/github/manifests/role/redis.pp: syntax ok modules/github/manifests/role/redis.pp - WARNING: => is not properly aligned on line 118 1 errors found, aborting commit.
  75. 75. ( How GitHub Writes Puppet and then buy rodjek a beer
  76. 76. ( How GitHub Writes Puppet rodjek/rspec-puppet
  77. 77. ( How GitHub Writes Puppet
  78. 78. ( How GitHub Writes Puppet if you aren't writing tests for your puppet code before running it on a server, you are doing it wrong
  79. 79. ( How GitHub Writes Puppet use whatever framework/library
  80. 80. ( How GitHub Writes Puppet rspec-puppet just happens to be a pretty good one
  81. 81. ( How GitHub Writes Puppet put it in a pre-commit hook
  82. 82. ( How GitHub Writes Puppet $ git commit -am "tests dont pass but whatever lol" 1) Expected redis::server would include class "more_than_a_single_c_thread" 1 failures encountered, aborting commit.
  83. 83. ( How GitHub Writes Puppet and then buy rodjek another beer
  84. 84. ( How GitHub Writes Puppet node definitions
  85. 85. ( How GitHub Writes Puppet we don't use an ENC to describe node classes
  86. 86. ( How GitHub Writes Puppet node /^github-redisd+/ { class { 'github::role::polling_redis': enabled => $::gpanel_enabled, environment => $::gpanel_environment, private_ipv4 => $::ipaddress, } }
  87. 87. ( How GitHub Writes Puppet abstractions all the way down
  88. 88. ( How GitHub Writes Puppet treat your site classes as cascades down to your dist classes
  89. 89. ( How GitHub Writes Puppet class redis::server( # params ) { class { 'redis::config': ... } -> class { 'redis::package': ... } ~> class { 'redis::service': ... } }
  90. 90. ( How GitHub Writes Puppet class github::redis( # params ) inherits github::defaults { $memory = $environment ? { 'stg' => '2G', default => $half_memory_gb } class { 'redis::server': ... } }
  91. 91. ( How GitHub Writes Puppet class github::role::polling_redis( # params ) { class { 'github::redis': # overrides based on specific node } }
  92. 92. ( How GitHub Writes Puppet augeas
  93. 93. ( How GitHub Writes Puppet you know what's not awesome?
  94. 94. ( How GitHub Writes Puppet an erb template that requires your class to take 52876423 parameters so you can configure every possible value in my.cnf
  95. 95. ( How GitHub Writes Puppet augeas { 'my.cnf/performance': context => '/files/etc/mysql/my.cnf/mysqld', changes => [ # automatic dump/restore 'set innodb_auto_lru_dump 18000', # innodb "set innodb_buffer_pool_size ${innodb_buffer_pool_size}", 'set innodb_log_file_size 256M', 'set innodb_log_buffer_size 8M', 'set innodb_lazy_drop_table 1', ], require => Percona::Server[$::fqdn] }
  96. 96. ( How GitHub Writes Puppet it can seem complex and scary
  97. 97. ( How GitHub Writes Puppet it is an amazing tool
  98. 98. How GitHub ships Puppet )
  99. 99. ) How GitHub Ships Puppet continuous integration
  100. 100. ) How GitHub Ships Puppet commit gets pushed
  101. 101. ) How GitHub Ships Puppet jenkins runs the test suite
  102. 102. ) How GitHub Ships Puppet status gets posted back to GitHub.com
  103. 103. ) How GitHub Ships Puppet tmm1/test-queue
  104. 104. ) How GitHub Ships Puppet
  105. 105. ) How GitHub Ships Puppet remembers how to better parallelize tests
  106. 106. ) How GitHub Ships Puppet continuous deployment
  107. 107. ) How GitHub Ships Puppet tests passed for commit, Hubot auto-deploys
  108. 108. ) How GitHub Ships Puppet branch deploy everything
  109. 109. ) How GitHub Ships Puppet a lot of shops have a few environments
  110. 110. ) How GitHub Ships Puppet testing staging production
  111. 111. ) How GitHub Ships Puppet currently we have 181 environments
  112. 112. ) How GitHub Ships Puppet that's not the same as 181 nodes not running production
  113. 113. ) How GitHub Ships Puppet we only have 5 of those
  114. 114. ) How GitHub Ships Puppet Hubot automatically merges the master branch before allowing any branch to deploy
  115. 115. ) How GitHub Ships Puppet ChatOps
  116. 116. ) How GitHub Ships Puppet 202 employees 100% access to Puppet 100% trust
  117. 117. ) How GitHub Ships Puppet the list of roles you'd never think would touch production puppet webkit developer core git developers core ruby developers svn developers search developer frontend designers windows developers mac developers support enterprise sales
  118. 118. ) How GitHub Ships Puppet every puppet run happens in chat
  119. 119. ) How GitHub Ships Puppet all puppet output goes to chat
  120. 120. ) How GitHub Ships Puppet everyone can see everything
  121. 121. ) How GitHub Ships Puppet everyone can do anything
  122. 122. ) How GitHub Ships Puppet hands-on learning, by accident
  123. 123. ) How GitHub Ships Puppet eliminate disruptive questions
  124. 124. The Future of Puppet at GitHub *
  125. 125. * The Future of Puppet at GitHub puppet 3.x
  126. 126. * The Future of Puppet at GitHub upgrading a large, old codebase from 2.7 to 3 is really painful
  127. 127. * The Future of Puppet at GitHub we've been talking about it for a year
  128. 128. * The Future of Puppet at GitHub we're still trying to push forward, but...
  129. 129. * The Future of Puppet at GitHub helping maintain security fixes for 2.7
  130. 130. * The Future of Puppet at GitHub goal is to be on 3.2 by end of year
  131. 131. * The Future of Puppet at GitHub ruby 2.0
  132. 132. * The Future of Puppet at GitHub moar faster rubby
  133. 133. * The Future of Puppet at GitHub we get this for free when we move to 3
  134. 134. * The Future of Puppet at GitHub mcollective
  135. 135. * The Future of Puppet at GitHub aka the story of github/shell and the wonders of ssh in a for-loop
  136. 136. * The Future of Puppet at GitHub there are a lot of cool things about mcollective
  137. 137. * The Future of Puppet at GitHub I think it will be a part of our stack in the future
  138. 138. * The Future of Puppet at GitHub but you can still go pretty damn far with bash and ssh
  139. 139. * The Future of Puppet at GitHub even more puppetdb tooling
  140. 140. * The Future of Puppet at GitHub waiting for multiple runs to converge exported resources is painful
  141. 141. * The Future of Puppet at GitHub use the puppetdb API to skip all that
  142. 142. * The Future of Puppet at GitHub even more gpanel integration
  143. 143. * The Future of Puppet at GitHub "databags"
  144. 144. * The Future of Puppet at GitHub "node search"
  145. 145. * The Future of Puppet at GitHub steal good ideas from other tools
  146. 146. closing thoughts
  147. 147. "all software is terrible" — anyone who's worked with software long enough
  148. 148. "no software is better than no software" — rtomayko
  149. 149. the plumbing doesn't matter when all you care about is the porcelain
  150. 150. write some damned good porcelain
  151. 151. THIS IS RODJEK
  152. 152. HE'S NORMALLY NOT THIS BLURRY
  153. 153. HE IS ALSO NOT MY BROTHER, BUT YOU REALLY SHOULD BUY HIM A BEER
  154. 154. YOU SHOULD BUY THIS MAN A LOT OF BEER
  155. 155. THANKS https://speakerdeck.com/wfarr/ puppet-at-github-puppetconf-2013 !

×