Story of Puppet @eBay Global Classifieds Group (eCG)


Published on

"Story of Puppet @eBay Global Classifieds Group (eCG)" by Vladimir Lazarenko of eBay at Puppet Camp Amsterdam 2013.

Published in: Technology, Self Improvement
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Story of Puppet @eBay Global Classifieds Group (eCG)

  1. 1. Story of Puppet @eBay Classifieds Groupor how we came to the setup we use
  2. 2. Some bio or “Who is Vladimir”! Sysadmin for eighteen years!  Currently in Solutions team of eCG’s!  All-round nerd!  Hobbyist programmer
  3. 3. Puppet at eBay Classifieds:!  9 countries!  13 platforms!  1 codebase!  1 change management process
  4. 4. Puppet at eBay Classifieds:!  We started adopting puppet in 2008!  Since then we had about 4 different iterations of “this is bloated and doesn’t work, let’s think more and re-do”!  Currently we believe we achieved a workable scenario where we: !  Enable teams to share and re-use puppet code !  Enforce the same code style and standards !  Allow for peer-reviews of the code, also enforce them “If it’s not in Puppet – it does not exist.”
  5. 5. What we are going to talk about:•  Different iterations of puppet implementation at eCG•  Go in-depth on the structure of the current one•  Process around the codebase•  Tools
  6. 6. “Bucket of recipes”Puppet at eBay Classifieds (1st implementation):!  A bunch of modules without visible structure (server-role based)!  Relations between modules on any levels (includes, instantiations)!  One Subversion repository, where everyone works on trunkWhat were the lessons learned?ü  As puppet adoption grew, modules for different platforms started clashing (same, yet a bit different functionality)ü  As number of modules and their dependencies grew people were afraid of making changes - dependency tree was very hard to followü  Resulted in tons of duplicate codeü  Change management was very risky – you commit something and it’s live
  7. 7. “Buckets of recipes”Puppet at eBay Classifieds (2nd implementation):!  What did we improve? !  Release management with Subversion branching !  Better module groupingWhat were the lessons learned?ü  Waiting on releases is killing. If you need a change - you need it now, not in 2 or 3 daysü  Provokes people to lock machines to their own puppet environmentü  Surprises at times of the releases. For some people it was daytime, for others it was middle of the night
  8. 8. Puppet at eBay Classifieds (3rd implementation):!  What did we improve? !  Switched to Git !  Completely redesigned modules structure (more on this later) !  Introduced extlookup for data separation vs configuration logic !  Release more oftenWhat were the lessons learned?ü  When release process is human-dependent it will be a bottleneckü  Extlookup is nice, but in many places unreadable
  9. 9. Puppet at eBay Classifieds (4th implementation):!  What did we improve? !  Introduced Gerrit for code reviews !  Implemented automatic puppet-lint, template syntax and PEP8 checkers !  Introduced automated quorum votes for peer reviews !  Release as soon as your code passed checks and YOU need it live !  Further improved module tree structure !  Replaced extlookup with hiera !  Introduced puppetdb
  10. 10. So how does it look now? !  Four module levels !  One include direction Config !  No includes on the same level Platform eCG Shared
  11. 11. Shared modules !  Shared modules are very basic, deploy one piece of software in its standard configuration (no Config customization) Platform !  Standalone. Includes no other modules !  Installs needed packages, basically eCG !  Parameterized to allow configuration override !  Examples: apache, ntpd, activemq, mysql, postgres Shared
  12. 12. eCG modules ! eCG modules add our specific eCG-wide approved customizations. Rule: it must play well Config with majority of teams to make it Platform !  Mostly include shared modules !  Never include each other eCG !  Override default shared module configuration with eCG specific one via parameters Shared !  May be parameterized to allow for configuration tweaks or overrides !  Examples: puppetmaster, nagios checks, collectd plugins
  13. 13. Platform modules !  Platform level introduces company-specific modules and configuration customizations Config !  May include eCG modules !  May include shared modules Platform !  Never include each other eCG Shared
  14. 14. Config modules ! Config level introduces server-role specific modules Config !  Are the starting point for our servers configuration Platform !  May include platform, eCG or shared modules !  Never include each other eCG !  Node definition must include config module Shared
  15. 15. Why did we do it like this? !  Easy starting point. Look up a node, see its class, open the class – you see on one page how your server is configured on a global level Config !  If you need to know more, dive deeper Platform !  We enable code sharing, we do not enforce it !  If you see a module in shared you can use – use eCG it. If you can’t use it, extend it in a backwards- compatible way or write other extension on an Shared appropriate level
  16. 16. Hiera !  Great way to separate code from data !  Company-based structure !  Lookup path based on facts to allow company data separation while keeping the same source tree :datadir: /var/lib/hiera/%{environment}/%{companyprefix} !  Inside company dirs a variety of fact-based lookups to allow for fine-grained data structures
  17. 17. PuppetDB!  Much faster catalog compilations for huge catalogs (nagios server with 200k resources)!  REST API to talk to for inventory and searches!  Many brilliant ways to (ab)use it !  Stores all facts (can be used to feed CMDBs) !  Stores most recent catalog (can be used for inspections) !  POC we are running - internal Puppet naginator module is very slow, collection of exported resources takes about 15 minutes. We use PuppetDB to generate the same nagios configs in just under 8 seconds !  “If there is data, it can be used”. PuppetDB is a central data store. Love.
  18. 18. So all good, but how does this work in practice? Local Test the change Push to Gerrit Code review Merge environment with --environment !  Per-user environments on puppetmasters !  Make the change in your own env !  Test it from your own env !  Push to Gerrit !  Ask team members to review !  Merge
  19. 19. Gerrit !  Great code review tool ( !  Acts as a Git server with fine-grained access control !  Each commit is a separate change, that requires votes for approvals !  Votes can be +1 for Code Review, +2 for change approval !  No +2 – no merge !  We use voting system with auto +2 !  Depending on platform size or number of platforms affected by the change more votes are needed to get a change through !  Each change can be “pushed through” in case of an emergency
  20. 20. More Gerrit goodies !  Allows for post-push hooks !  Puppet-lint !  Ruby templates parsing !  PEP8 (we love Python) !  Each change is separately checked out by hooks and tested for syntax or other obvious errors !  Concept of one commit – one change forces people to create atomic changes that are easy to revert in case of errors !  Keeping it all in the same system, even if you decide to split Git repositories still makes work done by colleagues VISIBLE
  21. 21. Gerrit and puppetmasters !  Master is automatically pulled by crons on puppetmasters ! Gerrit disallows pushing directly to master !  Even if you force-merge a change, it makes your change VISIBLE and easily REVERTIBLE !  Takes about half a minute extra time !  Ensures you didn’t slack out on syntax checkers
  22. 22. To sum it all up !  There is no single structure that fits every company !  Puppet code is like any other code !  Write !  Use !  Refactor if needed !  Layered modules structure can work well in many cases !  Complementary tooling around Puppet is a life- (and time-) saver !  Lint(s), Gerrit, PuppetDB
  23. 23. Work to be done in our house !  Approval process improvements !  Better quorum schemas !  Easier ways of code style and syntax checking !  See that people don’t leave changes “hanging” !  Integrate Gerrit with corporate GitHub !  Work more with community !  Open source some generic shared modules !  Offer changes to PuppetForge modules !  POC with Vagrant for puppet code testing
  24. 24. Questions?
  25. 25. Thank you! Special thanks to: •  PuppetLabs and eCG for the opportunity to speak here •  Teams of eCG for bearing with me all these years •  You all for attending and listening You can reach me by: •  Mailing to •  Following me on GitHub ( are hiring!Visit more info.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.