The document outlines the journey of Puppet implementation at eBay Classifieds, detailing four iterations from a disorganized codebase to a structured, modular approach that facilitates code sharing and peer reviews. Key improvements were made in version control, release management, and code review processes using tools like Gerrit and PuppetDB. The final structure emphasizes a layered module design, allowing for efficient configurations and fast catalog compilations while maintaining flexibility in code usage.

1. Story of Puppet @eBay Classifieds Group
or how we came to the setup we use

2. Some bio or “Who is Vladimir”
! Sysadmin for eighteen years
! Currently in Solutions team of eCG’s marktplaats.nl
! All-round nerd
! Hobbyist programmer

3. Puppet at eBay Classifieds:
! 9 countries
! 13 platforms
! 1 codebase
! 1 change management process

4. Puppet at eBay Classifieds:
! We started adopting puppet in 2008
! Since then we had about 4 different iterations of “this is bloated and
doesn’t work, let’s think more and re-do”
! Currently we believe we achieved a workable scenario where we:
! Enable teams to share and re-use puppet code
! Enforce the same code style and standards
! Allow for peer-reviews of the code, also enforce them
“If it’s not in Puppet – it does not exist.”

5. What we are going to talk about:
• Different iterations of puppet implementation at eCG
• Go in-depth on the structure of the current one
• Process around the codebase
• Tools

6. “Bucket of recipes”
Puppet at eBay Classifieds (1st implementation):
! A bunch of modules without visible structure (server-role based)
! Relations between modules on any levels (includes, instantiations)
! One Subversion repository, where everyone works on trunk
What were the lessons learned?
ü As puppet adoption grew, modules for different platforms started
clashing (same, yet a bit different functionality)
ü As number of modules and their dependencies grew people were
afraid of making changes - dependency tree was very hard to follow
ü Resulted in tons of duplicate code
ü Change management was very risky – you commit something and
it’s live

7. “Buckets of recipes”
Puppet at eBay Classifieds (2nd implementation):
! What did we improve?
! Release management with Subversion branching
! Better module grouping
What were the lessons learned?
ü Waiting on releases is killing. If you need a change - you need it
now, not in 2 or 3 days
ü Provokes people to lock machines to their own puppet environment
ü Surprises at times of the releases. For some people it was daytime,
for others it was middle of the night

8. Puppet at eBay Classifieds (3rd implementation):
! What did we improve?
! Switched to Git
! Completely redesigned modules structure (more on this later)
! Introduced extlookup for data separation vs configuration logic
! Release more often
What were the lessons learned?
ü When release process is human-dependent it will be a bottleneck
ü Extlookup is nice, but in many places unreadable

9. Puppet at eBay Classifieds (4th implementation):
! What did we improve?
! Introduced Gerrit for code reviews
! Implemented automatic puppet-lint, template syntax and PEP8 checkers
! Introduced automated quorum votes for peer reviews
! Release as soon as your code passed checks and YOU need it live
! Further improved module tree structure
! Replaced extlookup with hiera
! Introduced puppetdb

10. So how does it look now?
! Four module levels
! One include direction Config
! No includes on the same level
Platform
eCG
Shared

11. Shared modules
! Shared modules are very basic, deploy one piece
of software in its standard configuration (no Config
customization)
Platform
! Standalone. Includes no other modules
! Installs needed packages, basically
eCG
! Parameterized to allow configuration override
! Examples: apache, ntpd, activemq, mysql, postgres Shared

12. eCG modules
! eCG modules add our specific eCG-wide
approved customizations. Rule: it must play well Config
with majority of teams to make it
Platform
! Mostly include shared modules
! Never include each other
eCG
! Override default shared module configuration with
eCG specific one via parameters Shared
! May be parameterized to allow for configuration
tweaks or overrides
! Examples: puppetmaster, nagios checks, collectd
plugins

13. Platform modules
! Platform level introduces company-specific
modules and configuration customizations
Config
! May include eCG modules
! May include shared modules Platform
! Never include each other
eCG
Shared

14. Config modules
! Config level introduces server-role specific
modules
Config
! Are the starting point for our servers
configuration
Platform
! May include platform, eCG or shared modules
! Never include each other eCG
! Node definition must include config module Shared

15. Why did we do it like this?
! Easy starting point. Look up a node, see its
class, open the class – you see on one page
how your server is configured on a global level Config
! If you need to know more, dive deeper
Platform
! We enable code sharing, we do not enforce it
! If you see a module in shared you can use – use eCG
it. If you can’t use it, extend it in a backwards-
compatible way or write other extension on an Shared
appropriate level

16. Hiera
! Great way to separate code from data
! Company-based structure
! Lookup path based on facts to allow company data separation
while keeping the same source tree
:datadir: /var/lib/hiera/%{environment}/%{companyprefix}
! Inside company dirs a variety of fact-based lookups to allow for
fine-grained data structures

17. PuppetDB
! Much faster catalog compilations for huge catalogs (nagios server
with 200k resources)
! REST API to talk to for inventory and searches
! Many brilliant ways to (ab)use it
! Stores all facts (can be used to feed CMDBs)
! Stores most recent catalog (can be used for inspections)
! POC we are running - internal Puppet naginator module is very slow,
collection of exported resources takes about 15 minutes. We use
PuppetDB to generate the same nagios configs in just under 8
seconds
! “If there is data, it can be used”. PuppetDB is a central data store.
Love.

18. So all good, but how does this work in practice?
Local Test the change
Push to Gerrit Code review Merge
environment with --environment
! Per-user environments on puppetmasters
! Make the change in your own env
! Test it from your own env
! Push to Gerrit
! Ask team members to review
! Merge

19. Gerrit
! Great code review tool (https://code.google.com/p/gerrit/)
! Acts as a Git server with fine-grained access control
! Each commit is a separate change, that requires votes for
approvals
! Votes can be +1 for Code Review, +2 for change approval
! No +2 – no merge
! We use voting system with auto +2
! Depending on platform size or number of platforms affected by the
change more votes are needed to get a change through
! Each change can be “pushed through” in case of an emergency

20. More Gerrit goodies
! Allows for post-push hooks
! Puppet-lint
! Ruby templates parsing
! PEP8 (we love Python)
! Each change is separately checked out by hooks and tested for
syntax or other obvious errors
! Concept of one commit – one change forces people to create
atomic changes that are easy to revert in case of errors
! Keeping it all in the same system, even if you decide to split Git
repositories still makes work done by colleagues VISIBLE

21. Gerrit and puppetmasters
! Master is automatically pulled by crons on puppetmasters
! Gerrit disallows pushing directly to master
! Even if you force-merge a change, it makes your change
VISIBLE and easily REVERTIBLE
! Takes about half a minute extra time
! Ensures you didn’t slack out on syntax checkers

22. To sum it all up
! There is no single structure that fits every company
! Puppet code is like any other code
! Write
! Use
! Refactor if needed
! Layered modules structure can work well in many cases
! Complementary tooling around Puppet is a life- (and time-)
saver
! Lint(s), Gerrit, PuppetDB

23. Work to be done in our house
! Approval process improvements
! Better quorum schemas
! Easier ways of code style and syntax checking
! See that people don’t leave changes “hanging”
! Integrate Gerrit with corporate GitHub
! Work more with community
! Open source some generic shared modules
! Offer changes to PuppetForge modules
! POC with Vagrant for puppet code testing

24. Questions?

25. Thank you!
Special thanks to:
• PuppetLabs and eCG for the opportunity to speak here
• Teams of eCG for bearing with me all these years
• You all for attending and listening
You can reach me by:
• Mailing to favoretti@gmail.com
• Following me on GitHub (
https://github.com/favoretti)
We are hiring!
Visit http://ebayclassifiedsgroup.com/careers
for more info.