Gilt uses Puppet for infrastructure configuration management. They have around 1000 Puppet modules and typically run Puppetmaster with Apache and Passenger on CentOS. Changes go through code review and are deployed incrementally using a "canary" flag. An internal tool called Mothership acts as an external node classifier for Puppet and also handles provisioning, assets management, users/groups and DNS. Lessons learned include keeping modules small and simple, planning for change like OS upgrades, and needing at least two views (logical and physical) for node management.

1. Puppet Camp
Dublin - 2012
Andrés Pereira <apereira@gilt.com>
Roland Tritsch <rtritsch@gilt.com>

2. Gilt in a nutshell ...
• 5, 500, 1000/200, 300, 1, 18, ...

3. Gilt in a nutshell ...
Frontend (HTML[5], CSS, JS, ...) • 2 DCs + Amazon
EC2
Load Balancing (Zeus)
• CentOS Linux
• CDN (Akamai)
•
Aggregation (Java, Scala, ...)
Puppet, Cobbler, ...
Load Balancing (Zeus) • Zenoss, Nagios, ...
• Git & Gerrit
Services (Java, Scala, Ruby, ...)
• Jenkins
Database (PostgreSQL, MongoDB, ...)

4. Puppet at Gilt ...
• We've been using Puppet for almost 3 years
• ~ 1000 puppet modules
• Typical setup
o Apache httpd+passenger for Puppetmaster
o Puppetcommander (MCollective plugin) to schedule
puppet runs
o Puppet Dashboard (Reporting)
o In-house External Node Classifier (aka 'Mothership')

5. Puppet - Incremental rollout

6. Puppet - Incremental rollout
• Deployment of changes
o Commit to 'development' branch
o Push changes for code review
o Once changes have been reviewed and verified,
they get merged into 'development'
o A 'canary' flag can be set for a number of nodes
o Change is cherry-picked to 'master' branch
o A 'tag' from master is created and deployed
 See behavior in the canary environment
 Expand the canary env. Rinse & repeat
 Remove canary flag

7. Puppet - External node Classifier
http://mothership.sourceforge.net/

8. Puppet - External node Classifier
• Mothership
o In-house assets management & provisioning tool. It
also acts as an External Node Classifier for Puppet
 Can provision bare-metal & virtual machines
 Integrated with Cobbler
 Also manages users/groups & sudoers
• Synchronized to LDAP
• Puppet deploys sudoers file
 DNS management
 Hosts can have one or more labels (tags) that
are mapped to Puppet modules

11. Lessons learned / Best practices ...
Puppet ...
• Don't just run it!
• Node mgmt can become difficult (+100)
• Keep change in mind (e.g. OS upgrades)
Mothership ...
• ... is not enough. You need at least 2 views
o Logical (deployed/reserved), Physical
• You need to live with Physical != Logical
o Physical will be generated. It is a feedback loop.

12. Next steps ...

13. Requirements ...
• Accelerate our ability to do incremental
deployments (multiple times a day)
o ... with easy rollback
o ... accelerating our ability to innovate
o ... while maintaining 100% uptime
• The "Happy Path" needs to be without
human intervention
o From Commit to A/B test

14. Architecture ...
QA - Selenium / Deployment -
Code - Gerrit
TestComplete Puppet
Ion Cannon
Build - Jenkins/
Performance Analytics
Nexus

15. Takeaways ...
• Keep your modules small, keep them simple
• Keep change in mind
o Do incremental rollout of changes
o Provide a 'default' case in your puppet modules and
use 'fail' (sdtlib) as a default case.
• If you are looking for a complete CD
platform, then you need to embed Puppet
into a larger solution

16. Questions ...