Puppet at Gilt ...• Weve been using Puppet for almost 3 years• ~ 1000 puppet modules• Typical setup o Apache httpd+passenger for Puppetmaster o Puppetcommander (MCollective plugin) to schedule puppet runs o Puppet Dashboard (Reporting) o In-house External Node Classifier (aka Mothership)
Puppet - Incremental rollout• Deployment of changes o Commit to development branch o Push changes for code review o Once changes have been reviewed and verified, they get merged into development o A canary flag can be set for a number of nodes o Change is cherry-picked to master branch o A tag from master is created and deployed See behavior in the canary environment Expand the canary env. Rinse & repeat Remove canary flag
Puppet - External node Classifier• Mothership o In-house assets management & provisioning tool. It also acts as an External Node Classifier for Puppet Can provision bare-metal & virtual machines Integrated with Cobbler Also manages users/groups & sudoers • Synchronized to LDAP • Puppet deploys sudoers file DNS management Hosts can have one or more labels (tags) that are mapped to Puppet modules
Lessons learned / Best practices ...Puppet ...• Dont just run it!• Node mgmt can become difficult (+100)• Keep change in mind (e.g. OS upgrades)Mothership ...• ... is not enough. You need at least 2 views o Logical (deployed/reserved), Physical• You need to live with Physical != Logical o Physical will be generated. It is a feedback loop.
Requirements ...• Accelerate our ability to do incremental deployments (multiple times a day) o ... with easy rollback o ... accelerating our ability to innovate o ... while maintaining 100% uptime• The "Happy Path" needs to be without human intervention o From Commit to A/B test
Takeaways ...• Keep your modules small, keep them simple• Keep change in mind o Do incremental rollout of changes o Provide a default case in your puppet modules and use fail (sdtlib) as a default case.• If you are looking for a complete CD platform, then you need to embed Puppet into a larger solution