De-centralise and Conquer: Masterless Puppet in a Dynamic Environment


Published on

"De-centralise and Conquer: Masterless Puppet in a dynamic environment" by Sam Bashton of Bashton Ltd., at Puppet Camp London 2013. Learn about upcoming Puppet Camps at

Published in: Technology

De-centralise and Conquer: Masterless Puppet in a Dynamic Environment

  1. 1. De-centralise and ConquerMasterless Puppet in a Dynamic Environment Sam Bashton, Bashton Ltd
  2. 2. Who am I?● Linux guy since Slackware, floppy disks and root + boot● Using Puppet since 2007● Run a company Manchester, North West England
  3. 3. Our Environments● We provide outsourced ops for other companies● High traffic environments● Most are now on Amazon Web Services● #1 reason for moving to AWS? The ability to scale on demand
  4. 4. Server instances, single day
  5. 5. How we use Puppet● No Puppetmaster● Puppet manifests and modules distributed to all machines
  6. 6. Whats wrong with standard Puppet?● Pets vs Cattle● Standard Puppet configuration assumes that servers are pets, not cattle
  7. 7. Whats wrong with standard Puppet?● Standard Puppetmaster/Puppet Client configuration makes assumptions about environments ○ Machine creation is a manual operation ■ Sign certs ○ No in-built mechanism to automatically clean up old machines
  8. 8. Whats wrong with standard Puppet?● Puppetmaster is a single point of failure● When servers are pets, this isnt too much of a problem ○ Existing servers continue to work, but not any updates
  9. 9. Whats wrong with standard Puppet?● When servers are auto-scaling cattle, new instances can appear at any time● New instances require config to become operational● Configuration requires Puppet
  10. 10. Whats wrong with standard Puppet?● Our environments span multiple data centres (availability zones)● Imagine a data centre fails● New instances get auto-provisioned to replace missing capacity● But these instances need the Puppetmaster● ..which was in the failed AZ
  11. 11. Whats wrong with standard Puppet?● Resource contention● Even when Puppetmaster isnt in the failed zone, multiple concurrent connections slow things down
  12. 12. Whats wrong with standard Puppet?● None of these problems are insurmountable● We could have configured a Puppetmaster a cluster of Puppetmasters for our needs ○ With autosign ○ and some sort of certificate distribution mechanism ○ uuid certificate names ○ And a mechanism for cleaning up old machines
  13. 13. Meanwhile, on the other side of theroom...● Another team was evaluating Pulp● Provides yum repository management● To be used for managing security updates and deploying application code
  14. 14. Pulp● Allows cloning of repos, copying packages between repos● Allows us to push packages to clients ○ Uses qpid message queue● Has content distribution servers for easy replication + clustering
  15. 15. How we deploy code● Everything managed via the Jenkins continuous integration server● Jenkins uses Pulp to install code on remote machines
  16. 16. How we deploy code● Jenkins fetches code from source control (git)● An RPM is built● Tests are run● The RPM is added to the relevant Pulp repository● RPM installed on the target machine(s)
  17. 17. How we deploy code● Jenkins also manages deployment lifecycle● Promoted Builds plugin used to install previously built RPMs on staging● Promoted Builds plugin then used to install the same RPMs on live once testing is complete
  18. 18. Deploying configuration as code● Idea: Why not just build an RPM of our Puppet manifests + modules?● Have puppet apply as part of the % postinst
  19. 19. Deploying configuration as code● Allowed us to reuse our existing code deployment infrastructure● Manage configuration deployment from Jenkins
  20. 20. How we deploy configuration● Puppet manifests and modules are checked into git● Jenkins builds configuration into an RPM● Jenkins promoted builds plugin applies the updates to environments via Pulp
  21. 21. Our system architecture● Quite AWS specific● Concepts could be applied to other clouds ○ Once they catch up in terms of toolsets..
  22. 22. Separation of Roles● CloudFormation - defines infrastructure● Puppet manages configuration● Pulp manages package versions ○ Pulp in turn managed via Jenkins for custom repos
  23. 23. Instance Provisioning● Minimal images used● cloud-init the only addition beyond standard CentOS install● cloud-init allows us to specify script to be run at boot
  24. 24. Puppet bootstrap● cloud-init script adds local Puppet yum repo and installs the Puppet configuration RPM● Installing the RPM installs Puppet and applies the configuration
  25. 25. Machine metadata● cloud-init also sets some variables in /etc/environment● $HOST_TYPE - the type of machine this is, eg web, cache
  26. 26. Machine metadata● Also set facts to be used by facter, eg RDS database hostname ○ Values from CloudFormation● $FACTER_DBHOST set via cloud-init too, eg /root/.my.cnf
  27. 27. Defining machine roles● For each machine type there is a manifest /etc/puppet/manifests/$HOST_TYPE.pp● This file looks something like this: node default { import global ... }
  28. 28. Building the RPM● Puppet manifests and modules are all packed into an RPM● Owner set to root, mode 600● %postinst creates an at job set for now + 1 minute to run puppet apply
  29. 29. Deploying configuration
  30. 30. Free wins!
  31. 31. Free wins● Greater control over the timing of Puppet runs● Improved visibility - for ops and devs● Configuration changes now have to be deployed to testing/staging first
  32. 32. More free wins● Puppet configs now have a version● Easy to find config version on the machine itself● Config changelogs accessible on every machine ○ (Git changelog added to RPM)
  33. 33. Cheap wins
  34. 34. Cheap wins● Jenkins performs syntax checks with puppet parser validate● Jenkins also runs puppet-lint on manifests
  35. 35. Cheap wins● Config change required for new code? ○ Make the Puppet RPM version a dependency
  36. 36. The downsides● Puppet manifests and modules on all machines ○ Potentially a security issue?● No reporting*
  37. 37. Alternative implementations● Dont want to use Pulp?● Could do basically the same thing with yum s3 plugin
  38. 38. Questions? Comments? Sam Bashton Twitter: @bashtoni