Successfully reported this slideshow.

De-centralise and conquer: Masterless Puppet in a dynamic environment



Loading in …3
1 of 38
1 of 38

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

De-centralise and conquer: Masterless Puppet in a dynamic environment

  1. 1. De-centralise and Conquer Masterless Puppet in a Dynamic Environment Sam Bashton, Bashton Ltd
  2. 2. Who am I? ● Linux guy since Slackware, floppy disks and root + boot ● Using Puppet since 2007 ● Run a company Manchester, North West England
  3. 3. Our Environments ● We provide outsourced ops for other companies ● High traffic environments ● Most are now on Amazon Web Services ● #1 reason for moving to AWS? The ability to scale on demand
  4. 4. Server instances, single day
  5. 5. How we use Puppet ● No Puppetmaster ● Puppet manifests and modules distributed to all machines
  6. 6. What's wrong with standard Puppet? ● Pets vs Cattle ● Standard Puppet configuration assumes that servers are pets, not cattle
  7. 7. What's wrong with standard Puppet? ● Standard Puppetmaster/Puppet Client configuration makes assumptions about environments ○ Machine creation is a manual operation ■ Sign certs ○ No in-built mechanism to automatically clean up old machines
  8. 8. What's wrong with standard Puppet? ● Puppetmaster is a single point of failure ● When servers are pets, this isn't too much of a problem ○ Existing servers continue to work, but not any updates
  9. 9. What's wrong with standard Puppet? ● When servers are auto-scaling cattle, new instances can appear at any time ● New instances require config to become operational ● Configuration requires Puppet
  10. 10. What's wrong with standard Puppet? ● Our environments span multiple data centres ('availability zones') ● Imagine a data centre fails ● New instances get auto-provisioned to replace missing capacity ● But these instances need the Puppetmaster ● ..which was in the failed AZ
  11. 11. What's wrong with standard Puppet? ● Resource contention ● Even when Puppetmaster isn't in the failed zone, multiple concurrent connections slow things down
  12. 12. What's wrong with standard Puppet? ● None of these problems are insurmountable ● We could have configured a Puppetmaster a cluster of Puppetmasters for our needs ○ With autosign ○ and some sort of certificate distribution mechanism ○ uuid certificate names ○ And a mechanism for cleaning up old machines
  13. 13. Meanwhile, on the other side of the room... ● Another team was evaluating Pulp ● Provides yum repository management ● To be used for managing security updates and deploying application code
  14. 14. Pulp ● Allows cloning of repos, copying packages between repos ● Allows us to push packages to clients ○ Uses qpid message queue ● Has 'content distribution servers' for easy replication + clustering
  15. 15. How we deploy code ● Everything managed via the Jenkins continuous integration server ● Jenkins uses Pulp to install code on remote machines
  16. 16. How we deploy code ● Jenkins fetches code from source control (git) ● An RPM is built ● Tests are run ● The RPM is added to the relevant Pulp repository ● RPM installed on the target machine(s)
  17. 17. How we deploy code ● Jenkins also manages deployment lifecycle ● 'Promoted Builds' plugin used to install previously built RPMs on staging ● Promoted Builds plugin then used to install the same RPMs on live once testing is complete
  18. 18. Deploying configuration as code ● Idea: Why not just build an RPM of our Puppet manifests + modules? ● Have puppet apply as part of the % postinst
  19. 19. Deploying configuration as code ● Allowed us to reuse our existing code deployment infrastructure ● Manage configuration deployment from Jenkins
  20. 20. How we deploy configuration ● Puppet manifests and modules are checked into git ● Jenkins builds configuration into an RPM ● Jenkins promoted builds plugin applies the updates to environments via Pulp
  21. 21. Our system architecture ● Quite AWS specific ● Concepts could be applied to other clouds ○ Once they catch up in terms of toolsets..
  22. 22. Separation of Roles ● CloudFormation - defines infrastructure ● Puppet manages configuration ● Pulp manages package versions ○ Pulp in turn managed via Jenkins for custom repos
  23. 23. Instance Provisioning ● Minimal images used ● cloud-init the only addition beyond standard CentOS install ● cloud-init allows us to specify script to be run at boot
  24. 24. Puppet bootstrap ● cloud-init script adds local Puppet yum repo and installs the Puppet configuration RPM ● Installing the RPM installs Puppet and applies the configuration
  25. 25. Machine metadata ● cloud-init also sets some variables in /etc/environment ● $HOST_TYPE - the type of machine this is, eg web, cache
  26. 26. Machine metadata ● Also set facts to be used by facter, eg RDS database hostname ○ Values from CloudFormation ● $FACTER_DBHOST set via cloud-init too, eg /root/.my.cnf
  27. 27. Defining machine roles ● For each machine type there is a manifest /etc/puppet/manifests/$HOST_TYPE.pp ● This file looks something like this: node default { import global ... }
  28. 28. Building the RPM ● Puppet manifests and modules are all packed into an RPM ● Owner set to root, mode 600 ● %postinst creates an at job set for now + 1 minute to run puppet apply
  29. 29. Deploying configuration
  30. 30. Free wins!
  31. 31. Free wins ● Greater control over the timing of Puppet runs ● Improved visibility - for ops and devs ● Configuration changes now have to be deployed to testing/staging first
  32. 32. More free wins ● Puppet configs now have a version ● Easy to find config version on the machine itself ● Config changelogs accessible on every machine ○ (Git changelog added to RPM)
  33. 33. Cheap wins
  34. 34. Cheap wins ● Jenkins performs syntax checks with puppet parser validate ● Jenkins also runs puppet-lint on manifests
  35. 35. Cheap wins ● Config change required for new code? ○ Make the Puppet RPM version a dependency
  36. 36. The downsides ● Puppet manifests and modules on all machines ○ Potentially a security issue? ● No reporting*
  37. 37. Alternative implementations ● Don't want to use Pulp? ● Could do basically the same thing with yum s3 plugin
  38. 38. Questions? Comments? Sam Bashton Twitter: @bashtoni