Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
De-centralise and     ConquerMasterless Puppet in a Dynamic         Environment  Sam Bashton, Bashton Ltd
Who am I?● Linux guy since Slackware, floppy disks and  root + boot● Using Puppet since 2007● Run a company Manchester, No...
Our Environments● We provide outsourced ops for other  companies● High traffic environments● Most are now on Amazon Web Se...
Server instances, single day
How we use Puppet● No Puppetmaster● Puppet manifests and modules distributed to  all machines
Whats wrong with standard Puppet?● Pets vs Cattle● Standard Puppet configuration assumes that  servers are pets, not cattle
Whats wrong with standard Puppet?● Standard Puppetmaster/Puppet Client  configuration makes assumptions about  environment...
Whats wrong with standard Puppet?● Puppetmaster is a single point of failure● When servers are pets, this isnt too much of...
Whats wrong with standard Puppet?● When servers are auto-scaling cattle, new  instances can appear at any time● New instan...
Whats wrong with standard Puppet?● Our environments span multiple data centres  (availability zones)● Imagine a data centr...
Whats wrong with standard Puppet?● Resource contention● Even when Puppetmaster isnt in the failed  zone, multiple concurre...
Whats wrong with standard Puppet?● None of these problems are insurmountable● We could have configured a Puppetmaster a  c...
Meanwhile, on the other side of theroom...● Another team was evaluating Pulp● Provides yum repository management● To be us...
Pulp● Allows cloning of repos, copying packages  between repos● Allows us to push packages to clients   ○ Uses qpid messag...
How we deploy code● Everything managed via the Jenkins  continuous integration server● Jenkins uses Pulp to install code o...
How we deploy code● Jenkins fetches code from source control  (git)● An RPM is built● Tests are run● The RPM is added to t...
How we deploy code● Jenkins also manages deployment lifecycle● Promoted Builds plugin used to install  previously built RP...
Deploying configuration as code● Idea: Why not just build an RPM of our  Puppet manifests + modules?● Have puppet apply as...
Deploying configuration as code● Allowed us to reuse our existing code  deployment infrastructure● Manage configuration de...
How we deploy configuration● Puppet manifests and modules are checked  into git● Jenkins builds configuration into an RPM●...
Our system architecture● Quite AWS specific● Concepts could be applied to other clouds  ○ Once they catch up in terms of t...
Separation of Roles● CloudFormation - defines infrastructure● Puppet manages configuration● Pulp manages package versions ...
Instance Provisioning● Minimal images used● cloud-init the only addition beyond standard  CentOS install● cloud-init allow...
Puppet bootstrap● cloud-init script adds local Puppet yum repo  and installs the Puppet configuration RPM● Installing the ...
Machine metadata● cloud-init also sets some variables in  /etc/environment● $HOST_TYPE - the type of machine this is, eg  ...
Machine metadata● Also set facts to be used by facter, eg RDS  database hostname  ○ Values from CloudFormation● $FACTER_DB...
Defining machine roles● For each machine type there is a manifest  /etc/puppet/manifests/$HOST_TYPE.pp● This file looks so...
Building the RPM● Puppet manifests and modules are all  packed into an RPM● Owner set to root, mode 600● %postinst creates...
Deploying configuration
Free wins!
Free wins● Greater control over the timing of Puppet  runs● Improved visibility - for ops and devs● Configuration changes ...
More free wins● Puppet configs now have a version● Easy to find config version on the machine  itself● Config changelogs a...
Cheap wins
Cheap wins● Jenkins performs syntax checks with  puppet parser validate● Jenkins also runs puppet-lint on  manifests
Cheap wins● Config change required for new code?  ○ Make the Puppet RPM version a dependency
The downsides● Puppet manifests and modules on all  machines  ○ Potentially a security issue?● No reporting*
Alternative implementations● Dont want to use Pulp?● Could do basically the same thing with yum  s3 plugin  https://github...
Questions? Comments?          Sam Bashton        sam@bashton.com        Twitter: @bashtoni
Upcoming SlideShare
Loading in …5
×

De-centralise and conquer: Masterless Puppet in a dynamic environment

2,664 views

Published on

A talk by Sam Bashton, Bashton Ltd, presented at PuppetCamp London 2013.

How we use Puppet within Amazon Web Services

http://www.bashton.com/

Published in: Technology
  • Be the first to comment

De-centralise and conquer: Masterless Puppet in a dynamic environment

  1. 1. De-centralise and ConquerMasterless Puppet in a Dynamic Environment Sam Bashton, Bashton Ltd
  2. 2. Who am I?● Linux guy since Slackware, floppy disks and root + boot● Using Puppet since 2007● Run a company Manchester, North West England
  3. 3. Our Environments● We provide outsourced ops for other companies● High traffic environments● Most are now on Amazon Web Services● #1 reason for moving to AWS? The ability to scale on demand
  4. 4. Server instances, single day
  5. 5. How we use Puppet● No Puppetmaster● Puppet manifests and modules distributed to all machines
  6. 6. Whats wrong with standard Puppet?● Pets vs Cattle● Standard Puppet configuration assumes that servers are pets, not cattle
  7. 7. Whats wrong with standard Puppet?● Standard Puppetmaster/Puppet Client configuration makes assumptions about environments ○ Machine creation is a manual operation ■ Sign certs ○ No in-built mechanism to automatically clean up old machines
  8. 8. Whats wrong with standard Puppet?● Puppetmaster is a single point of failure● When servers are pets, this isnt too much of a problem ○ Existing servers continue to work, but not any updates
  9. 9. Whats wrong with standard Puppet?● When servers are auto-scaling cattle, new instances can appear at any time● New instances require config to become operational● Configuration requires Puppet
  10. 10. Whats wrong with standard Puppet?● Our environments span multiple data centres (availability zones)● Imagine a data centre fails● New instances get auto-provisioned to replace missing capacity● But these instances need the Puppetmaster● ..which was in the failed AZ
  11. 11. Whats wrong with standard Puppet?● Resource contention● Even when Puppetmaster isnt in the failed zone, multiple concurrent connections slow things down
  12. 12. Whats wrong with standard Puppet?● None of these problems are insurmountable● We could have configured a Puppetmaster a cluster of Puppetmasters for our needs ○ With autosign ○ and some sort of certificate distribution mechanism ○ uuid certificate names ○ And a mechanism for cleaning up old machines
  13. 13. Meanwhile, on the other side of theroom...● Another team was evaluating Pulp● Provides yum repository management● To be used for managing security updates and deploying application code http://pulpproject.org/
  14. 14. Pulp● Allows cloning of repos, copying packages between repos● Allows us to push packages to clients ○ Uses qpid message queue● Has content distribution servers for easy replication + clustering
  15. 15. How we deploy code● Everything managed via the Jenkins continuous integration server● Jenkins uses Pulp to install code on remote machines
  16. 16. How we deploy code● Jenkins fetches code from source control (git)● An RPM is built● Tests are run● The RPM is added to the relevant Pulp repository● RPM installed on the target machine(s)
  17. 17. How we deploy code● Jenkins also manages deployment lifecycle● Promoted Builds plugin used to install previously built RPMs on staging● Promoted Builds plugin then used to install the same RPMs on live once testing is complete
  18. 18. Deploying configuration as code● Idea: Why not just build an RPM of our Puppet manifests + modules?● Have puppet apply as part of the % postinst
  19. 19. Deploying configuration as code● Allowed us to reuse our existing code deployment infrastructure● Manage configuration deployment from Jenkins
  20. 20. How we deploy configuration● Puppet manifests and modules are checked into git● Jenkins builds configuration into an RPM● Jenkins promoted builds plugin applies the updates to environments via Pulp
  21. 21. Our system architecture● Quite AWS specific● Concepts could be applied to other clouds ○ Once they catch up in terms of toolsets..
  22. 22. Separation of Roles● CloudFormation - defines infrastructure● Puppet manages configuration● Pulp manages package versions ○ Pulp in turn managed via Jenkins for custom repos
  23. 23. Instance Provisioning● Minimal images used● cloud-init the only addition beyond standard CentOS install● cloud-init allows us to specify script to be run at boot
  24. 24. Puppet bootstrap● cloud-init script adds local Puppet yum repo and installs the Puppet configuration RPM● Installing the RPM installs Puppet and applies the configuration
  25. 25. Machine metadata● cloud-init also sets some variables in /etc/environment● $HOST_TYPE - the type of machine this is, eg web, cache
  26. 26. Machine metadata● Also set facts to be used by facter, eg RDS database hostname ○ Values from CloudFormation● $FACTER_DBHOST set via cloud-init too, eg /root/.my.cnf
  27. 27. Defining machine roles● For each machine type there is a manifest /etc/puppet/manifests/$HOST_TYPE.pp● This file looks something like this: node default { import global ... }
  28. 28. Building the RPM● Puppet manifests and modules are all packed into an RPM● Owner set to root, mode 600● %postinst creates an at job set for now + 1 minute to run puppet apply
  29. 29. Deploying configuration
  30. 30. Free wins!
  31. 31. Free wins● Greater control over the timing of Puppet runs● Improved visibility - for ops and devs● Configuration changes now have to be deployed to testing/staging first
  32. 32. More free wins● Puppet configs now have a version● Easy to find config version on the machine itself● Config changelogs accessible on every machine ○ (Git changelog added to RPM)
  33. 33. Cheap wins
  34. 34. Cheap wins● Jenkins performs syntax checks with puppet parser validate● Jenkins also runs puppet-lint on manifests
  35. 35. Cheap wins● Config change required for new code? ○ Make the Puppet RPM version a dependency
  36. 36. The downsides● Puppet manifests and modules on all machines ○ Potentially a security issue?● No reporting*
  37. 37. Alternative implementations● Dont want to use Pulp?● Could do basically the same thing with yum s3 plugin https://github.com/jbraeuer/yum-s3-plugin
  38. 38. Questions? Comments? Sam Bashton sam@bashton.com Twitter: @bashtoni

×