1.
De-centralise and ConquerMasterless Puppet in a Dynamic Environment Sam Bashton, Bashton Ltd

2.
Who am I?● Linux guy since Slackware, floppy disks and root + boot● Using Puppet since 2007● Run a company Manchester, North West England

3.
Our Environments● We provide outsourced ops for other companies● High traffic environments● Most are now on Amazon Web Services● #1 reason for moving to AWS? The ability to scale on demand

4.
Server instances, single day

5.
How we use Puppet● No Puppetmaster● Puppet manifests and modules distributed to all machines

6.
Whats wrong with standard Puppet?● Pets vs Cattle● Standard Puppet configuration assumes that servers are pets, not cattle

7.
Whats wrong with standard Puppet?● Standard Puppetmaster/Puppet Client configuration makes assumptions about environments ○ Machine creation is a manual operation ■ Sign certs ○ No in-built mechanism to automatically clean up old machines

8.
Whats wrong with standard Puppet?● Puppetmaster is a single point of failure● When servers are pets, this isnt too much of a problem ○ Existing servers continue to work, but not any updates

9.
Whats wrong with standard Puppet?● When servers are auto-scaling cattle, new instances can appear at any time● New instances require config to become operational● Configuration requires Puppet

10.
Whats wrong with standard Puppet?● Our environments span multiple data centres (availability zones)● Imagine a data centre fails● New instances get auto-provisioned to replace missing capacity● But these instances need the Puppetmaster● ..which was in the failed AZ

11.
Whats wrong with standard Puppet?● Resource contention● Even when Puppetmaster isnt in the failed zone, multiple concurrent connections slow things down

12.
Whats wrong with standard Puppet?● None of these problems are insurmountable● We could have configured a Puppetmaster a cluster of Puppetmasters for our needs ○ With autosign ○ and some sort of certificate distribution mechanism ○ uuid certificate names ○ And a mechanism for cleaning up old machines

13.
Meanwhile, on the other side of theroom...● Another team was evaluating Pulp● Provides yum repository management● To be used for managing security updates and deploying application code http://pulpproject.org/

14.
Pulp● Allows cloning of repos, copying packages between repos● Allows us to push packages to clients ○ Uses qpid message queue● Has content distribution servers for easy replication + clustering

15.
How we deploy code● Everything managed via the Jenkins continuous integration server● Jenkins uses Pulp to install code on remote machines

16.
How we deploy code● Jenkins fetches code from source control (git)● An RPM is built● Tests are run● The RPM is added to the relevant Pulp repository● RPM installed on the target machine(s)

17.
How we deploy code● Jenkins also manages deployment lifecycle● Promoted Builds plugin used to install previously built RPMs on staging● Promoted Builds plugin then used to install the same RPMs on live once testing is complete

18.
Deploying configuration as code● Idea: Why not just build an RPM of our Puppet manifests + modules?● Have puppet apply as part of the % postinst

19.
Deploying configuration as code● Allowed us to reuse our existing code deployment infrastructure● Manage configuration deployment from Jenkins

20.
How we deploy configuration● Puppet manifests and modules are checked into git● Jenkins builds configuration into an RPM● Jenkins promoted builds plugin applies the updates to environments via Pulp

21.
Our system architecture● Quite AWS specific● Concepts could be applied to other clouds ○ Once they catch up in terms of toolsets..

22.
Separation of Roles● CloudFormation - defines infrastructure● Puppet manages configuration● Pulp manages package versions ○ Pulp in turn managed via Jenkins for custom repos

23.
Instance Provisioning● Minimal images used● cloud-init the only addition beyond standard CentOS install● cloud-init allows us to specify script to be run at boot

24.
Puppet bootstrap● cloud-init script adds local Puppet yum repo and installs the Puppet configuration RPM● Installing the RPM installs Puppet and applies the configuration

25.
Machine metadata● cloud-init also sets some variables in /etc/environment● $HOST_TYPE - the type of machine this is, eg web, cache

26.
Machine metadata● Also set facts to be used by facter, eg RDS database hostname ○ Values from CloudFormation● $FACTER_DBHOST set via cloud-init too, eg /root/.my.cnf

27.
Defining machine roles● For each machine type there is a manifest /etc/puppet/manifests/$HOST_TYPE.pp● This file looks something like this: node default { import global ... }

28.
Building the RPM● Puppet manifests and modules are all packed into an RPM● Owner set to root, mode 600● %postinst creates an at job set for now + 1 minute to run puppet apply

29.
Deploying configuration

30.
Free wins!

31.
Free wins● Greater control over the timing of Puppet runs● Improved visibility - for ops and devs● Configuration changes now have to be deployed to testing/staging first

32.
More free wins● Puppet configs now have a version● Easy to find config version on the machine itself● Config changelogs accessible on every machine ○ (Git changelog added to RPM)

33.
Cheap wins

34.
Cheap wins● Jenkins performs syntax checks with puppet parser validate● Jenkins also runs puppet-lint on manifests

35.
Cheap wins● Config change required for new code? ○ Make the Puppet RPM version a dependency

36.
The downsides● Puppet manifests and modules on all machines ○ Potentially a security issue?● No reporting*

37.
Alternative implementations● Dont want to use Pulp?● Could do basically the same thing with yum s3 plugin https://github.com/jbraeuer/yum-s3-plugin

38.
Questions? Comments? Sam Bashton sam@bashton.com Twitter: @bashtoni