Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Continuously-Integrated Puppet in a Dynamic Environment


Published on

This talk will show how we deploy Puppet without a Puppetmaster on an autoscaling Amazon Web Services infrastructure. Key points of interest: - Masterless Puppet - Use of Jenkins for Puppet manifest testing and environment promotion (test->staging->production) - Puppet integration with Amazon CloudFormation

Sam Bashton
Director, Bashton Ltd
After working for a number of Internet Service Providers, Sam founded Bashton Ltd in 2004. Focussing exclusively on Linux and Open Source software, Sam and his team provide consultancy, support and 24/7 infrastructure management for a number of high-traffic websites. A serial early adopter, Sam has travelled the world providing training and consultancy and generally spreading the Open Source message. Sam lives in Manchester, UK.

  • Be the first to comment

Continuously-Integrated Puppet in a Dynamic Environment

  3. 3. ABOUTME Linux guy since Slackware, floppy disks and root + boot Using Puppet since 2007 Run a company in Manchester, North West England We provide outsourced ops for other companies
  4. 4. OUR FULLYMANAGED ENVIRONMENTS Primarily transactional websites (e-commerce) Majority (70%+) on Amazon Web Services (AWS) Majority using CentOS
  5. 5. HOWWEWORK Simple is better than complex Complexity is worth adding only if it provides obvious functional benefits Re-usability Resilience
  6. 6. WHY DID WE PICK AWS? Featureset and toolset massively in advance of any other cloud provider, public or private #1 customer reason for switching to AWS? The ability to scale on demand
  7. 7. TOOLSWEUSEFOR BUILDINGANDMANAGING Do one thing and do it well CloudFormation - Amazon tool to manage infrastructure Puppet - Manage system configuration Pulp - centralised repository, manages package revisions Jenkins
  8. 8. HOWWEUSEPUPPET No Puppetmaster Puppet manifests, hieradata and modules distributed to all machines via RPM All machines boot with a common, blank image and get configured at first boot
  9. 9. WHAT'SWRONGWITH MASTER BASEDPUPPET? Pets vs Cattle Puppet designed for a world of servers as pets We do not live in that world
  10. 10. PUPPETDESIGNEDFOR PETS Many assumptions in Puppet presume that your servers are pets Some of these work against us when managing a herd
  11. 11. MANUALCERTIFICATE SIGNING Clearly unsuitable when machines are automatically provisioned
  12. 12. POTENTIAL WORKAROUNDS: Autosign Use/write another automated certificate generation mechanism Possibly tied in with autoscaling
  13. 13. NO MECHANISMFOR CLEANINGOLDHOSTS Likely to have host-names reused, causing machines to fail to configure Puppetmaster will fill with certificates for machines that ran for a few hours and went away again
  14. 14. POTENTIAL WORKAROUNDS: Use UUID certificates Agree not to look in the certificate directory Write mechanism for cleaning up old certificates
  15. 15. HOSTSCONFIGUREDBASED ON HOSTNAME Our machines have names like ip-172-26-5-123 How does Puppet know what type of machine this is?
  16. 16. POTENTIAL WORKAROUNDS Use an external node classifier Use some mechanism for giving a better hostname, eg web-172-26-5-123 and use regex for node names
  17. 17. PUPPETMASTER ISASINGLE POINTOFFAILURE If the Puppetmaster fails, we can no longer autoscale up In particular, this could be a problem if there is availability zone failure
  18. 18. POTENTIAL WORKAROUNDS Clustered Puppetmasters
  19. 19. WORKAROUNDRECAP Use/write alternative certificate management software Write an external node classifier / mechanism for setting hostname appropriately Cluster multiple Puppetmasters
  20. 20. WHATWEDIDINSTEAD Decided using a Puppetmaster was trying to fit a square peg into a round hole Instead, decided to run Puppet without a master
  21. 21. APPLYINGLOCALPUPPET MANIFESTS puppet apply --modulepath=/etc/puppet/modules example.pp
  22. 22. DISTRIBUTINGMANIFESTS Use RPM Distribute full set of manifests/modules to each machine Apply only the manifest relevant to that machine
  23. 23. PACKINGPUPPET MANIFESTSIN RPM Build an RPM containing everything under /etc/puppet Make files readable only by root
  24. 24. APPLYPUPPETMANIFESTS Have an RPM %postinstcommand apply the Puppet config This isn't as straightforward as running the puppet applyfrom %postinst Puppet needs to install packages via yum, but yum is running installing the Puppet package Instead, we work around with a dirty hack: have the %postinstcreate an atscript which checks if yum has finished and then runs the puppet apply
  25. 25. RPMINSTALLATION AND MANAGEMENT How do we get these RPMs on our machines?
  26. 26. PULP We were already using Pulp Provides yum repository management Used for managing security updates and deploying application code
  27. 27. WHATISPULP Repository manager Allows us to easily audit what packages and versions are installed where Allows us to push package installations Uses qpid message queue Has concept of 'content distrubtion servers' for easy replication and clustering
  28. 28. HOWWEUSEPULP Puppet contains details of what packages should be installed Pulp manages which version of the package should be installed Pulp allows us to clone repos and copy packages between them for easy qa->stage->live environment management
  29. 29. DEPLOYING CONFIGURATION ASCODE Allows us to reuse our existing code deployment infrastructure Manage configuration deployment from Jenkins
  30. 30. HOWWEDEPLOYCODE Everything managed via the Jenkins continuous integration server Jenkins uses Pulp to install code on remote machines
  31. 31. DETAILSON HOWWE DEPLOYCODE Jenkins fetches code from source control (git) An RPM is built Tests are run If tests pass, the RPM is added to the relevant Pulp repository RPM installed on the target machine(s)
  32. 32. DEPLOYMENTLIFE-CYCLE Jenkins also manages deployment life-cycle RPMs are installed on staging Promoted Builds plugin then used to install the same RPMs on live once testing is complete
  33. 33. PUPPETDEPLOYMENT PROCESS Puppet manifests are checked into git Lint tests via Jenkins pulls in modules with librarian-puppet, then builds an RPM Deployment to test environments, functional tests for wider code-base run Jenkins Warnings plugin
  34. 34. PUTTINGITINTO PRODUCTION Once suitable tests (automated and manual) have been carried out, we promote Puppet config into production We use the Jenkins 'Promoted Builds' plugin for this
  36. 36. EXCEPT.. How does a machine get from a bare image to the state where we can push packages to it from Pulp? How does a machine know what type of machine it is? How do we find other resources, eg database hostname?
  37. 37. CLOUDFORMATION Amazon tool for specifying infrastructure Everything* we provision inside AWS is provisioned via CloudFormation JSON templates * Everything except for the things Amazon doesn't expose via CloudFormation..
  38. 38. CLOUD-INIT Works with multiple cloud types Sorts out things like SSH keys, allows us to configure host names Also allows us to provide a bash script to run on startup
  39. 39. PROVISIONINGABARE INSTANCE cloud-init automatically manually adds the pulp repo which contains Pulp, Puppet and our Puppet manifests/modules Installs appropriate RPMs Puppet runs, subscribing the machine to the relevant Pulp repos, and installing packages in the usual Puppet way
  40. 40. HOWDOESITKNOWWHAT TYPEOFMACHINEITIS? We tell it! Use an environmental variable $HOSTTYPE Simply run puppet apply --modulepath=/etc/puppet/modules ${HOSTTYPE}.pp
  41. 41. EXTRAFACTS Custom facter facts Also specified in an environmental variable Data comes from within the CloudFormation template On our list of things to look at: FACTER_HOSTENVIRONMENT=live FACTER_STACKNAME=customer-web-live
  42. 42. OTHER RESOURCES We either: Provide details as a facter fact `FACTER_DBHOST=xyz Also use this approach to limit distribution of secure details, eg DB passwords Discover via the EC2 API Eg Varnish servers discover web backends by calling API and finding hosts tagged appropriately
  43. 43. FREEWINS!
  44. 44. FREEWINS! Greater control over the timing of Puppet runs Improved visibility - for ops and devs Configuration changes now have to be deployed to testing/staging first
  45. 45. MOREFREEWINS! Puppet configs now have a version Easy to find config version on the machine itself Config changelogs accessible on every machine (Git changelog added to RPM)
  46. 46. THEDOWNSIDES Puppet manifests and modules on all machines Potentially a security issue? Mitigated by CloudFormation holding most sensitive data
  47. 47. ALTERNATIVE IMPLEMENTATIONS Don't want to use Pulp? Could do basically the same thing with yum s3 plugin Use mcollective to push package updates
  48. 48. FUTUREIMPROVEMENTS Build AMIs using Packer instead of configuring at boot time Decrease time to autoscale Would probably still need to run Puppet at first boot to configure machine specific settings
  49. 49. QUESTIONS? COMMENTS? Sam Bashton Twitter: @bashtoni (Psst.. )