Your SlideShare is downloading. ×
0
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Continuously-Integrated Puppet in a Dynamic Environment

10,166

Published on

This talk will show how we deploy Puppet without a Puppetmaster on an autoscaling Amazon Web Services infrastructure. Key points of interest: - Masterless Puppet - Use of Jenkins for Puppet manifest …

This talk will show how we deploy Puppet without a Puppetmaster on an autoscaling Amazon Web Services infrastructure. Key points of interest: - Masterless Puppet - Use of Jenkins for Puppet manifest testing and environment promotion (test->staging->production) - Puppet integration with Amazon CloudFormation

Sam Bashton
Director, Bashton Ltd
After working for a number of Internet Service Providers, Sam founded Bashton Ltd in 2004. Focussing exclusively on Linux and Open Source software, Sam and his team provide consultancy, support and 24/7 infrastructure management for a number of high-traffic websites. A serial early adopter, Sam has travelled the world providing training and consultancy and generally spreading the Open Source message. Sam lives in Manchester, UK.

0 Comments
15 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
10,166
On Slideshare
0
From Embeds
0
Number of Embeds
17
Actions
Shares
0
Downloads
110
Comments
0
Likes
15
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. CONTINUOUSLY INTEGRATEDPUPPETIN A DYNAMIC ENVIRONMENT SAM BASHTON, BASHTON LTD
  • 2. CONTINUOUSLY INTEGRATEDPUPPETIN A DYNAMIC ENVIRONMENT MASTERLESSPUPPET:WHY ANDHOW SAM BASHTON, BASHTON LTD
  • 3. ABOUTME Linux guy since Slackware, floppy disks and root + boot Using Puppet since 2007 Run a company in Manchester, North West England We provide outsourced ops for other companies
  • 4. OUR FULLYMANAGED ENVIRONMENTS Primarily transactional websites (e-commerce) Majority (70%+) on Amazon Web Services (AWS) Majority using CentOS
  • 5. HOWWEWORK Simple is better than complex Complexity is worth adding only if it provides obvious functional benefits Re-usability Resilience
  • 6. WHY DID WE PICK AWS? Featureset and toolset massively in advance of any other cloud provider, public or private #1 customer reason for switching to AWS? The ability to scale on demand
  • 7. TOOLSWEUSEFOR BUILDINGANDMANAGING Do one thing and do it well CloudFormation - Amazon tool to manage infrastructure Puppet - Manage system configuration Pulp - centralised repository, manages package revisions Jenkins
  • 8. HOWWEUSEPUPPET No Puppetmaster Puppet manifests, hieradata and modules distributed to all machines via RPM All machines boot with a common, blank image and get configured at first boot
  • 9. WHAT'SWRONGWITH MASTER BASEDPUPPET? Pets vs Cattle Puppet designed for a world of servers as pets We do not live in that world
  • 10. PUPPETDESIGNEDFOR PETS Many assumptions in Puppet presume that your servers are pets Some of these work against us when managing a herd
  • 11. MANUALCERTIFICATE SIGNING Clearly unsuitable when machines are automatically provisioned
  • 12. POTENTIAL WORKAROUNDS: Autosign Use/write another automated certificate generation mechanism Possibly tied in with autoscaling
  • 13. NO MECHANISMFOR CLEANINGOLDHOSTS Likely to have host-names reused, causing machines to fail to configure Puppetmaster will fill with certificates for machines that ran for a few hours and went away again
  • 14. POTENTIAL WORKAROUNDS: Use UUID certificates Agree not to look in the certificate directory Write mechanism for cleaning up old certificates
  • 15. HOSTSCONFIGUREDBASED ON HOSTNAME Our machines have names like ip-172-26-5-123 How does Puppet know what type of machine this is?
  • 16. POTENTIAL WORKAROUNDS Use an external node classifier Use some mechanism for giving a better hostname, eg web-172-26-5-123 and use regex for node names
  • 17. PUPPETMASTER ISASINGLE POINTOFFAILURE If the Puppetmaster fails, we can no longer autoscale up In particular, this could be a problem if there is availability zone failure
  • 18. POTENTIAL WORKAROUNDS Clustered Puppetmasters
  • 19. WORKAROUNDRECAP Use/write alternative certificate management software Write an external node classifier / mechanism for setting hostname appropriately Cluster multiple Puppetmasters
  • 20. WHATWEDIDINSTEAD Decided using a Puppetmaster was trying to fit a square peg into a round hole Instead, decided to run Puppet without a master
  • 21. APPLYINGLOCALPUPPET MANIFESTS puppet apply --modulepath=/etc/puppet/modules example.pp
  • 22. DISTRIBUTINGMANIFESTS Use RPM Distribute full set of manifests/modules to each machine Apply only the manifest relevant to that machine
  • 23. PACKINGPUPPET MANIFESTSIN RPM Build an RPM containing everything under /etc/puppet Make files readable only by root
  • 24. APPLYPUPPETMANIFESTS Have an RPM %postinstcommand apply the Puppet config This isn't as straightforward as running the puppet applyfrom %postinst Puppet needs to install packages via yum, but yum is running installing the Puppet package Instead, we work around with a dirty hack: have the %postinstcreate an atscript which checks if yum has finished and then runs the puppet apply
  • 25. RPMINSTALLATION AND MANAGEMENT How do we get these RPMs on our machines?
  • 26. PULP We were already using Pulp Provides yum repository management Used for managing security updates and deploying application code http://pulpproject.org/
  • 27. WHATISPULP Repository manager Allows us to easily audit what packages and versions are installed where Allows us to push package installations Uses qpid message queue Has concept of 'content distrubtion servers' for easy replication and clustering
  • 28. HOWWEUSEPULP Puppet contains details of what packages should be installed Pulp manages which version of the package should be installed Pulp allows us to clone repos and copy packages between them for easy qa->stage->live environment management
  • 29. DEPLOYING CONFIGURATION ASCODE Allows us to reuse our existing code deployment infrastructure Manage configuration deployment from Jenkins
  • 30. HOWWEDEPLOYCODE Everything managed via the Jenkins continuous integration server Jenkins uses Pulp to install code on remote machines
  • 31. DETAILSON HOWWE DEPLOYCODE Jenkins fetches code from source control (git) An RPM is built Tests are run If tests pass, the RPM is added to the relevant Pulp repository RPM installed on the target machine(s)
  • 32. DEPLOYMENTLIFE-CYCLE Jenkins also manages deployment life-cycle RPMs are installed on staging Promoted Builds plugin then used to install the same RPMs on live once testing is complete
  • 33. PUPPETDEPLOYMENT PROCESS Puppet manifests are checked into git Lint tests via Jenkins pulls in modules with librarian-puppet, then builds an RPM Deployment to test environments, functional tests for wider code-base run Jenkins Warnings plugin
  • 34. PUTTINGITINTO PRODUCTION Once suitable tests (automated and manual) have been carried out, we promote Puppet config into production We use the Jenkins 'Promoted Builds' plugin for this
  • 35. JENKINS:PROMOTION
  • 36. EXCEPT.. How does a machine get from a bare image to the state where we can push packages to it from Pulp? How does a machine know what type of machine it is? How do we find other resources, eg database hostname?
  • 37. CLOUDFORMATION Amazon tool for specifying infrastructure Everything* we provision inside AWS is provisioned via CloudFormation JSON templates * Everything except for the things Amazon doesn't expose via CloudFormation..
  • 38. CLOUD-INIT Works with multiple cloud types Sorts out things like SSH keys, allows us to configure host names Also allows us to provide a bash script to run on startup
  • 39. PROVISIONINGABARE INSTANCE cloud-init automatically manually adds the pulp repo which contains Pulp, Puppet and our Puppet manifests/modules Installs appropriate RPMs Puppet runs, subscribing the machine to the relevant Pulp repos, and installing packages in the usual Puppet way
  • 40. HOWDOESITKNOWWHAT TYPEOFMACHINEITIS? We tell it! Use an environmental variable $HOSTTYPE Simply run puppet apply --modulepath=/etc/puppet/modules ${HOSTTYPE}.pp
  • 41. EXTRAFACTS Custom facter facts Also specified in an environmental variable Data comes from within the CloudFormation template On our list of things to look at: FACTER_HOSTENVIRONMENT=live FACTER_STACKNAME=customer-web-live https://github.com/fanduel/hiera-cloudformation
  • 42. OTHER RESOURCES We either: Provide details as a facter fact `FACTER_DBHOST=xyz Also use this approach to limit distribution of secure details, eg DB passwords Discover via the EC2 API Eg Varnish servers discover web backends by calling API and finding hosts tagged appropriately
  • 43. FREEWINS!
  • 44. FREEWINS! Greater control over the timing of Puppet runs Improved visibility - for ops and devs Configuration changes now have to be deployed to testing/staging first
  • 45. MOREFREEWINS! Puppet configs now have a version Easy to find config version on the machine itself Config changelogs accessible on every machine (Git changelog added to RPM)
  • 46. THEDOWNSIDES Puppet manifests and modules on all machines Potentially a security issue? Mitigated by CloudFormation holding most sensitive data
  • 47. ALTERNATIVE IMPLEMENTATIONS Don't want to use Pulp? Could do basically the same thing with yum s3 plugin Use mcollective to push package updates https://github.com/jbraeuer/yum-s3-plugin
  • 48. FUTUREIMPROVEMENTS Build AMIs using Packer instead of configuring at boot time Decrease time to autoscale Would probably still need to run Puppet at first boot to configure machine specific settings
  • 49. QUESTIONS? COMMENTS? Sam Bashton sam@bashton.com Twitter: @bashtoni (Psst.. )http://www.bashton.com/jobs/

×