CONTINUOUSLY
INTEGRATEDPUPPETIN A
DYNAMIC ENVIRONMENT
SAM BASHTON, BASHTON LTD
CONTINUOUSLY
INTEGRATEDPUPPETIN A
DYNAMIC ENVIRONMENT
MASTERLESSPUPPET:WHY
ANDHOW
SAM BASHTON, BASHTON LTD
ABOUTME
Linux guy since Slackware, floppy disks and root + boot
Using Puppet since 2007
Run a company in Manchester, North...
OUR FULLYMANAGED
ENVIRONMENTS
Primarily transactional websites (e-commerce)
Majority (70%+) on Amazon Web Services (AWS)
M...
HOWWEWORK
Simple is better than complex
Complexity is worth adding only if it provides obvious
functional benefits
Re-usab...
WHY DID WE PICK AWS?
Featureset and toolset massively in advance of any
other cloud provider, public or private
#1 custome...
TOOLSWEUSEFOR
BUILDINGANDMANAGING
Do one thing and do it well
CloudFormation - Amazon tool to manage
infrastructure
Puppet...
HOWWEUSEPUPPET
No Puppetmaster
Puppet manifests, hieradata and modules distributed
to all machines via RPM
All machines bo...
WHAT'SWRONGWITH
MASTER BASEDPUPPET?
Pets vs Cattle
Puppet designed for a world of servers as pets
We do not live in that w...
PUPPETDESIGNEDFOR PETS
Many assumptions in Puppet presume that your
servers are pets
Some of these work against us when ma...
MANUALCERTIFICATE
SIGNING
Clearly unsuitable when machines are automatically
provisioned
POTENTIAL WORKAROUNDS:
Autosign
Use/write another automated certificate generation
mechanism
Possibly tied in with autosca...
NO MECHANISMFOR
CLEANINGOLDHOSTS
Likely to have host-names reused, causing machines to
fail to configure
Puppetmaster will...
POTENTIAL WORKAROUNDS:
Use UUID certificates
Agree not to look in the certificate directory
Write mechanism for cleaning u...
HOSTSCONFIGUREDBASED
ON HOSTNAME
Our machines have names like ip-172-26-5-123
How does Puppet know what type of machine th...
POTENTIAL WORKAROUNDS
Use an external node classifier
Use some mechanism for giving a better hostname, eg
web-172-26-5-123...
PUPPETMASTER ISASINGLE
POINTOFFAILURE
If the Puppetmaster fails, we can no longer autoscale
up
In particular, this could b...
POTENTIAL WORKAROUNDS
Clustered Puppetmasters
WORKAROUNDRECAP
Use/write alternative certificate management software
Write an external node classifier / mechanism for
se...
WHATWEDIDINSTEAD
Decided using a Puppetmaster was trying to fit a square
peg into a round hole
Instead, decided to run Pup...
APPLYINGLOCALPUPPET
MANIFESTS
puppet apply --modulepath=/etc/puppet/modules example.pp
DISTRIBUTINGMANIFESTS
Use RPM
Distribute full set of manifests/modules to each
machine
Apply only the manifest relevant to...
PACKINGPUPPET
MANIFESTSIN RPM
Build an RPM containing everything under /etc/puppet
Make files readable only by root
APPLYPUPPETMANIFESTS
Have an RPM %postinstcommand apply the Puppet
config
This isn't as straightforward as running the pup...
RPMINSTALLATION AND
MANAGEMENT
How do we get these RPMs on our machines?
PULP
We were already using Pulp
Provides yum repository management
Used for managing security updates and deploying
applic...
WHATISPULP
Repository manager
Allows us to easily audit what packages and versions
are installed where
Allows us to push p...
HOWWEUSEPULP
Puppet contains details of what packages should be
installed
Pulp manages which version of the package should...
DEPLOYING
CONFIGURATION ASCODE
Allows us to reuse our existing code deployment
infrastructure
Manage configuration deploym...
HOWWEDEPLOYCODE
Everything managed via the Jenkins continuous
integration server
Jenkins uses Pulp to install code on remo...
DETAILSON HOWWE
DEPLOYCODE
Jenkins fetches code from source control (git)
An RPM is built
Tests are run
If tests pass, the...
DEPLOYMENTLIFE-CYCLE
Jenkins also manages deployment life-cycle
RPMs are installed on staging
Promoted Builds plugin then ...
PUPPETDEPLOYMENT
PROCESS
Puppet manifests are checked into git
Lint tests via
Jenkins pulls in modules with librarian-pupp...
PUTTINGITINTO
PRODUCTION
Once suitable tests (automated and manual) have been
carried out, we promote Puppet config into p...
JENKINS:PROMOTION
EXCEPT..
How does a machine get from a bare image to the state
where we can push packages to it from Pulp?
How does a mach...
CLOUDFORMATION
Amazon tool for specifying infrastructure
Everything* we provision inside AWS is provisioned via
CloudForma...
CLOUD-INIT
Works with multiple cloud types
Sorts out things like SSH keys, allows us to configure
host names
Also allows u...
PROVISIONINGABARE
INSTANCE
cloud-init automatically manually adds the pulp repo
which contains Pulp, Puppet and our Puppet...
HOWDOESITKNOWWHAT
TYPEOFMACHINEITIS?
We tell it!
Use an environmental variable $HOSTTYPE
Simply run
puppet apply 
--module...
EXTRAFACTS
Custom facter facts
Also specified in an environmental variable
Data comes from within the CloudFormation templ...
OTHER RESOURCES
We either:
Provide details as a facter fact
`FACTER_DBHOST=xyz
Also use this approach to limit distributio...
FREEWINS!
FREEWINS!
Greater control over the timing of Puppet runs
Improved visibility - for ops and devs
Configuration changes now ...
MOREFREEWINS!
Puppet configs now have a version
Easy to find config version on the machine itself
Config changelogs access...
THEDOWNSIDES
Puppet manifests and modules on all machines
Potentially a security issue?
Mitigated by CloudFormation holdin...
ALTERNATIVE
IMPLEMENTATIONS
Don't want to use Pulp?
Could do basically the same thing with yum s3 plugin
Use mcollective t...
FUTUREIMPROVEMENTS
Build AMIs using Packer instead of configuring at boot
time
Decrease time to autoscale
Would probably s...
QUESTIONS? COMMENTS?
Sam Bashton
sam@bashton.com
Twitter: @bashtoni
(Psst.. )http://www.bashton.com/jobs/
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
Upcoming SlideShare
Loading in...5
×

Continuously-Integrated Puppet in a Dynamic Environment

10,382

Published on

This talk will show how we deploy Puppet without a Puppetmaster on an autoscaling Amazon Web Services infrastructure. Key points of interest: - Masterless Puppet - Use of Jenkins for Puppet manifest testing and environment promotion (test->staging->production) - Puppet integration with Amazon CloudFormation

Sam Bashton
Director, Bashton Ltd
After working for a number of Internet Service Providers, Sam founded Bashton Ltd in 2004. Focussing exclusively on Linux and Open Source software, Sam and his team provide consultancy, support and 24/7 infrastructure management for a number of high-traffic websites. A serial early adopter, Sam has travelled the world providing training and consultancy and generally spreading the Open Source message. Sam lives in Manchester, UK.

0 Comments
16 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
10,382
On Slideshare
0
From Embeds
0
Number of Embeds
17
Actions
Shares
0
Downloads
115
Comments
0
Likes
16
Embeds 0
No embeds

No notes for slide

Continuously-Integrated Puppet in a Dynamic Environment

  1. 1. CONTINUOUSLY INTEGRATEDPUPPETIN A DYNAMIC ENVIRONMENT SAM BASHTON, BASHTON LTD
  2. 2. CONTINUOUSLY INTEGRATEDPUPPETIN A DYNAMIC ENVIRONMENT MASTERLESSPUPPET:WHY ANDHOW SAM BASHTON, BASHTON LTD
  3. 3. ABOUTME Linux guy since Slackware, floppy disks and root + boot Using Puppet since 2007 Run a company in Manchester, North West England We provide outsourced ops for other companies
  4. 4. OUR FULLYMANAGED ENVIRONMENTS Primarily transactional websites (e-commerce) Majority (70%+) on Amazon Web Services (AWS) Majority using CentOS
  5. 5. HOWWEWORK Simple is better than complex Complexity is worth adding only if it provides obvious functional benefits Re-usability Resilience
  6. 6. WHY DID WE PICK AWS? Featureset and toolset massively in advance of any other cloud provider, public or private #1 customer reason for switching to AWS? The ability to scale on demand
  7. 7. TOOLSWEUSEFOR BUILDINGANDMANAGING Do one thing and do it well CloudFormation - Amazon tool to manage infrastructure Puppet - Manage system configuration Pulp - centralised repository, manages package revisions Jenkins
  8. 8. HOWWEUSEPUPPET No Puppetmaster Puppet manifests, hieradata and modules distributed to all machines via RPM All machines boot with a common, blank image and get configured at first boot
  9. 9. WHAT'SWRONGWITH MASTER BASEDPUPPET? Pets vs Cattle Puppet designed for a world of servers as pets We do not live in that world
  10. 10. PUPPETDESIGNEDFOR PETS Many assumptions in Puppet presume that your servers are pets Some of these work against us when managing a herd
  11. 11. MANUALCERTIFICATE SIGNING Clearly unsuitable when machines are automatically provisioned
  12. 12. POTENTIAL WORKAROUNDS: Autosign Use/write another automated certificate generation mechanism Possibly tied in with autoscaling
  13. 13. NO MECHANISMFOR CLEANINGOLDHOSTS Likely to have host-names reused, causing machines to fail to configure Puppetmaster will fill with certificates for machines that ran for a few hours and went away again
  14. 14. POTENTIAL WORKAROUNDS: Use UUID certificates Agree not to look in the certificate directory Write mechanism for cleaning up old certificates
  15. 15. HOSTSCONFIGUREDBASED ON HOSTNAME Our machines have names like ip-172-26-5-123 How does Puppet know what type of machine this is?
  16. 16. POTENTIAL WORKAROUNDS Use an external node classifier Use some mechanism for giving a better hostname, eg web-172-26-5-123 and use regex for node names
  17. 17. PUPPETMASTER ISASINGLE POINTOFFAILURE If the Puppetmaster fails, we can no longer autoscale up In particular, this could be a problem if there is availability zone failure
  18. 18. POTENTIAL WORKAROUNDS Clustered Puppetmasters
  19. 19. WORKAROUNDRECAP Use/write alternative certificate management software Write an external node classifier / mechanism for setting hostname appropriately Cluster multiple Puppetmasters
  20. 20. WHATWEDIDINSTEAD Decided using a Puppetmaster was trying to fit a square peg into a round hole Instead, decided to run Puppet without a master
  21. 21. APPLYINGLOCALPUPPET MANIFESTS puppet apply --modulepath=/etc/puppet/modules example.pp
  22. 22. DISTRIBUTINGMANIFESTS Use RPM Distribute full set of manifests/modules to each machine Apply only the manifest relevant to that machine
  23. 23. PACKINGPUPPET MANIFESTSIN RPM Build an RPM containing everything under /etc/puppet Make files readable only by root
  24. 24. APPLYPUPPETMANIFESTS Have an RPM %postinstcommand apply the Puppet config This isn't as straightforward as running the puppet applyfrom %postinst Puppet needs to install packages via yum, but yum is running installing the Puppet package Instead, we work around with a dirty hack: have the %postinstcreate an atscript which checks if yum has finished and then runs the puppet apply
  25. 25. RPMINSTALLATION AND MANAGEMENT How do we get these RPMs on our machines?
  26. 26. PULP We were already using Pulp Provides yum repository management Used for managing security updates and deploying application code http://pulpproject.org/
  27. 27. WHATISPULP Repository manager Allows us to easily audit what packages and versions are installed where Allows us to push package installations Uses qpid message queue Has concept of 'content distrubtion servers' for easy replication and clustering
  28. 28. HOWWEUSEPULP Puppet contains details of what packages should be installed Pulp manages which version of the package should be installed Pulp allows us to clone repos and copy packages between them for easy qa->stage->live environment management
  29. 29. DEPLOYING CONFIGURATION ASCODE Allows us to reuse our existing code deployment infrastructure Manage configuration deployment from Jenkins
  30. 30. HOWWEDEPLOYCODE Everything managed via the Jenkins continuous integration server Jenkins uses Pulp to install code on remote machines
  31. 31. DETAILSON HOWWE DEPLOYCODE Jenkins fetches code from source control (git) An RPM is built Tests are run If tests pass, the RPM is added to the relevant Pulp repository RPM installed on the target machine(s)
  32. 32. DEPLOYMENTLIFE-CYCLE Jenkins also manages deployment life-cycle RPMs are installed on staging Promoted Builds plugin then used to install the same RPMs on live once testing is complete
  33. 33. PUPPETDEPLOYMENT PROCESS Puppet manifests are checked into git Lint tests via Jenkins pulls in modules with librarian-puppet, then builds an RPM Deployment to test environments, functional tests for wider code-base run Jenkins Warnings plugin
  34. 34. PUTTINGITINTO PRODUCTION Once suitable tests (automated and manual) have been carried out, we promote Puppet config into production We use the Jenkins 'Promoted Builds' plugin for this
  35. 35. JENKINS:PROMOTION
  36. 36. EXCEPT.. How does a machine get from a bare image to the state where we can push packages to it from Pulp? How does a machine know what type of machine it is? How do we find other resources, eg database hostname?
  37. 37. CLOUDFORMATION Amazon tool for specifying infrastructure Everything* we provision inside AWS is provisioned via CloudFormation JSON templates * Everything except for the things Amazon doesn't expose via CloudFormation..
  38. 38. CLOUD-INIT Works with multiple cloud types Sorts out things like SSH keys, allows us to configure host names Also allows us to provide a bash script to run on startup
  39. 39. PROVISIONINGABARE INSTANCE cloud-init automatically manually adds the pulp repo which contains Pulp, Puppet and our Puppet manifests/modules Installs appropriate RPMs Puppet runs, subscribing the machine to the relevant Pulp repos, and installing packages in the usual Puppet way
  40. 40. HOWDOESITKNOWWHAT TYPEOFMACHINEITIS? We tell it! Use an environmental variable $HOSTTYPE Simply run puppet apply --modulepath=/etc/puppet/modules ${HOSTTYPE}.pp
  41. 41. EXTRAFACTS Custom facter facts Also specified in an environmental variable Data comes from within the CloudFormation template On our list of things to look at: FACTER_HOSTENVIRONMENT=live FACTER_STACKNAME=customer-web-live https://github.com/fanduel/hiera-cloudformation
  42. 42. OTHER RESOURCES We either: Provide details as a facter fact `FACTER_DBHOST=xyz Also use this approach to limit distribution of secure details, eg DB passwords Discover via the EC2 API Eg Varnish servers discover web backends by calling API and finding hosts tagged appropriately
  43. 43. FREEWINS!
  44. 44. FREEWINS! Greater control over the timing of Puppet runs Improved visibility - for ops and devs Configuration changes now have to be deployed to testing/staging first
  45. 45. MOREFREEWINS! Puppet configs now have a version Easy to find config version on the machine itself Config changelogs accessible on every machine (Git changelog added to RPM)
  46. 46. THEDOWNSIDES Puppet manifests and modules on all machines Potentially a security issue? Mitigated by CloudFormation holding most sensitive data
  47. 47. ALTERNATIVE IMPLEMENTATIONS Don't want to use Pulp? Could do basically the same thing with yum s3 plugin Use mcollective to push package updates https://github.com/jbraeuer/yum-s3-plugin
  48. 48. FUTUREIMPROVEMENTS Build AMIs using Packer instead of configuring at boot time Decrease time to autoscale Would probably still need to run Puppet at first boot to configure machine specific settings
  49. 49. QUESTIONS? COMMENTS? Sam Bashton sam@bashton.com Twitter: @bashtoni (Psst.. )http://www.bashton.com/jobs/
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×