Rapid scaling in_the_cloud_with_puppet

2,143 views

Published on

  • Be the first to comment

Rapid scaling in_the_cloud_with_puppet

  1. 1. Rapid scaling and management in the cloud with puppet Carl Caum [email_address] @ccaum
  2. 2. What is Puppet? <ul><li>Express infrastructure as.... </li></ul><ul><ul><li>code </li></ul></ul><ul><ul><ul><li>Manage your infrastructure just like software </li></ul></ul></ul><ul><ul><ul><li>Use version control </li></ul></ul></ul><ul><ul><ul><li>QA changes </li></ul></ul></ul><ul><ul><ul><li>Continuous Integration </li></ul></ul></ul><ul><ul><ul><li>Prevent problems from re-occurring  </li></ul></ul></ul>
  3. 3. What is Puppet? <ul><li>Express infrastructure as.... </li></ul><ul><ul><li>code </li></ul></ul><ul><ul><li>resources </li></ul></ul><ul><ul><ul><li>What, not how </li></ul></ul></ul><ul><ul><ul><li>Relationships not order </li></ul></ul></ul>
  4. 4. What is Puppet? <ul><li>Express infrastructure as.... </li></ul><ul><ul><li>code </li></ul></ul><ul><ul><li>resources </li></ul></ul><ul><ul><li>state </li></ul></ul><ul><ul><ul><li>Idempotent </li></ul></ul></ul><ul><ul><ul><li>What, not how </li></ul></ul></ul>
  5. 5. The Cycle
  6. 6. The Cycle
  7. 7. Resources <ul><li>Resources are the building blocks of puppet </li></ul><ul><li>All resources have: </li></ul><ul><ul><li>type </li></ul></ul><ul><ul><li>title </li></ul></ul><ul><ul><li>attributes </li></ul></ul><ul><li>file  {  '/etc/motd' :     ensure   => file,     owner    => 'root',     content  => 'Managed by Puppet',     mode     => 0755, } </li></ul>
  8. 8. The Resource Abstraction Layer (RAL) <ul><li>The Resource Abstraction Layer allows puppet to introspect the system about resource types. </li></ul>
  9. 9. Modules <ul><li>Modules contain everything puppet needs to manage something.  </li></ul><ul><li>For example: </li></ul><ul><ul><li>apache </li></ul></ul><ul><ul><li>bacula </li></ul></ul><ul><ul><li>mysql </li></ul></ul><ul><ul><li>subversion </li></ul></ul><ul><ul><li>etc </li></ul></ul>
  10. 10. Modules <ul><li>Layout: </li></ul><ul><li>module_name </li></ul><ul><li>  | </li></ul><ul><li>  -- manifests (puppet code) </li></ul><ul><li>  | </li></ul><ul><li>  -- files          (files to serve to clients) </li></ul><ul><li>  | </li></ul><ul><li>  -- templates (ERB templates)   | </li></ul><ul><li>  -- lib             (puppet plugins) </li></ul>
  11. 11. Modules <ul><li>Where do I get them? </li></ul><ul><li>The Forge </li></ul><ul><li>http://forge.puppetlabs.com </li></ul><ul><li>Github </li></ul><ul><li>http://github.com </li></ul><ul><li>Puppet Module Tool </li></ul><ul><li>#  puppet-module install puppetlabs/apache </li></ul>
  12. 12. QA <ul><li>Q:  How do I QA my puppet code before pushing to production? </li></ul>
  13. 13. QA <ul><li>Q:  How do I QA my puppet code before pushing to production? </li></ul><ul><li>A : Environments!! </li></ul>
  14. 14. The Graph <ul><li>Puppet uses a graph to know the relationship between resources </li></ul><ul><li>package { 'ntp':  </li></ul><ul><li>  ensure => present,  </li></ul><ul><li>}  </li></ul><ul><li>file { '/etc/ntp.conf':  </li></ul><ul><li>  owner => 'root', </li></ul><ul><li>  group => 'root', </li></ul><ul><li>  mode => '0644', </li></ul><ul><li>  require => Package['ntp'],  </li></ul><ul><li>}  </li></ul><ul><li>service { 'ntpd':  </li></ul><ul><li>  ensure => running,  </li></ul><ul><li>  enable => true,  </li></ul><ul><li>  subscribe => File['/etc/ntp.conf'],  </li></ul><ul><li>} </li></ul>
  15. 15. The Graph <ul><li>Puppet uses a graph to know the relationship between resources </li></ul>
  16. 16. Puppet is highly customizable <ul><li>Using Ruby, you can add custom..... </li></ul><ul><ul><li>puppet subcommands </li></ul></ul><ul><ul><li>types/providers </li></ul></ul><ul><ul><li>facts </li></ul></ul><ul><ul><li>report processors </li></ul></ul>
  17. 17. Puppet is highly customizable <ul><li>Custom Fact </li></ul><ul><li>Facter.add ( &quot;role&quot; ) do   </li></ul><ul><li>  setcode do   </li></ul><ul><li>    Facter::Util::Resolution.exec ( &quot;cat /etc/role&quot; )  </li></ul><ul><li>  end  </li></ul><ul><li>end </li></ul>
  18. 18. Puppet is highly customizable <ul><li>Custom Report Processor </li></ul><ul><li>Puppet::Reports.register_report ( :autoami )  do </li></ul><ul><li>   def process </li></ul><ul><li>     .. do stuff .. </li></ul><ul><li>   end </li></ul><ul><li>end </li></ul>
  19. 19. Demo <ul><li>  </li></ul>
  20. 20. Cloud Provisioner Instant cloud management with puppet
  21. 21. Technologies <ul><li>Puppet Faces </li></ul><ul><li>     A new API for creating Puppet subcommands and actions. </li></ul><ul><li>     Introduced in 2.7.0   </li></ul><ul><li>     http://www.puppetlabs.com/faces/ </li></ul><ul><li>Fog </li></ul><ul><li>     Ruby gem designed to control a variety of cloud services  </li></ul><ul><li>     through a unified API. </li></ul><ul><li>     https://github.com/geemus/fog </li></ul><ul><li>     </li></ul>
  22. 22. Simplicity and Scriptability <ul><ul><li>Create a new instance from an AMI </li></ul></ul><ul><ul><li>Install Puppet (from community packages or Puppet Enterprise installer) </li></ul></ul><ul><ul><li>Generate and sign SSL cert for new agent on master node </li></ul></ul><ul><li># puppet node bootstrap  </li></ul><ul><li>  --image ami-d812efb1  </li></ul><ul><li>  --keyname my_keyname  </li></ul><ul><li>  --type m1.small  </li></ul><ul><li>  --login root  </li></ul><ul><li>  --keyfile /path/to/my_keypair.pem  </li></ul><ul><li>  --node-group webserver  </li></ul><ul><li>  --server puppet.domain.com  </li></ul><ul><li>  --enc-ssl --enc-auth-user console  </li></ul><ul><li>  --enc-auth-passwd console_pass  </li></ul><ul><li>  --enc-port 443 </li></ul>
  23. 23. Simplicity and Scriptability <ul><li>List nodes instances </li></ul><ul><li># puppet node_aws list </li></ul><ul><li>i-d22612b2: </li></ul><ul><li>  created_at: Wed Oct 12 16:50:02 UTC 2011 </li></ul><ul><li>  dns_name: ec2-184-73-33-225.compute-1.amazonaws.com </li></ul><ul><li>  id: i-d22612b2 </li></ul><ul><li>  state: running </li></ul><ul><li>i-f1b54b92: </li></ul><ul><li>  created_at: Wed Oct 26 13:46:44 UTC 2011 </li></ul><ul><li>  dns_name: ec2-174-129-228-163.compute-1.amazonaws.com </li></ul><ul><li>  id: i-f1b54b92 </li></ul><ul><li>  state: running </li></ul>
  24. 24. Simplicity and Scriptability <ul><li>Destroy instances </li></ul><ul><li># puppet node terminate ec2-75-101-181-145.compute-1.amazonaws.com </li></ul>
  25. 25. Simplicity and Scriptability <ul><li>require 'puppet' </li></ul><ul><li>require 'puppet/face' </li></ul><ul><li>opts = { :image      => 'ami-d812efb1',  </li></ul><ul><li>              :keyname => 'my_keypair',  </li></ul><ul><li>              :type        => 'm1.small',  </li></ul><ul><li>              :login       => 'root',  </li></ul><ul><li>              :keyfile     => '/path/to/my_keyfile.pem,  </li></ul><ul><li>              :server     => 'puppet.mydomain.com' </li></ul><ul><li>} </li></ul><ul><li>Puppet::Face[:node_aws, '0.0.1'].bootstrap(opts) </li></ul>
  26. 26. AMI Management <ul><li>This is a technique, not a tool! </li></ul>
  27. 27. AMI Management <ul><ul><li>Not fun </li></ul></ul><ul><ul><ul><li>No, really. It sucks </li></ul></ul></ul>
  28. 28. AMI Management <ul><ul><li>Not fun </li></ul></ul><ul><ul><li>Difficult to know when to update </li></ul></ul><ul><ul><ul><li>Usually requires a human to kick off a process </li></ul></ul></ul>
  29. 29. AMI Management <ul><ul><li>Not fun </li></ul></ul><ul><ul><li>Difficult to know when to update </li></ul></ul><ul><ul><li>Needs to be registered with load balancer </li></ul></ul><ul><ul><ul><li>Whoever/whatever updates the AMI needs to register the new AMI with the load balancer and/or auto scaler </li></ul></ul></ul>
  30. 30. AMI Management <ul><ul><li>Not fun </li></ul></ul><ul><ul><li>Difficult to know when to update </li></ul></ul><ul><ul><li>Needs to be registered with load balancer </li></ul></ul><ul><ul><li>Necessary? </li></ul></ul><ul><ul><ul><li>Many choose to just have puppet always configure a stock AMI </li></ul></ul></ul>
  31. 31. Initial Puppet Run <ul><li>Can be slow if... </li></ul><ul><ul><li>You haven't updated your AMIs in a while </li></ul></ul><ul><ul><li>You're running on a stock AMI to prevent image management </li></ul></ul>
  32. 32. Best of Both Worlds <ul><ul><li>Use Cloud Provisioner to spawn new instances of AMIs you want to manage </li></ul></ul><ul><ul><li>Use Puppet custom report processors to detect if anything changed </li></ul></ul><ul><ul><li>Use custom face to snapshot instances if anything changes. Build new AMI off of snapshot </li></ul></ul><ul><ul><li>Register new AMI with load balancer and delete old one </li></ul></ul>
  33. 33. EBS backed images (Elastic Block Store) <ul><ul><li>Persistent </li></ul></ul><ul><ul><li>Allows for snapshots </li></ul></ul><ul><ul><li>Many public ones available </li></ul></ul>
  34. 34. Autoami <ul><li>Module containing custom face to manage images and custom report processor </li></ul><ul><li>http://github.com/ccaum/puppet-autoami </li></ul>
  35. 35. Autoami <ul><li>Steps: </li></ul><ul><ul><li>Manually release production puppet code in VCS (git/svn) </li></ul></ul>
  36. 36. Autoami <ul><li>Steps: </li></ul><ul><ul><li>Manually release production puppet code in VCS (git/svn) </li></ul></ul><ul><ul><li>Launch an instance of every AMI you want to manage </li></ul></ul>
  37. 37. Autoami <ul><li>Steps: </li></ul><ul><ul><li>Manually release production puppet code in VCS (git/svn) </li></ul></ul><ul><ul><li>Launch an instance of every AMI you want to manage </li></ul></ul><ul><ul><li>Record the certificate name </li></ul></ul>
  38. 38. Autoami <ul><li>Steps: </li></ul><ul><ul><li>Manually release production puppet code in VCS (git/svn) </li></ul></ul><ul><ul><li>Launch an instance of every AMI you want to manage </li></ul></ul><ul><ul><li>Record the certificate name </li></ul></ul><ul><ul><li>Classify the instance with Console </li></ul></ul>
  39. 39. Autoami <ul><li>Steps: </li></ul><ul><ul><li>Manually release production puppet code in VCS (git/svn) </li></ul></ul><ul><ul><li>Launch an instance of every AMI you want to manage </li></ul></ul><ul><ul><li>Record the certificate name </li></ul></ul><ul><ul><li>Classify the instance with Console </li></ul></ul><ul><ul><li>Sign the certificate </li></ul></ul>
  40. 40. Autoami <ul><li>Steps: </li></ul><ul><ul><li>Manually release production puppet code in VCS (git/svn) </li></ul></ul><ul><ul><li>Launch an instance of every AMI you want to manage </li></ul></ul><ul><ul><li>Record the certificate name </li></ul></ul><ul><ul><li>Classify the instance with Console </li></ul></ul><ul><ul><li>Sign the certificate </li></ul></ul><ul><ul><li>Wait for instance to report </li></ul></ul>
  41. 41. Autoami <ul><li>Steps: </li></ul><ul><ul><li>Manually release production puppet code in VCS (git/svn) </li></ul></ul><ul><ul><li>Launch an instance of every AMI you want to manage </li></ul></ul><ul><ul><li>Record the certificate name </li></ul></ul><ul><ul><li>Classify the instance with Console </li></ul></ul><ul><ul><li>Sign the certificate </li></ul></ul><ul><ul><li>Wait for instance to report </li></ul></ul><ul><ul><li>If changes occurred (and were successful), generate new AMI </li></ul></ul>
  42. 42. Autoami <ul><li>Steps: </li></ul><ul><ul><li>Manually release production puppet code in VCS (git/svn) </li></ul></ul><ul><ul><li>Launch an instance of every AMI you want to manage </li></ul></ul><ul><ul><li>Record the certificate name </li></ul></ul><ul><ul><li>Classify the instance with Console </li></ul></ul><ul><ul><li>Sign the certificate </li></ul></ul><ul><ul><li>Wait for instance to report </li></ul></ul><ul><ul><li>If changes occurred (and were successful), generate new AMI </li></ul></ul><ul><ul><li>Record AMI for load balancers </li></ul></ul>
  43. 43. Autoami <ul><li>Custom Report Processor </li></ul><ul><li>Puppet::Reports.register_report ( :autoami ) do </li></ul><ul><li>  def process </li></ul><ul><li>    .. do stuff .. </li></ul><ul><li>  end </li></ul><ul><li>end </li></ul>
  44. 44. Autoami <ul><li>  </li></ul>
  45. 45. Demo <ul><li>  </li></ul>

×