Rapid scaling in_the_cloud_with_puppet

Rapid scaling and management in the cloud with puppet Carl Caum [email_address] @ccaum

What is Puppet?
Express infrastructure as....
code
Manage your infrastructure just like software
Use version control
QA changes
Continuous Integration
Prevent problems from re-occurring

What is Puppet?
code
resources
What, not how
Relationships not order

What is Puppet?
code
resources
state
Idempotent
What, not how

The Cycle

Resources
Resources are the building blocks of puppet
All resources have:
type
title
attributes
file { '/etc/motd' : ensure => file, owner => 'root', content => 'Managed by Puppet', mode => 0755, }

The Resource Abstraction Layer (RAL)
The Resource Abstraction Layer allows puppet to introspect the system about resource types.

Modules
Modules contain everything puppet needs to manage something.
For example:
apache
bacula
mysql
subversion
etc

Modules
Layout:
module_name
|
-- manifests (puppet code)
|
-- files (files to serve to clients)
|
-- templates (ERB templates) |
-- lib (puppet plugins)

Modules
Where do I get them?
The Forge
http://forge.puppetlabs.com
Github
http://github.com
Puppet Module Tool
# puppet-module install puppetlabs/apache

QA
Q: How do I QA my puppet code before pushing to production?

QA
Q: How do I QA my puppet code before pushing to production?
A : Environments!!

The Graph
Puppet uses a graph to know the relationship between resources
package { 'ntp':
ensure => present,
}
file { '/etc/ntp.conf':
owner => 'root',
group => 'root',
mode => '0644',
require => Package['ntp'],
}
service { 'ntpd':
ensure => running,
enable => true,
subscribe => File['/etc/ntp.conf'],
}

The Graph
Puppet uses a graph to know the relationship between resources

Puppet is highly customizable
Using Ruby, you can add custom.....
puppet subcommands
types/providers
facts
report processors

Custom Fact
Facter.add ( "role" ) do
setcode do
Facter::Util::Resolution.exec ( "cat /etc/role" )
end
end

Custom Report Processor
Puppet::Reports.register_report ( :autoami ) do
def process
.. do stuff ..
end
end

Cloud Provisioner Instant cloud management with puppet

Technologies
Puppet Faces
A new API for creating Puppet subcommands and actions.
Introduced in 2.7.0
http://www.puppetlabs.com/faces/
Fog
Ruby gem designed to control a variety of cloud services
through a unified API.
https://github.com/geemus/fog

Simplicity and Scriptability
Create a new instance from an AMI
Install Puppet (from community packages or Puppet Enterprise installer)
Generate and sign SSL cert for new agent on master node
# puppet node bootstrap
--image ami-d812efb1
--keyname my_keyname
--type m1.small
--login root
--keyfile /path/to/my_keypair.pem
--node-group webserver
--server puppet.domain.com
--enc-ssl --enc-auth-user console
--enc-auth-passwd console_pass
--enc-port 443

List nodes instances
# puppet node_aws list
i-d22612b2:
created_at: Wed Oct 12 16:50:02 UTC 2011
dns_name: ec2-184-73-33-225.compute-1.amazonaws.com
id: i-d22612b2
state: running
i-f1b54b92:
created_at: Wed Oct 26 13:46:44 UTC 2011
dns_name: ec2-174-129-228-163.compute-1.amazonaws.com
id: i-f1b54b92
state: running

Destroy instances
# puppet node terminate ec2-75-101-181-145.compute-1.amazonaws.com

require 'puppet'
require 'puppet/face'
opts = { :image => 'ami-d812efb1',
:keyname => 'my_keypair',
:type => 'm1.small',
:login => 'root',
:keyfile => '/path/to/my_keyfile.pem,
:server => 'puppet.mydomain.com'
}
Puppet::Face[:node_aws, '0.0.1'].bootstrap(opts)

AMI Management
This is a technique, not a tool!

AMI Management
Not fun
No, really. It sucks

AMI Management
Not fun
Difficult to know when to update
Usually requires a human to kick off a process

AMI Management
Not fun
Needs to be registered with load balancer
Whoever/whatever updates the AMI needs to register the new AMI with the load balancer and/or auto scaler

AMI Management
Not fun
Needs to be registered with load balancer
Necessary?
Many choose to just have puppet always configure a stock AMI

Initial Puppet Run
Can be slow if...
You haven't updated your AMIs in a while
You're running on a stock AMI to prevent image management

Best of Both Worlds
Use Cloud Provisioner to spawn new instances of AMIs you want to manage
Use Puppet custom report processors to detect if anything changed
Use custom face to snapshot instances if anything changes. Build new AMI off of snapshot
Register new AMI with load balancer and delete old one

EBS backed images (Elastic Block Store)
Persistent
Allows for snapshots
Many public ones available

Autoami
Module containing custom face to manage images and custom report processor
http://github.com/ccaum/puppet-autoami

Autoami
Steps:
Manually release production puppet code in VCS (git/svn)

Autoami
Steps:
Launch an instance of every AMI you want to manage

Autoami
Steps:
Record the certificate name

Autoami
Steps:
Classify the instance with Console

Autoami
Steps:
Sign the certificate

Autoami
Steps:
Wait for instance to report

Autoami
Steps:
If changes occurred (and were successful), generate new AMI

Autoami
Steps:
If changes occurred (and were successful), generate new AMI
Record AMI for load balancers

Autoami
Custom Report Processor
Puppet::Reports.register_report ( :autoami ) do
def process
.. do stuff ..
end
end

Autoami

Rapid scaling in_the_cloud_with_puppet

by Carl Caum on Feb 07, 2012

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Rapid scaling in_the_cloud_with_puppet — Presentation Transcript