Being a Puppet Master: Automating Amazon EC2 with Puppet & Friends

Being a Puppet MasterAutomating Amazon EC2 with Puppet & Friends Mark Stanislav <mark.stanislav@gmail.com>

Puppet: A Quick Overview

Puppet: A Quick OverviewStop administrating your environment and start developing it...

Puppet: A Quick OverviewStop administrating your environment and start developing it...Re-usable code for managing your software & configurations

Puppet: A Quick OverviewStop administrating your environment and start developing it...Re-usable code for managing your software & configurationsProvides a Domain Specific Language (DSL) to script with Classes, conditionals, selectors, variables, basic math, etc.

Puppet: A Quick OverviewStop administrating your environment and start developing it...Re-usable code for managing your software & configurationsProvides a Domain Specific Language (DSL) to script with Classes, conditionals, selectors, variables, basic math, etc.Supports Linux, Solaris, BSD, OS X; Windows in process!

Puppet: A Quick OverviewStop administrating your environment and start developing it...Re-usable code for managing your software & configurationsProvides a Domain Specific Language (DSL) to script with Classes, conditionals, selectors, variables, basic math, etc.Supports Linux, Solaris, BSD, OS X; Windows in process!Project ran by Luke Kanies; Founder/CEO of Puppet Labs $5M Series B in July 2010; ~$7M total funding

Puppet: A Quick OverviewStop administrating your environment and start developing it...Re-usable code for managing your software & configurationsProvides a Domain Specific Language (DSL) to script with Classes, conditionals, selectors, variables, basic math, etc.Supports Linux, Solaris, BSD, OS X; Windows in process!Project ran by Luke Kanies; Founder/CEO of Puppet Labs $5M Series B in July 2010; ~$7M total fundingCFEngine & Chef are similar projects; both are quality, too.

Puppet: A Quick OverviewStop administrating your environment and start developing it...Re-usable code for managing your software & configurationsProvides a Domain Specific Language (DSL) to script with Classes, conditionals, selectors, variables, basic math, etc.Supports Linux, Solaris, BSD, OS X; Windows in process!Project ran by Luke Kanies; Founder/CEO of Puppet Labs $5M Series B in July 2010; ~$7M total fundingCFEngine & Chef are similar projects; both are quality, too.Sun, Stanford, Match.com, Media Temple, & Digg all use it!

High-Level Puppet Overview

High-Level Puppet Overview Modules Puppet Master Configuration

High-Level Puppet Overview Modules Puppet Master Configuration Puppet Clients

High-Level Puppet Overview Modules Puppet Master Configuration General Cloud InfrastructureMonitoring DNS Syslog LDAP Nagios/Munin BIND Nameserver rsyslog Server OpenLDAP Server Puppet Clients

High-Level Puppet Overview Modules Puppet Master Configuration General Cloud Infrastructure Monitoring DNS Syslog LDAP Nagios/Munin BIND Nameserver rsyslog Server OpenLDAP Server Software Development EnvironmentsDevelopment Testing Review ProductionApache, Tomcat, Passenger Apache, Tomcat, Passenger Apache, Tomcat, Passenger Apache, Tomcat, Passenger Puppet Clients

Puppet Network Overview

Puppet Network Overview Puppet Master 8140/TCP 8139/TCP Client Initiated SSL Server Initiated puppetd -t puppetrun Puppet Clients

Puppet Network Overview Puppet Master 8140/TCP 8139/TCP Client Initiated SSL Server Initiated puppetd -t puppetrun Puppet ClientsConfiguration allows for manual synchronizations or a set increment

Puppet Network Overview Puppet Master 8140/TCP 8139/TCP Client Initiated SSL Server Initiated puppetd -t puppetrun Puppet ClientsConfiguration allows for manual synchronizations or a set incrementClient or server initiated synchronizations

Puppet Network Overview Puppet Master 8140/TCP 8139/TCP Client Initiated SSL Server Initiated puppetd -t puppetrun Puppet ClientsConfiguration allows for manual synchronizations or a set incrementClient or server initiated synchronizationsClient/Server configuration leverages a Certificate Authority (CA) on thePuppet Master to sign client certificates to verify authenticity

Puppet Network Overview Puppet Master 8140/TCP 8139/TCP Client Initiated SSL Server Initiated puppetd -t puppetrun Puppet ClientsConfiguration allows for manual synchronizations or a set incrementClient or server initiated synchronizationsClient/Server configuration leverages a Certificate Authority (CA) on thePuppet Master to sign client certificates to verify authenticityTransmissions of all data between a master & client are encrypted

Why EC2 IaaS is Tiring... =

Why EC2 IaaS is Tiring...An Amazon Machine Image(AMI) is very inflexible =

Why EC2 IaaS is Tiring...An Amazon Machine Image(AMI) is very inflexibleBuilding and deploying anew AMI is time consuming =

Why EC2 IaaS is Tiring...An Amazon Machine Image(AMI) is very inflexibleBuilding and deploying anew AMI is time consuming“What do you mean you =want to update a file? Wecan’t just do that...”

Why EC2 IaaS is Tiring...An Amazon Machine Image(AMI) is very inflexibleBuilding and deploying anew AMI is time consuming“What do you mean you =want to update a file? Wecan’t just do that...”Auto-scaling is fantastic butmanaging the scaling hostsis not

Why EC2 IaaS is Tiring...An Amazon Machine Image(AMI) is very inflexibleBuilding and deploying anew AMI is time consuming“What do you mean you =want to update a file? Wecan’t just do that...”Auto-scaling is fantastic butmanaging the scaling hostsis notTime to deploy & configureoffsets benefits of IaaS

Puppet is an EC2 Superhero

Puppet is an EC2 SuperheroDeployment of a “base” EC2AMI - just what you alwaysneed on any standard image

Puppet is an EC2 SuperheroDeployment of a “base” EC2AMI - just what you alwaysneed on any standard imageLeverage EC2 securitygroups to give context to anew instance

Puppet is an EC2 SuperheroDeployment of a “base” EC2AMI - just what you alwaysneed on any standard imageLeverage EC2 securitygroups to give context to anew instancePuppet knows what youwant out of the box;configure a new instancewithout interaction

Puppet is an EC2 SuperheroDeployment of a “base” EC2AMI - just what you alwaysneed on any standard imageLeverage EC2 securitygroups to give context to anew instancePuppet knows what youwant out of the box;configure a new instancewithout interactionUpdate a package orconfiguration file at anytime

New EC2 Puppet Client Flow

New EC2 Puppet Client FlowEC2 Instance WithPuppet Spawned

New EC2 Puppet Client FlowEC2 Instance With Puppet ServicePuppet Spawned Starts For First Time

New EC2 Puppet Client FlowEC2 Instance With Puppet Service Client GeneratesPuppet Spawned Starts For First Time SSL Certificate

New EC2 Puppet Client FlowEC2 Instance With Puppet Service Client GeneratesPuppet Spawned Starts For First Time SSL Certificate Client Sends SSL Certificate to Master

New EC2 Puppet Client FlowEC2 Instance With Puppet Service Client GeneratesPuppet Spawned Starts For First Time SSL Certificate Master Signs SSL Client Sends SSL Certificate Certificate to Master

New EC2 Puppet Client FlowEC2 Instance With Puppet Service Client GeneratesPuppet Spawned Starts For First Time SSL Certificate Puppet Client Master Signs SSL Client Sends SSL Synchronizes Certificate Certificate to Master

New EC2 Puppet Client Flow EC2 Instance With Puppet Service Client Generates Puppet Spawned Starts For First Time SSL Certificate Puppet Client Master Signs SSL Client Sends SSL Synchronizes Certificate Certificate to MasterMethods to Sign Client SSL Certificates:

New EC2 Puppet Client Flow EC2 Instance With Puppet Service Client Generates Puppet Spawned Starts For First Time SSL Certificate Puppet Client Master Signs SSL Client Sends SSL Synchronizes Certificate Certificate to MasterMethods to Sign Client SSL Certificates: Puppet Master can allow certain domain scopes (*.example.com) to be auto-signed when asked by a valid hostname

New EC2 Puppet Client Flow EC2 Instance With Puppet Service Client Generates Puppet Spawned Starts For First Time SSL Certificate Puppet Client Master Signs SSL Client Sends SSL Synchronizes Certificate Certificate to MasterMethods to Sign Client SSL Certificates: Puppet Master can allow certain domain scopes (*.example.com) to be auto-signed when asked by a valid hostname Create a crontab script that executes every minute looking for new SSL certificates in a certain directory and signs them

New EC2 Puppet Client Flow EC2 Instance With Puppet Service Client Generates Puppet Spawned Starts For First Time SSL Certificate Puppet Client Master Signs SSL Client Sends SSL Synchronizes Certificate Certificate to MasterMethods to Sign Client SSL Certificates: Puppet Master can allow certain domain scopes (*.example.com) to be auto-signed when asked by a valid hostname Create a crontab script that executes every minute looking for new SSL certificates in a certain directory and signs them Auto-sign everything you are asked to sign without question

New EC2 Puppet Client Flow EC2 Instance With Puppet Service Client Generates Puppet Spawned Starts For First Time SSL Certificate Puppet Client Master Signs SSL Client Sends SSL Synchronizes Certificate Certificate to MasterMethods to Sign Client SSL Certificates: Puppet Master can allow certain domain scopes (*.example.com) to be auto-signed when asked by a valid hostname Create a crontab script that executes every minute looking for new SSL certificates in a certain directory and signs them Auto-sign everything you are asked to sign without question Manually sign each certificate when you add a new Puppet Client

Puppet Module Structure

Puppet Module Structure Module Folder

Puppet Module Structure Module Foldermanifests/ Tell the module how to work

Puppet Module Structure Module Foldermanifests/ Tell the module how to work files/ Static files needed for deployment

Puppet Module Structure Module Foldermanifests/ Tell the module how to work files/ Static files needed for deploymenttemplates/ Dynamic Ruby-based templates

Puppet Module Structure Module Foldermanifests/ Tell the module how to work files/ Static files needed for deploymenttemplates/ Dynamic Ruby-based templates lib/ Relevant Ruby-based libraries

A Partial List of Puppet ‘types’

A Partial List of Puppet ‘types’Files & Directories

A Partial List of Puppet ‘types’Files & DirectoriesUsers & Groups

A Partial List of Puppet ‘types’Files & DirectoriesUsers & GroupsServices

A Partial List of Puppet ‘types’Files & DirectoriesUsers & GroupsServicesPackages

A Partial List of Puppet ‘types’Files & Directories CrontabsUsers & GroupsServicesPackages

A Partial List of Puppet ‘types’Files & Directories CrontabsUsers & Groups /etc/hostsServicesPackages

A Partial List of Puppet ‘types’Files & Directories CrontabsUsers & Groups /etc/hostsServices Mail AliasesPackages

A Partial List of Puppet ‘types’Files & Directories CrontabsUsers & Groups /etc/hostsServices Mail AliasesPackages Mount Points

A Partial List of Puppet ‘types’Files & Directories Crontabs NagiosUsers & Groups /etc/hostsServices Mail AliasesPackages Mount Points

A Partial List of Puppet ‘types’Files & Directories Crontabs NagiosUsers & Groups /etc/hosts SELinuxServices Mail AliasesPackages Mount Points

A Partial List of Puppet ‘types’Files & Directories Crontabs NagiosUsers & Groups /etc/hosts SELinuxServices Mail Aliases SSH KeysPackages Mount Points

A Partial List of Puppet ‘types’Files & Directories Crontabs NagiosUsers & Groups /etc/hosts SELinuxServices Mail Aliases SSH KeysPackages Mount Points Yum Repos

A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Supports 23 different package providers

A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Supports 23 different package providers Abstracted for your OS automatically

A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Supports 23 different package providers Abstracted for your OS automatically Specify ‘installed’, ‘absent’, or ‘latest’ for desired state

A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Supports 23 different package providers Abstracted for your OS automatically Specify ‘installed’, ‘absent’, or ‘latest’ for desired state Change from ‘installed’ to ‘latest’ and deploy for quick

A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Services: Supports 23 different Supports 10 different package providers ‘init’ frameworks Abstracted for your OS automatically Specify ‘installed’, ‘absent’, or ‘latest’ for desired state Change from ‘installed’ to ‘latest’ and deploy for quick

A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Services: Supports 23 different Supports 10 different package providers ‘init’ frameworks Abstracted for your Control whether a OS automatically service starts on boot or is required to Specify ‘installed’, be running always ‘absent’, or ‘latest’ for desired state Change from ‘installed’ to ‘latest’ and deploy for quick

A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Services: Supports 23 different Supports 10 different package providers ‘init’ frameworks Abstracted for your Control whether a OS automatically service starts on boot or is required to Specify ‘installed’, be running always ‘absent’, or ‘latest’ for desired state A service can be notified to restart if a Change from configuration file has ‘installed’ to ‘latest’ and deploy for quick

A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Services: Files/Directories: Supports 23 different Supports 10 different Specify ownership & package providers ‘init’ frameworks permissions Abstracted for your Control whether a OS automatically service starts on boot or is required to Specify ‘installed’, be running always ‘absent’, or ‘latest’ for desired state A service can be notified to restart if a Change from configuration file has ‘installed’ to ‘latest’ and deploy for quick

A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Services: Files/Directories: Supports 23 different Supports 10 different Specify ownership & package providers ‘init’ frameworks permissions Abstracted for your Control whether a Load content from OS automatically service starts on ‘files/’, ‘templates/’ boot or is required to or custom strings Specify ‘installed’, be running always ‘absent’, or ‘latest’ for desired state A service can be notified to restart if a Change from configuration file has ‘installed’ to ‘latest’ and deploy for quick

A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Services: Files/Directories: Supports 23 different Supports 10 different Specify ownership & package providers ‘init’ frameworks permissions Abstracted for your Control whether a Load content from OS automatically service starts on ‘files/’, ‘templates/’ boot or is required to or custom strings Specify ‘installed’, be running always ‘absent’, or ‘latest’ Create symlinks for desired state A service can be notified to restart if a Change from configuration file has ‘installed’ to ‘latest’ and deploy for quick

A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Services: Files/Directories: Supports 23 different Supports 10 different Specify ownership & package providers ‘init’ frameworks permissions Abstracted for your Control whether a Load content from OS automatically service starts on ‘files/’, ‘templates/’ boot or is required to or custom strings Specify ‘installed’, be running always ‘absent’, or ‘latest’ Create symlinks for desired state A service can be notified to restart if a Supports 5 types to Change from configuration file has verify a file checksum ‘installed’ to ‘latest’ and deploy for quick

A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Services: Files/Directories: Supports 23 different Supports 10 different Specify ownership & package providers ‘init’ frameworks permissions Abstracted for your Control whether a Load content from OS automatically service starts on ‘files/’, ‘templates/’ boot or is required to or custom strings Specify ‘installed’, be running always ‘absent’, or ‘latest’ Create symlinks for desired state A service can be notified to restart if a Supports 5 types to Change from configuration file has verify a file checksum ‘installed’ to ‘latest’ and deploy for quick Purge a directory of

General Puppet Syntax

General Puppet SyntaxClass Configuration: Single Class: class ntp { ... } Inherited Class: class sftp inherits ssh { ... } Nested Class: class foo { class bar { ... } } Scoped Class: class ntp::base { ... }

General Puppet SyntaxClass Configuration: Single Class: class ntp { ... } Inherited Class: class sftp inherits ssh { ... } Nested Class: class foo { class bar { ... } } Scoped Class: class ntp::base { ... }Selectors: $admin = $user_id ? { ‘0’ => ‘root’, }

General Puppet SyntaxClass Configuration: If-Else Conditionals: Single Class: if ($ec2_security_groups == ‘DNS’) { class ntp { ... } include bind::server Inherited Class: } else { include bind::client class sftp inherits ssh { ... } } Nested Class: class foo { class bar { ... } } Scoped Class: class ntp::base { ... }Selectors: $admin = $user_id ? { ‘0’ => ‘root’, }

General Puppet SyntaxClass Configuration: If-Else Conditionals: Single Class: if ($ec2_security_groups == ‘DNS’) { class ntp { ... } include bind::server Inherited Class: } else { include bind::client class sftp inherits ssh { ... } } Nested Class: Case Statements: class foo { case $ec2_security_groups { class bar { ... } Monitoring: { include nagios } Developer: { include mercurial } } } Scoped Class: class ntp::base { ... }Selectors: $admin = $user_id ? { ‘0’ => ‘root’, }

General Puppet SyntaxClass Configuration: If-Else Conditionals: Single Class: if ($ec2_security_groups == ‘DNS’) { class ntp { ... } include bind::server Inherited Class: } else { include bind::client class sftp inherits ssh { ... } } Nested Class: Case Statements: class foo { case $ec2_security_groups { class bar { ... } Monitoring: { include nagios } Developer: { include mercurial } } } Scoped Class: Set a Variable: class ntp::base { ... } $lib_path = “/usr/local/lib64/”Selectors: $admin = $user_id ? { ‘0’ => ‘root’, }

General Puppet SyntaxClass Configuration: If-Else Conditionals: Single Class: if ($ec2_security_groups == ‘DNS’) { class ntp { ... } include bind::server Inherited Class: } else { include bind::client class sftp inherits ssh { ... } } Nested Class: Case Statements: class foo { case $ec2_security_groups { class bar { ... } Monitoring: { include nagios } Developer: { include mercurial } } } Scoped Class: Set a Variable: class ntp::base { ... } $lib_path = “/usr/local/lib64/”Selectors: $admin = $user_id ? { Basic Math: ‘0’ => ‘root’, $file_size = $bytes * 1024 }

A Simple NTP Puppet Module

A Simple NTP Puppet Modulentpd/manifests/init.pp: class ntp { package { "ntp": ensure => latest } service { "ntpd": ensure => running, enable => true, hasrestart => true, hasstatus => true, require => Package["ntp"], } file { "/etc/ntp.conf": ensure => present, owner => root, group => root, mode => 0644, source => "puppet:///modules/ntp/ ntp.conf", notify => Service["ntpd"]; "/etc/sysconfig/ntpd": ensure => present, owner => root, group => root, mode => 0644, source => "puppet:///modules/ntp/ntpd", notify => Service["ntpd"]; } }

A Simple NTP Puppet Modulentpd/manifests/init.pp: ntpd/files/ntp.conf: class ntp { restrict default kod nomodify notrap nopeer noquery package { "ntp": ensure => latest } restrict 127.0.0.1 service { "ntpd": ensure => running, server nist.netservicesgroup.com enable => true, server time.nist.gov hasrestart => true, server time-a.nist.gov hasstatus => true, server time-b.nist.gov require => Package["ntp"], } server 127.127.1.0 fudge 127.127.1.0 stratum 10 file { "/etc/ntp.conf": driftfile /var/lib/ntp/drift ensure => present, owner => root, keys /etc/ntp/keys group => root, mode => 0644, source => "puppet:///modules/ntp/ ntp.conf", notify => Service["ntpd"]; "/etc/sysconfig/ntpd": ensure => present, owner => root, group => root, mode => 0644, source => "puppet:///modules/ntp/ntpd", notify => Service["ntpd"]; } }

A Simple NTP Puppet Modulentpd/manifests/init.pp: ntpd/files/ntp.conf: class ntp { restrict default kod nomodify notrap nopeer noquery package { "ntp": ensure => latest } restrict 127.0.0.1 service { "ntpd": ensure => running, server nist.netservicesgroup.com enable => true, server time.nist.gov hasrestart => true, server time-a.nist.gov hasstatus => true, server time-b.nist.gov require => Package["ntp"], } server 127.127.1.0 fudge 127.127.1.0 stratum 10 file { "/etc/ntp.conf": driftfile /var/lib/ntp/drift ensure => present, owner => root, keys /etc/ntp/keys group => root, mode => 0644, source => "puppet:///modules/ntp/ ntp.conf", ntpd/files/ntpd: notify => Service["ntpd"]; OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid" "/etc/sysconfig/ntpd": SYNC_HWCLOCK=yes ensure => present, NTPDATE_OPTIONS="-g -x" owner => root, group => root, mode => 0644, source => "puppet:///modules/ntp/ntpd", notify => Service["ntpd"]; } }

EC2 Security Group Magic

EC2 Security Group MagicEC2 security groups are anamed set of inboundfirewall rules for a giveninstance

EC2 Security Group MagicEC2 security groups are anamed set of inboundfirewall rules for a giveninstancePuppet can learn about EC2meta-data very easily

EC2 Security Group MagicEC2 security groups are anamed set of inboundfirewall rules for a giveninstancePuppet can learn about EC2meta-data very easilyTell Puppet to configureinstances based on theirsecurity group

EC2 Security Group MagicEC2 security groups are anamed set of inboundfirewall rules for a giveninstancePuppet can learn about EC2meta-data very easilyTell Puppet to configureinstances based on theirsecurity groupScales for 1 instance or 100

EC2 Security Group MagicEC2 security groups are anamed set of inboundfirewall rules for a giveninstancePuppet can learn about EC2meta-data very easilyTell Puppet to configureinstances based on theirsecurity groupScales for 1 instance or 100Rinse and repeat for eachservice group you have

EC2 Security Groups + Puppet

EC2 Security Groups + Puppet‘DNS’ EC2 Security Group:

EC2 Security Groups + Puppet‘DNS’ EC2 Security Group: Inbound Firewall Rules: 22/TCP for SSH for remote access 53/{TCP,UDP} for DNS nameserver

EC2 Security Groups + Puppet‘DNS’ EC2 Security Group: Inbound Firewall Rules: Puppet Modules Enabled: 22/TCP for SSH for remote access ssh - SSH server configuration 53/{TCP,UDP} for DNS nameserver bind - BIND nameserver

EC2 Security Groups + Puppet‘DNS’ EC2 Security Group: Inbound Firewall Rules: Puppet Modules Enabled: 22/TCP for SSH for remote access ssh - SSH server configuration 53/{TCP,UDP} for DNS nameserver bind - BIND nameserver The Puppet type ‘file’ allows for variable-replacement in filenames and use-on-first-match

EC2 Security Groups + Puppet‘DNS’ EC2 Security Group: Inbound Firewall Rules: Puppet Modules Enabled: 22/TCP for SSH for remote access ssh - SSH server configuration 53/{TCP,UDP} for DNS nameserver bind - BIND nameserver The Puppet type ‘file’ allows for variable-replacement in filenames and use-on-first-match file { "/etc/ssh/sshd_config": source => [ “puppet:///modules/ssh/{$ec2_security_groups}-sshd_config”, "puppet:///modules/ssh/sshd_config" ]; }

EC2 Security Groups + Puppet‘DNS’ EC2 Security Group: Inbound Firewall Rules: Puppet Modules Enabled: 22/TCP for SSH for remote access ssh - SSH server configuration 53/{TCP,UDP} for DNS nameserver bind - BIND nameserver The Puppet type ‘file’ allows for variable-replacement in filenames and use-on-first-match file { "/etc/ssh/sshd_config": source => [ “puppet:///modules/ssh/{$ec2_security_groups}-sshd_config”, "puppet:///modules/ssh/sshd_config" ]; } Puppet will use ‘DNS-sshd_config’ if it exists. If the file does not exist, it will use ‘sshd_config’

Client Meta-Data with Facter

Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!

Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!Examples:

Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!Examples: $architecture - Create files that are based on architecture

Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!Examples: $architecture - Create files that are based on architecture $hostname/$ip_address_eth0 - Create an /etc/hosts entry

Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!Examples: $architecture - Create files that are based on architecture $hostname/$ip_address_eth0 - Create an /etc/hosts entry $uptime_days - Update all packages after 30 days uptime

Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!Examples: $architecture - Create files that are based on architecture $hostname/$ip_address_eth0 - Create an /etc/hosts entry $uptime_days - Update all packages after 30 days uptime $selinux - Configure packages based on SELinux contexts

Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!Examples: $architecture - Create files that are based on architecture $hostname/$ip_address_eth0 - Create an /etc/hosts entry $uptime_days - Update all packages after 30 days uptime $selinux - Configure packages based on SELinux contexts $operatingsystemrelease - Run OS version specific tasks

Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!Examples: $architecture - Create files that are based on architecture $hostname/$ip_address_eth0 - Create an /etc/hosts entry $uptime_days - Update all packages after 30 days uptime $selinux - Configure packages based on SELinux contexts $operatingsystemrelease - Run OS version specific tasks $is_virtual - Configure hosts based on VM vs. Physical

Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!Examples: $architecture - Create files that are based on architecture $hostname/$ip_address_eth0 - Create an /etc/hosts entry $uptime_days - Update all packages after 30 days uptime $selinux - Configure packages based on SELinux contexts $operatingsystemrelease - Run OS version specific tasks $is_virtual - Configure hosts based on VM vs. Physical $ec2_ami_id - Update configuration for the EC2 AMI used

Nagios ‘Type’

Nagios ‘Type’Puppet natively supports creating Nagios configuration

Nagios ‘Type’Puppet natively supports creating Nagios configurationEasily generate specific configuration for n hosts automatically

Nagios ‘Type’Puppet natively supports creating Nagios configurationEasily generate specific configuration for n hosts automaticallyNever again manually include hosts/services in groupings

Nagios ‘Type’Puppet natively supports creating Nagios configurationEasily generate specific configuration for n hosts automaticallyNever again manually include hosts/services in groupingsNagios Service:@@nagios_service { "load_check_${hostname}": service_description => "Load Averages", check_command => "load_check!3!5", host_name => "$fqdn", use => "generic-service";}

Nagios ‘Type’Puppet natively supports creating Nagios configurationEasily generate specific configuration for n hosts automaticallyNever again manually include hosts/services in groupingsNagios Service:@@nagios_service { "load_check_${hostname}": service_description => "Load Averages", check_command => "load_check!3!5", host_name => "$fqdn", use => "generic-service";}Nagios Service Group:@@nagios_servicegroup { "apache_servers": alias => "Apache Servers";}

Nagios ‘Type’Puppet natively supports creating Nagios configurationEasily generate specific configuration for n hosts automaticallyNever again manually include hosts/services in groupingsNagios Service: Nagios Host:@@nagios_service { "load_check_${hostname}": @@nagios_host { $fqdn: service_description => "Load ensure => present, Averages", hostgroups => "ldap", check_command => "load_check!3!5", use => "generic-host"; host_name => "$fqdn", } use => "generic-service";}Nagios Service Group:@@nagios_servicegroup { "apache_servers": alias => "Apache Servers";}

Nagios ‘Type’Puppet natively supports creating Nagios configurationEasily generate specific configuration for n hosts automaticallyNever again manually include hosts/services in groupingsNagios Service: Nagios Host:@@nagios_service { "load_check_${hostname}": @@nagios_host { $fqdn: service_description => "Load ensure => present, Averages", hostgroups => "ldap", check_command => "load_check!3!5", use => "generic-host"; host_name => "$fqdn", } use => "generic-service";}Nagios Service Group: Nagios Host Group:@@nagios_servicegroup { @@nagios_hostgroup { "apache_servers": "load_balancers": alias => "Apache Servers"; alias => "Load Balancers";} }

Puppet Generated Host/Service Checks

Puppet Generated Munin Metrics/Groupings

The Foreman: A Heavy Lifter

The Foreman: A Heavy LifterEasy-to-use Puppet webinterface for many tasks

The Foreman: A Heavy LifterEasy-to-use Puppet webinterface for many tasksReview Puppet reportsregarding your hosts easily

The Foreman: A Heavy LifterEasy-to-use Puppet webinterface for many tasksReview Puppet reportsregarding your hosts easilyEdit host facts and groups

The Foreman: A Heavy LifterEasy-to-use Puppet webinterface for many tasksReview Puppet reportsregarding your hosts easilyEdit host facts and groupsLDAP authentication

The Foreman: A Heavy LifterEasy-to-use Puppet webinterface for many tasksReview Puppet reportsregarding your hosts easilyEdit host facts and groupsLDAP authenticationStatistical graphs for metrics

The Foreman: A Heavy LifterEasy-to-use Puppet webinterface for many tasksReview Puppet reportsregarding your hosts easilyEdit host facts and groupsLDAP authenticationStatistical graphs for metricsExecute puppetrun on hosts

The Foreman: A Heavy LifterEasy-to-use Puppet webinterface for many tasksReview Puppet reportsregarding your hosts easilyEdit host facts and groupsLDAP authenticationStatistical graphs for metricsExecute puppetrun on hostsProvision hosts from the web

The Foreman ‘Overview’ Page

Interact with ‘Facter Facts’

Evaluate Puppet Efficiency with Reports

General Statistics for Puppet Clients

Marionette Collective

Marionette CollectiveManage/Control/Execute: Services Packages Process Information Facter Facts Pings

Marionette CollectiveManage/Control/Execute: Services Packages Process Information Facter Facts PingsDecide which hosts you actupon by any Facter Fact

Marionette CollectiveManage/Control/Execute: Services Packages Process Information Facter Facts PingsDecide which hosts you actupon by any Facter FactEasily manage a largeamount of diverse hosts

View Any Service’s Status Across Hosts

Check Versions That Are Installed

View Processes On Hosts Matching a ‘Fact’

Quickly Retrieve a List of MCollective Hosts

Consider This Scenario

Consider This Scenario1. You reserve 10 Elastic IPs for a network of hosts

Consider This Scenario1. You reserve 10 Elastic IPs for a network of hosts2. Each instance starts and Puppet gives it an elastic IP

Consider This Scenario1. You reserve 10 Elastic IPs for a network of hosts2. Each instance starts and Puppet gives it an elastic IP3. Based on an ‘IP -> NEED’ map, each new instance is created for a specific need (DNS, WWW, IMAP, etc.)

Consider This Scenario1. You reserve 10 Elastic IPs for a network of hosts2. Each instance starts and Puppet gives it an elastic IP3. Based on an ‘IP -> NEED’ map, each new instance is created for a specific need (DNS, WWW, IMAP, etc.)4. Hosts that become ‘WWW’ servers automatically are added to the Elastic Load Balancer (ELB) instance

Consider This Scenario1. You reserve 10 Elastic IPs for a network of hosts2. Each instance starts and Puppet gives it an elastic IP3. Based on an ‘IP -> NEED’ map, each new instance is created for a specific need (DNS, WWW, IMAP, etc.)4. Hosts that become ‘WWW’ servers automatically are added to the Elastic Load Balancer (ELB) instance5. Nagios & Munin configuration is done automatically

Consider This Scenario1. You reserve 10 Elastic IPs for a network of hosts2. Each instance starts and Puppet gives it an elastic IP3. Based on an ‘IP -> NEED’ map, each new instance is created for a specific need (DNS, WWW, IMAP, etc.)4. Hosts that become ‘WWW’ servers automatically are added to the Elastic Load Balancer (ELB) instance5. Nagios & Munin configuration is done automatically6. If an instance dies, the next time a new instance starts it is given the old host’s IP and that service is fulfilled again

Consider This Scenario1. You reserve 10 Elastic IPs for a network of hosts2. Each instance starts and Puppet gives it an elastic IP3. Based on an ‘IP -> NEED’ map, each new instance is created for a specific need (DNS, WWW, IMAP, etc.)4. Hosts that become ‘WWW’ servers automatically are added to the Elastic Load Balancer (ELB) instance5. Nagios & Munin configuration is done automatically6. If an instance dies, the next time a new instance starts it is given the old host’s IP and that service is fulfilled again ...most importantly, you’ve done nothing :)

Take Your Environment

Take Your EnvironmentPuppet: Provides you with the means to handle ad-hocEC2 instance scaling with granular updates/configurationchanges based on any ‘Fact’ you can supply.

Take Your EnvironmentPuppet: Provides you with the means to handle ad-hocEC2 instance scaling with granular updates/configurationchanges based on any ‘Fact’ you can supply.The Foreman: Manage your hosts from a well designedfront-end. View reports, check for deployment efficiency,get the ‘big picture’ on your infrastructure; even deployhosts from scratch!

Take Your EnvironmentPuppet: Provides you with the means to handle ad-hocEC2 instance scaling with granular updates/configurationchanges based on any ‘Fact’ you can supply.The Foreman: Manage your hosts from a well designedfront-end. View reports, check for deployment efficiency,get the ‘big picture’ on your infrastructure; even deployhosts from scratch!MCollective: Handle your mass administrative tasks withconsistency and structure. Utilize ‘Facter’ to intelligentlyexecute tasks only against certain sub-sets of hosts.

Take Your EnvironmentPuppet: Provides you with the means to handle ad-hocEC2 instance scaling with granular updates/configurationchanges based on any ‘Fact’ you can supply.The Foreman: Manage your hosts from a well designedfront-end. View reports, check for deployment efficiency,get the ‘big picture’ on your infrastructure; even deployhosts from scratch!MCollective: Handle your mass administrative tasks withconsistency and structure. Utilize ‘Facter’ to intelligentlyexecute tasks only against certain sub-sets of hosts.Nagios/Munin: Automatically deploy full monitoring &metrics for hosts without ever hand configuring a file.

Thanks! Questions?mark.stanislav@gmail.comuncompiled.com@markstanislav

http://tweetedtimes.com	12
http://twitter.com	10
https://twitter.com	6
http://us-w1.rockmelt.com	4
http://trunk.ly	3
http://a0.twimg.com	3
http://mstanislav.posterous.com	1

Being a Puppet Master: Automating Amazon EC2 with Puppet & Friends

by Mark Stanislav on May 18, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

7 Embeds 39

Statistics

Being a Puppet Master: Automating Amazon EC2 with Puppet & Friends — Presentation Transcript