Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Being a Puppet Master: Automating Amazon EC2 with Puppet & Friends

27,201 views

Published on

Being a Puppet Master: Automating Amazon EC2 with Puppet & Friends

  1. 1. Being a Puppet MasterAutomating Amazon EC2 with Puppet & Friends Mark Stanislav <mark.stanislav@gmail.com>
  2. 2. Puppet: A Quick Overview
  3. 3. Puppet: A Quick OverviewStop administrating your environment and start developing it...
  4. 4. Puppet: A Quick OverviewStop administrating your environment and start developing it...Re-usable code for managing your software & configurations
  5. 5. Puppet: A Quick OverviewStop administrating your environment and start developing it...Re-usable code for managing your software & configurationsProvides a Domain Specific Language (DSL) to script with Classes, conditionals, selectors, variables, basic math, etc.
  6. 6. Puppet: A Quick OverviewStop administrating your environment and start developing it...Re-usable code for managing your software & configurationsProvides a Domain Specific Language (DSL) to script with Classes, conditionals, selectors, variables, basic math, etc.Supports Linux, Solaris, BSD, OS X; Windows in process!
  7. 7. Puppet: A Quick OverviewStop administrating your environment and start developing it...Re-usable code for managing your software & configurationsProvides a Domain Specific Language (DSL) to script with Classes, conditionals, selectors, variables, basic math, etc.Supports Linux, Solaris, BSD, OS X; Windows in process!Project ran by Luke Kanies; Founder/CEO of Puppet Labs $5M Series B in July 2010; ~$7M total funding
  8. 8. Puppet: A Quick OverviewStop administrating your environment and start developing it...Re-usable code for managing your software & configurationsProvides a Domain Specific Language (DSL) to script with Classes, conditionals, selectors, variables, basic math, etc.Supports Linux, Solaris, BSD, OS X; Windows in process!Project ran by Luke Kanies; Founder/CEO of Puppet Labs $5M Series B in July 2010; ~$7M total fundingCFEngine & Chef are similar projects; both are quality, too.
  9. 9. Puppet: A Quick OverviewStop administrating your environment and start developing it...Re-usable code for managing your software & configurationsProvides a Domain Specific Language (DSL) to script with Classes, conditionals, selectors, variables, basic math, etc.Supports Linux, Solaris, BSD, OS X; Windows in process!Project ran by Luke Kanies; Founder/CEO of Puppet Labs $5M Series B in July 2010; ~$7M total fundingCFEngine & Chef are similar projects; both are quality, too.Sun, Stanford, Match.com, Media Temple, & Digg all use it!
  10. 10. High-Level Puppet Overview
  11. 11. High-Level Puppet Overview Modules Puppet Master Configuration
  12. 12. High-Level Puppet Overview Modules Puppet Master Configuration Puppet Clients
  13. 13. High-Level Puppet Overview Modules Puppet Master Configuration General Cloud InfrastructureMonitoring DNS Syslog LDAP Nagios/Munin BIND Nameserver rsyslog Server OpenLDAP Server Puppet Clients
  14. 14. High-Level Puppet Overview Modules Puppet Master Configuration General Cloud Infrastructure Monitoring DNS Syslog LDAP Nagios/Munin BIND Nameserver rsyslog Server OpenLDAP Server Software Development EnvironmentsDevelopment Testing Review ProductionApache, Tomcat, Passenger Apache, Tomcat, Passenger Apache, Tomcat, Passenger Apache, Tomcat, Passenger Puppet Clients
  15. 15. Puppet Network Overview
  16. 16. Puppet Network Overview Puppet Master 8140/TCP 8139/TCP Client Initiated SSL Server Initiated puppetd -t puppetrun Puppet Clients
  17. 17. Puppet Network Overview Puppet Master 8140/TCP 8139/TCP Client Initiated SSL Server Initiated puppetd -t puppetrun Puppet ClientsConfiguration allows for manual synchronizations or a set increment
  18. 18. Puppet Network Overview Puppet Master 8140/TCP 8139/TCP Client Initiated SSL Server Initiated puppetd -t puppetrun Puppet ClientsConfiguration allows for manual synchronizations or a set incrementClient or server initiated synchronizations
  19. 19. Puppet Network Overview Puppet Master 8140/TCP 8139/TCP Client Initiated SSL Server Initiated puppetd -t puppetrun Puppet ClientsConfiguration allows for manual synchronizations or a set incrementClient or server initiated synchronizationsClient/Server configuration leverages a Certificate Authority (CA) on thePuppet Master to sign client certificates to verify authenticity
  20. 20. Puppet Network Overview Puppet Master 8140/TCP 8139/TCP Client Initiated SSL Server Initiated puppetd -t puppetrun Puppet ClientsConfiguration allows for manual synchronizations or a set incrementClient or server initiated synchronizationsClient/Server configuration leverages a Certificate Authority (CA) on thePuppet Master to sign client certificates to verify authenticityTransmissions of all data between a master & client are encrypted
  21. 21. Why EC2 IaaS is Tiring... =
  22. 22. Why EC2 IaaS is Tiring...An Amazon Machine Image(AMI) is very inflexible =
  23. 23. Why EC2 IaaS is Tiring...An Amazon Machine Image(AMI) is very inflexibleBuilding and deploying anew AMI is time consuming =
  24. 24. Why EC2 IaaS is Tiring...An Amazon Machine Image(AMI) is very inflexibleBuilding and deploying anew AMI is time consuming“What do you mean you =want to update a file? Wecan’t just do that...”
  25. 25. Why EC2 IaaS is Tiring...An Amazon Machine Image(AMI) is very inflexibleBuilding and deploying anew AMI is time consuming“What do you mean you =want to update a file? Wecan’t just do that...”Auto-scaling is fantastic butmanaging the scaling hostsis not
  26. 26. Why EC2 IaaS is Tiring...An Amazon Machine Image(AMI) is very inflexibleBuilding and deploying anew AMI is time consuming“What do you mean you =want to update a file? Wecan’t just do that...”Auto-scaling is fantastic butmanaging the scaling hostsis notTime to deploy & configureoffsets benefits of IaaS
  27. 27. Puppet is an EC2 Superhero
  28. 28. Puppet is an EC2 SuperheroDeployment of a “base” EC2AMI - just what you alwaysneed on any standard image
  29. 29. Puppet is an EC2 SuperheroDeployment of a “base” EC2AMI - just what you alwaysneed on any standard imageLeverage EC2 securitygroups to give context to anew instance
  30. 30. Puppet is an EC2 SuperheroDeployment of a “base” EC2AMI - just what you alwaysneed on any standard imageLeverage EC2 securitygroups to give context to anew instancePuppet knows what youwant out of the box;configure a new instancewithout interaction
  31. 31. Puppet is an EC2 SuperheroDeployment of a “base” EC2AMI - just what you alwaysneed on any standard imageLeverage EC2 securitygroups to give context to anew instancePuppet knows what youwant out of the box;configure a new instancewithout interactionUpdate a package orconfiguration file at anytime
  32. 32. New EC2 Puppet Client Flow
  33. 33. New EC2 Puppet Client FlowEC2 Instance WithPuppet Spawned
  34. 34. New EC2 Puppet Client FlowEC2 Instance With Puppet ServicePuppet Spawned Starts For First Time
  35. 35. New EC2 Puppet Client FlowEC2 Instance With Puppet Service Client GeneratesPuppet Spawned Starts For First Time SSL Certificate
  36. 36. New EC2 Puppet Client FlowEC2 Instance With Puppet Service Client GeneratesPuppet Spawned Starts For First Time SSL Certificate Client Sends SSL Certificate to Master
  37. 37. New EC2 Puppet Client FlowEC2 Instance With Puppet Service Client GeneratesPuppet Spawned Starts For First Time SSL Certificate Master Signs SSL Client Sends SSL Certificate Certificate to Master
  38. 38. New EC2 Puppet Client FlowEC2 Instance With Puppet Service Client GeneratesPuppet Spawned Starts For First Time SSL Certificate Puppet Client Master Signs SSL Client Sends SSL Synchronizes Certificate Certificate to Master
  39. 39. New EC2 Puppet Client Flow EC2 Instance With Puppet Service Client Generates Puppet Spawned Starts For First Time SSL Certificate Puppet Client Master Signs SSL Client Sends SSL Synchronizes Certificate Certificate to MasterMethods to Sign Client SSL Certificates:
  40. 40. New EC2 Puppet Client Flow EC2 Instance With Puppet Service Client Generates Puppet Spawned Starts For First Time SSL Certificate Puppet Client Master Signs SSL Client Sends SSL Synchronizes Certificate Certificate to MasterMethods to Sign Client SSL Certificates: Puppet Master can allow certain domain scopes (*.example.com) to be auto-signed when asked by a valid hostname
  41. 41. New EC2 Puppet Client Flow EC2 Instance With Puppet Service Client Generates Puppet Spawned Starts For First Time SSL Certificate Puppet Client Master Signs SSL Client Sends SSL Synchronizes Certificate Certificate to MasterMethods to Sign Client SSL Certificates: Puppet Master can allow certain domain scopes (*.example.com) to be auto-signed when asked by a valid hostname Create a crontab script that executes every minute looking for new SSL certificates in a certain directory and signs them
  42. 42. New EC2 Puppet Client Flow EC2 Instance With Puppet Service Client Generates Puppet Spawned Starts For First Time SSL Certificate Puppet Client Master Signs SSL Client Sends SSL Synchronizes Certificate Certificate to MasterMethods to Sign Client SSL Certificates: Puppet Master can allow certain domain scopes (*.example.com) to be auto-signed when asked by a valid hostname Create a crontab script that executes every minute looking for new SSL certificates in a certain directory and signs them Auto-sign everything you are asked to sign without question
  43. 43. New EC2 Puppet Client Flow EC2 Instance With Puppet Service Client Generates Puppet Spawned Starts For First Time SSL Certificate Puppet Client Master Signs SSL Client Sends SSL Synchronizes Certificate Certificate to MasterMethods to Sign Client SSL Certificates: Puppet Master can allow certain domain scopes (*.example.com) to be auto-signed when asked by a valid hostname Create a crontab script that executes every minute looking for new SSL certificates in a certain directory and signs them Auto-sign everything you are asked to sign without question Manually sign each certificate when you add a new Puppet Client
  44. 44. Puppet Module Structure
  45. 45. Puppet Module Structure Module Folder
  46. 46. Puppet Module Structure Module Foldermanifests/ Tell the module how to work
  47. 47. Puppet Module Structure Module Foldermanifests/ Tell the module how to work files/ Static files needed for deployment
  48. 48. Puppet Module Structure Module Foldermanifests/ Tell the module how to work files/ Static files needed for deploymenttemplates/ Dynamic Ruby-based templates
  49. 49. Puppet Module Structure Module Foldermanifests/ Tell the module how to work files/ Static files needed for deploymenttemplates/ Dynamic Ruby-based templates lib/ Relevant Ruby-based libraries
  50. 50. A Partial List of Puppet ‘types’
  51. 51. A Partial List of Puppet ‘types’Files & Directories
  52. 52. A Partial List of Puppet ‘types’Files & DirectoriesUsers & Groups
  53. 53. A Partial List of Puppet ‘types’Files & DirectoriesUsers & GroupsServices
  54. 54. A Partial List of Puppet ‘types’Files & DirectoriesUsers & GroupsServicesPackages
  55. 55. A Partial List of Puppet ‘types’Files & Directories CrontabsUsers & GroupsServicesPackages
  56. 56. A Partial List of Puppet ‘types’Files & Directories CrontabsUsers & Groups /etc/hostsServicesPackages
  57. 57. A Partial List of Puppet ‘types’Files & Directories CrontabsUsers & Groups /etc/hostsServices Mail AliasesPackages
  58. 58. A Partial List of Puppet ‘types’Files & Directories CrontabsUsers & Groups /etc/hostsServices Mail AliasesPackages Mount Points
  59. 59. A Partial List of Puppet ‘types’Files & Directories Crontabs NagiosUsers & Groups /etc/hostsServices Mail AliasesPackages Mount Points
  60. 60. A Partial List of Puppet ‘types’Files & Directories Crontabs NagiosUsers & Groups /etc/hosts SELinuxServices Mail AliasesPackages Mount Points
  61. 61. A Partial List of Puppet ‘types’Files & Directories Crontabs NagiosUsers & Groups /etc/hosts SELinuxServices Mail Aliases SSH KeysPackages Mount Points
  62. 62. A Partial List of Puppet ‘types’Files & Directories Crontabs NagiosUsers & Groups /etc/hosts SELinuxServices Mail Aliases SSH KeysPackages Mount Points Yum Repos
  63. 63. A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Supports 23 different package providers
  64. 64. A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Supports 23 different package providers Abstracted for your OS automatically
  65. 65. A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Supports 23 different package providers Abstracted for your OS automatically Specify ‘installed’, ‘absent’, or ‘latest’ for desired state
  66. 66. A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Supports 23 different package providers Abstracted for your OS automatically Specify ‘installed’, ‘absent’, or ‘latest’ for desired state Change from ‘installed’ to ‘latest’ and deploy for quick
  67. 67. A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Services: Supports 23 different Supports 10 different package providers ‘init’ frameworks Abstracted for your OS automatically Specify ‘installed’, ‘absent’, or ‘latest’ for desired state Change from ‘installed’ to ‘latest’ and deploy for quick
  68. 68. A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Services: Supports 23 different Supports 10 different package providers ‘init’ frameworks Abstracted for your Control whether a OS automatically service starts on boot or is required to Specify ‘installed’, be running always ‘absent’, or ‘latest’ for desired state Change from ‘installed’ to ‘latest’ and deploy for quick
  69. 69. A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Services: Supports 23 different Supports 10 different package providers ‘init’ frameworks Abstracted for your Control whether a OS automatically service starts on boot or is required to Specify ‘installed’, be running always ‘absent’, or ‘latest’ for desired state A service can be notified to restart if a Change from configuration file has ‘installed’ to ‘latest’ and deploy for quick
  70. 70. A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Services: Files/Directories: Supports 23 different Supports 10 different Specify ownership & package providers ‘init’ frameworks permissions Abstracted for your Control whether a OS automatically service starts on boot or is required to Specify ‘installed’, be running always ‘absent’, or ‘latest’ for desired state A service can be notified to restart if a Change from configuration file has ‘installed’ to ‘latest’ and deploy for quick
  71. 71. A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Services: Files/Directories: Supports 23 different Supports 10 different Specify ownership & package providers ‘init’ frameworks permissions Abstracted for your Control whether a Load content from OS automatically service starts on ‘files/’, ‘templates/’ boot or is required to or custom strings Specify ‘installed’, be running always ‘absent’, or ‘latest’ for desired state A service can be notified to restart if a Change from configuration file has ‘installed’ to ‘latest’ and deploy for quick
  72. 72. A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Services: Files/Directories: Supports 23 different Supports 10 different Specify ownership & package providers ‘init’ frameworks permissions Abstracted for your Control whether a Load content from OS automatically service starts on ‘files/’, ‘templates/’ boot or is required to or custom strings Specify ‘installed’, be running always ‘absent’, or ‘latest’ Create symlinks for desired state A service can be notified to restart if a Change from configuration file has ‘installed’ to ‘latest’ and deploy for quick
  73. 73. A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Services: Files/Directories: Supports 23 different Supports 10 different Specify ownership & package providers ‘init’ frameworks permissions Abstracted for your Control whether a Load content from OS automatically service starts on ‘files/’, ‘templates/’ boot or is required to or custom strings Specify ‘installed’, be running always ‘absent’, or ‘latest’ Create symlinks for desired state A service can be notified to restart if a Supports 5 types to Change from configuration file has verify a file checksum ‘installed’ to ‘latest’ and deploy for quick
  74. 74. A Partial List of Puppet ‘types’ Files & Directories Crontabs Nagios Users & Groups /etc/hosts SELinux Services Mail Aliases SSH Keys Packages Mount Points Yum ReposPackages: Services: Files/Directories: Supports 23 different Supports 10 different Specify ownership & package providers ‘init’ frameworks permissions Abstracted for your Control whether a Load content from OS automatically service starts on ‘files/’, ‘templates/’ boot or is required to or custom strings Specify ‘installed’, be running always ‘absent’, or ‘latest’ Create symlinks for desired state A service can be notified to restart if a Supports 5 types to Change from configuration file has verify a file checksum ‘installed’ to ‘latest’ and deploy for quick Purge a directory of
  75. 75. General Puppet Syntax
  76. 76. General Puppet SyntaxClass Configuration: Single Class: class ntp { ... } Inherited Class: class sftp inherits ssh { ... } Nested Class: class foo { class bar { ... } } Scoped Class: class ntp::base { ... }
  77. 77. General Puppet SyntaxClass Configuration: Single Class: class ntp { ... } Inherited Class: class sftp inherits ssh { ... } Nested Class: class foo { class bar { ... } } Scoped Class: class ntp::base { ... }Selectors: $admin = $user_id ? { ‘0’ => ‘root’, }
  78. 78. General Puppet SyntaxClass Configuration: If-Else Conditionals: Single Class: if ($ec2_security_groups == ‘DNS’) { class ntp { ... } include bind::server Inherited Class: } else { include bind::client class sftp inherits ssh { ... } } Nested Class: class foo { class bar { ... } } Scoped Class: class ntp::base { ... }Selectors: $admin = $user_id ? { ‘0’ => ‘root’, }
  79. 79. General Puppet SyntaxClass Configuration: If-Else Conditionals: Single Class: if ($ec2_security_groups == ‘DNS’) { class ntp { ... } include bind::server Inherited Class: } else { include bind::client class sftp inherits ssh { ... } } Nested Class: Case Statements: class foo { case $ec2_security_groups { class bar { ... } Monitoring: { include nagios } Developer: { include mercurial } } } Scoped Class: class ntp::base { ... }Selectors: $admin = $user_id ? { ‘0’ => ‘root’, }
  80. 80. General Puppet SyntaxClass Configuration: If-Else Conditionals: Single Class: if ($ec2_security_groups == ‘DNS’) { class ntp { ... } include bind::server Inherited Class: } else { include bind::client class sftp inherits ssh { ... } } Nested Class: Case Statements: class foo { case $ec2_security_groups { class bar { ... } Monitoring: { include nagios } Developer: { include mercurial } } } Scoped Class: Set a Variable: class ntp::base { ... } $lib_path = “/usr/local/lib64/”Selectors: $admin = $user_id ? { ‘0’ => ‘root’, }
  81. 81. General Puppet SyntaxClass Configuration: If-Else Conditionals: Single Class: if ($ec2_security_groups == ‘DNS’) { class ntp { ... } include bind::server Inherited Class: } else { include bind::client class sftp inherits ssh { ... } } Nested Class: Case Statements: class foo { case $ec2_security_groups { class bar { ... } Monitoring: { include nagios } Developer: { include mercurial } } } Scoped Class: Set a Variable: class ntp::base { ... } $lib_path = “/usr/local/lib64/”Selectors: $admin = $user_id ? { Basic Math: ‘0’ => ‘root’, $file_size = $bytes * 1024 }
  82. 82. A Simple NTP Puppet Module
  83. 83. A Simple NTP Puppet Modulentpd/manifests/init.pp: class ntp { package { "ntp": ensure => latest } service { "ntpd": ensure => running, enable => true, hasrestart => true, hasstatus => true, require => Package["ntp"], } file { "/etc/ntp.conf": ensure => present, owner => root, group => root, mode => 0644, source => "puppet:///modules/ntp/ ntp.conf", notify => Service["ntpd"]; "/etc/sysconfig/ntpd": ensure => present, owner => root, group => root, mode => 0644, source => "puppet:///modules/ntp/ntpd", notify => Service["ntpd"]; } }
  84. 84. A Simple NTP Puppet Modulentpd/manifests/init.pp: class ntp { package { "ntp": ensure => latest } service { "ntpd": ensure => running, enable => true, hasrestart => true, hasstatus => true, require => Package["ntp"], } file { "/etc/ntp.conf": ensure => present, owner => root, group => root, mode => 0644, source => "puppet:///modules/ntp/ ntp.conf", notify => Service["ntpd"]; "/etc/sysconfig/ntpd": ensure => present, owner => root, group => root, mode => 0644, source => "puppet:///modules/ntp/ntpd", notify => Service["ntpd"]; } }
  85. 85. A Simple NTP Puppet Modulentpd/manifests/init.pp: ntpd/files/ntp.conf: class ntp { restrict default kod nomodify notrap nopeer noquery package { "ntp": ensure => latest } restrict 127.0.0.1 service { "ntpd": ensure => running, server nist.netservicesgroup.com enable => true, server time.nist.gov hasrestart => true, server time-a.nist.gov hasstatus => true, server time-b.nist.gov require => Package["ntp"], } server 127.127.1.0 fudge 127.127.1.0 stratum 10 file { "/etc/ntp.conf": driftfile /var/lib/ntp/drift ensure => present, owner => root, keys /etc/ntp/keys group => root, mode => 0644, source => "puppet:///modules/ntp/ ntp.conf", notify => Service["ntpd"]; "/etc/sysconfig/ntpd": ensure => present, owner => root, group => root, mode => 0644, source => "puppet:///modules/ntp/ntpd", notify => Service["ntpd"]; } }
  86. 86. A Simple NTP Puppet Modulentpd/manifests/init.pp: ntpd/files/ntp.conf: class ntp { restrict default kod nomodify notrap nopeer noquery package { "ntp": ensure => latest } restrict 127.0.0.1 service { "ntpd": ensure => running, server nist.netservicesgroup.com enable => true, server time.nist.gov hasrestart => true, server time-a.nist.gov hasstatus => true, server time-b.nist.gov require => Package["ntp"], } server 127.127.1.0 fudge 127.127.1.0 stratum 10 file { "/etc/ntp.conf": driftfile /var/lib/ntp/drift ensure => present, owner => root, keys /etc/ntp/keys group => root, mode => 0644, source => "puppet:///modules/ntp/ ntp.conf", ntpd/files/ntpd: notify => Service["ntpd"]; OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid" "/etc/sysconfig/ntpd": SYNC_HWCLOCK=yes ensure => present, NTPDATE_OPTIONS="-g -x" owner => root, group => root, mode => 0644, source => "puppet:///modules/ntp/ntpd", notify => Service["ntpd"]; } }
  87. 87. EC2 Security Group Magic
  88. 88. EC2 Security Group MagicEC2 security groups are anamed set of inboundfirewall rules for a giveninstance
  89. 89. EC2 Security Group MagicEC2 security groups are anamed set of inboundfirewall rules for a giveninstancePuppet can learn about EC2meta-data very easily
  90. 90. EC2 Security Group MagicEC2 security groups are anamed set of inboundfirewall rules for a giveninstancePuppet can learn about EC2meta-data very easilyTell Puppet to configureinstances based on theirsecurity group
  91. 91. EC2 Security Group MagicEC2 security groups are anamed set of inboundfirewall rules for a giveninstancePuppet can learn about EC2meta-data very easilyTell Puppet to configureinstances based on theirsecurity groupScales for 1 instance or 100
  92. 92. EC2 Security Group MagicEC2 security groups are anamed set of inboundfirewall rules for a giveninstancePuppet can learn about EC2meta-data very easilyTell Puppet to configureinstances based on theirsecurity groupScales for 1 instance or 100Rinse and repeat for eachservice group you have
  93. 93. EC2 Security Groups + Puppet
  94. 94. EC2 Security Groups + Puppet‘DNS’ EC2 Security Group:
  95. 95. EC2 Security Groups + Puppet‘DNS’ EC2 Security Group: Inbound Firewall Rules: 22/TCP for SSH for remote access 53/{TCP,UDP} for DNS nameserver
  96. 96. EC2 Security Groups + Puppet‘DNS’ EC2 Security Group: Inbound Firewall Rules: Puppet Modules Enabled: 22/TCP for SSH for remote access ssh - SSH server configuration 53/{TCP,UDP} for DNS nameserver bind - BIND nameserver
  97. 97. EC2 Security Groups + Puppet‘DNS’ EC2 Security Group: Inbound Firewall Rules: Puppet Modules Enabled: 22/TCP for SSH for remote access ssh - SSH server configuration 53/{TCP,UDP} for DNS nameserver bind - BIND nameserver
  98. 98. EC2 Security Groups + Puppet‘DNS’ EC2 Security Group: Inbound Firewall Rules: Puppet Modules Enabled: 22/TCP for SSH for remote access ssh - SSH server configuration 53/{TCP,UDP} for DNS nameserver bind - BIND nameserver The Puppet type ‘file’ allows for variable-replacement in filenames and use-on-first-match
  99. 99. EC2 Security Groups + Puppet‘DNS’ EC2 Security Group: Inbound Firewall Rules: Puppet Modules Enabled: 22/TCP for SSH for remote access ssh - SSH server configuration 53/{TCP,UDP} for DNS nameserver bind - BIND nameserver The Puppet type ‘file’ allows for variable-replacement in filenames and use-on-first-match file { "/etc/ssh/sshd_config": source => [ “puppet:///modules/ssh/{$ec2_security_groups}-sshd_config”, "puppet:///modules/ssh/sshd_config" ]; }
  100. 100. EC2 Security Groups + Puppet‘DNS’ EC2 Security Group: Inbound Firewall Rules: Puppet Modules Enabled: 22/TCP for SSH for remote access ssh - SSH server configuration 53/{TCP,UDP} for DNS nameserver bind - BIND nameserver The Puppet type ‘file’ allows for variable-replacement in filenames and use-on-first-match file { "/etc/ssh/sshd_config": source => [ “puppet:///modules/ssh/{$ec2_security_groups}-sshd_config”, "puppet:///modules/ssh/sshd_config" ]; } Puppet will use ‘DNS-sshd_config’ if it exists. If the file does not exist, it will use ‘sshd_config’
  101. 101. Client Meta-Data with Facter
  102. 102. Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!
  103. 103. Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!Examples:
  104. 104. Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!Examples: $architecture - Create files that are based on architecture
  105. 105. Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!Examples: $architecture - Create files that are based on architecture $hostname/$ip_address_eth0 - Create an /etc/hosts entry
  106. 106. Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!Examples: $architecture - Create files that are based on architecture $hostname/$ip_address_eth0 - Create an /etc/hosts entry $uptime_days - Update all packages after 30 days uptime
  107. 107. Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!Examples: $architecture - Create files that are based on architecture $hostname/$ip_address_eth0 - Create an /etc/hosts entry $uptime_days - Update all packages after 30 days uptime $selinux - Configure packages based on SELinux contexts
  108. 108. Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!Examples: $architecture - Create files that are based on architecture $hostname/$ip_address_eth0 - Create an /etc/hosts entry $uptime_days - Update all packages after 30 days uptime $selinux - Configure packages based on SELinux contexts $operatingsystemrelease - Run OS version specific tasks
  109. 109. Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!Examples: $architecture - Create files that are based on architecture $hostname/$ip_address_eth0 - Create an /etc/hosts entry $uptime_days - Update all packages after 30 days uptime $selinux - Configure packages based on SELinux contexts $operatingsystemrelease - Run OS version specific tasks $is_virtual - Configure hosts based on VM vs. Physical
  110. 110. Client Meta-Data with FacterRetrieve useful ‘facts’ about a client host to determine how tointeract with it. ‘facter’ quickly inventories all system metrics!Examples: $architecture - Create files that are based on architecture $hostname/$ip_address_eth0 - Create an /etc/hosts entry $uptime_days - Update all packages after 30 days uptime $selinux - Configure packages based on SELinux contexts $operatingsystemrelease - Run OS version specific tasks $is_virtual - Configure hosts based on VM vs. Physical $ec2_ami_id - Update configuration for the EC2 AMI used
  111. 111. Nagios ‘Type’
  112. 112. Nagios ‘Type’Puppet natively supports creating Nagios configuration
  113. 113. Nagios ‘Type’Puppet natively supports creating Nagios configurationEasily generate specific configuration for n hosts automatically
  114. 114. Nagios ‘Type’Puppet natively supports creating Nagios configurationEasily generate specific configuration for n hosts automaticallyNever again manually include hosts/services in groupings
  115. 115. Nagios ‘Type’Puppet natively supports creating Nagios configurationEasily generate specific configuration for n hosts automaticallyNever again manually include hosts/services in groupings
  116. 116. Nagios ‘Type’Puppet natively supports creating Nagios configurationEasily generate specific configuration for n hosts automaticallyNever again manually include hosts/services in groupingsNagios Service:@@nagios_service { "load_check_${hostname}": service_description => "Load Averages", check_command => "load_check!3!5", host_name => "$fqdn", use => "generic-service";}
  117. 117. Nagios ‘Type’Puppet natively supports creating Nagios configurationEasily generate specific configuration for n hosts automaticallyNever again manually include hosts/services in groupingsNagios Service:@@nagios_service { "load_check_${hostname}": service_description => "Load Averages", check_command => "load_check!3!5", host_name => "$fqdn", use => "generic-service";}Nagios Service Group:@@nagios_servicegroup { "apache_servers": alias => "Apache Servers";}
  118. 118. Nagios ‘Type’Puppet natively supports creating Nagios configurationEasily generate specific configuration for n hosts automaticallyNever again manually include hosts/services in groupingsNagios Service: Nagios Host:@@nagios_service { "load_check_${hostname}": @@nagios_host { $fqdn: service_description => "Load ensure => present, Averages", hostgroups => "ldap", check_command => "load_check!3!5", use => "generic-host"; host_name => "$fqdn", } use => "generic-service";}Nagios Service Group:@@nagios_servicegroup { "apache_servers": alias => "Apache Servers";}
  119. 119. Nagios ‘Type’Puppet natively supports creating Nagios configurationEasily generate specific configuration for n hosts automaticallyNever again manually include hosts/services in groupingsNagios Service: Nagios Host:@@nagios_service { "load_check_${hostname}": @@nagios_host { $fqdn: service_description => "Load ensure => present, Averages", hostgroups => "ldap", check_command => "load_check!3!5", use => "generic-host"; host_name => "$fqdn", } use => "generic-service";}Nagios Service Group: Nagios Host Group:@@nagios_servicegroup { @@nagios_hostgroup { "apache_servers": "load_balancers": alias => "Apache Servers"; alias => "Load Balancers";} }
  120. 120. Puppet Generated Host/Service Checks
  121. 121. Puppet Generated Munin Metrics/Groupings
  122. 122. The Foreman: A Heavy Lifter
  123. 123. The Foreman: A Heavy LifterEasy-to-use Puppet webinterface for many tasks
  124. 124. The Foreman: A Heavy LifterEasy-to-use Puppet webinterface for many tasksReview Puppet reportsregarding your hosts easily
  125. 125. The Foreman: A Heavy LifterEasy-to-use Puppet webinterface for many tasksReview Puppet reportsregarding your hosts easilyEdit host facts and groups
  126. 126. The Foreman: A Heavy LifterEasy-to-use Puppet webinterface for many tasksReview Puppet reportsregarding your hosts easilyEdit host facts and groupsLDAP authentication
  127. 127. The Foreman: A Heavy LifterEasy-to-use Puppet webinterface for many tasksReview Puppet reportsregarding your hosts easilyEdit host facts and groupsLDAP authenticationStatistical graphs for metrics
  128. 128. The Foreman: A Heavy LifterEasy-to-use Puppet webinterface for many tasksReview Puppet reportsregarding your hosts easilyEdit host facts and groupsLDAP authenticationStatistical graphs for metricsExecute puppetrun on hosts
  129. 129. The Foreman: A Heavy LifterEasy-to-use Puppet webinterface for many tasksReview Puppet reportsregarding your hosts easilyEdit host facts and groupsLDAP authenticationStatistical graphs for metricsExecute puppetrun on hostsProvision hosts from the web
  130. 130. The Foreman ‘Overview’ Page
  131. 131. Interact with ‘Facter Facts’
  132. 132. Evaluate Puppet Efficiency with Reports
  133. 133. General Statistics for Puppet Clients
  134. 134. Marionette Collective
  135. 135. Marionette CollectiveManage/Control/Execute: Services Packages Process Information Facter Facts Pings
  136. 136. Marionette CollectiveManage/Control/Execute: Services Packages Process Information Facter Facts PingsDecide which hosts you actupon by any Facter Fact
  137. 137. Marionette CollectiveManage/Control/Execute: Services Packages Process Information Facter Facts PingsDecide which hosts you actupon by any Facter FactEasily manage a largeamount of diverse hosts
  138. 138. View Any Service’s Status Across Hosts
  139. 139. Check Versions That Are Installed
  140. 140. View Processes On Hosts Matching a ‘Fact’
  141. 141. Quickly Retrieve a List of MCollective Hosts
  142. 142. Consider This Scenario
  143. 143. Consider This Scenario1. You reserve 10 Elastic IPs for a network of hosts
  144. 144. Consider This Scenario1. You reserve 10 Elastic IPs for a network of hosts2. Each instance starts and Puppet gives it an elastic IP
  145. 145. Consider This Scenario1. You reserve 10 Elastic IPs for a network of hosts2. Each instance starts and Puppet gives it an elastic IP3. Based on an ‘IP -> NEED’ map, each new instance is created for a specific need (DNS, WWW, IMAP, etc.)
  146. 146. Consider This Scenario1. You reserve 10 Elastic IPs for a network of hosts2. Each instance starts and Puppet gives it an elastic IP3. Based on an ‘IP -> NEED’ map, each new instance is created for a specific need (DNS, WWW, IMAP, etc.)4. Hosts that become ‘WWW’ servers automatically are added to the Elastic Load Balancer (ELB) instance
  147. 147. Consider This Scenario1. You reserve 10 Elastic IPs for a network of hosts2. Each instance starts and Puppet gives it an elastic IP3. Based on an ‘IP -> NEED’ map, each new instance is created for a specific need (DNS, WWW, IMAP, etc.)4. Hosts that become ‘WWW’ servers automatically are added to the Elastic Load Balancer (ELB) instance5. Nagios & Munin configuration is done automatically
  148. 148. Consider This Scenario1. You reserve 10 Elastic IPs for a network of hosts2. Each instance starts and Puppet gives it an elastic IP3. Based on an ‘IP -> NEED’ map, each new instance is created for a specific need (DNS, WWW, IMAP, etc.)4. Hosts that become ‘WWW’ servers automatically are added to the Elastic Load Balancer (ELB) instance5. Nagios & Munin configuration is done automatically6. If an instance dies, the next time a new instance starts it is given the old host’s IP and that service is fulfilled again
  149. 149. Consider This Scenario1. You reserve 10 Elastic IPs for a network of hosts2. Each instance starts and Puppet gives it an elastic IP3. Based on an ‘IP -> NEED’ map, each new instance is created for a specific need (DNS, WWW, IMAP, etc.)4. Hosts that become ‘WWW’ servers automatically are added to the Elastic Load Balancer (ELB) instance5. Nagios & Munin configuration is done automatically6. If an instance dies, the next time a new instance starts it is given the old host’s IP and that service is fulfilled again ...most importantly, you’ve done nothing :)
  150. 150. Take Your Environment
  151. 151. Take Your EnvironmentPuppet: Provides you with the means to handle ad-hocEC2 instance scaling with granular updates/configurationchanges based on any ‘Fact’ you can supply.
  152. 152. Take Your EnvironmentPuppet: Provides you with the means to handle ad-hocEC2 instance scaling with granular updates/configurationchanges based on any ‘Fact’ you can supply.The Foreman: Manage your hosts from a well designedfront-end. View reports, check for deployment efficiency,get the ‘big picture’ on your infrastructure; even deployhosts from scratch!
  153. 153. Take Your EnvironmentPuppet: Provides you with the means to handle ad-hocEC2 instance scaling with granular updates/configurationchanges based on any ‘Fact’ you can supply.The Foreman: Manage your hosts from a well designedfront-end. View reports, check for deployment efficiency,get the ‘big picture’ on your infrastructure; even deployhosts from scratch!MCollective: Handle your mass administrative tasks withconsistency and structure. Utilize ‘Facter’ to intelligentlyexecute tasks only against certain sub-sets of hosts.
  154. 154. Take Your EnvironmentPuppet: Provides you with the means to handle ad-hocEC2 instance scaling with granular updates/configurationchanges based on any ‘Fact’ you can supply.The Foreman: Manage your hosts from a well designedfront-end. View reports, check for deployment efficiency,get the ‘big picture’ on your infrastructure; even deployhosts from scratch!MCollective: Handle your mass administrative tasks withconsistency and structure. Utilize ‘Facter’ to intelligentlyexecute tasks only against certain sub-sets of hosts.Nagios/Munin: Automatically deploy full monitoring &metrics for hosts without ever hand configuring a file.
  155. 155. Thanks! Questions?mark.stanislav@gmail.comuncompiled.com@markstanislav

×