Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Puppet modules for Fun and Profit

Presentation held at PuppetConf 2012 about the Next Generation Puppet modules

  • Be the first to comment

Puppet modules for Fun and Profit

  1. 1. PUPPET MODULESFOR FUN AND PROFIT Puppet Conf San Francisco 2012 Alessandro Franceschi Lab42 / GrandSla
  2. 2. PUPPET @ Lab 422007 - Meet Puppet. Managed the Bank of Italy webfarm2008 - First generation of Lab42 Puppet Modules2009 - Multi OS support and standardization2010 - A redesigned and coherent Example42 Module set Puppet Modules Standards and Interoperability (PuppetCamp 2010 - Belgium) Re-Use your Modules! (PuppetCamp 2010 - San Francisco)2011 - Introducing Puppi Puppi: Puppet strings to the shell (PuppetCamp Europe 2011 - Amsterdam)2012 - Example42 Next Gen modules - GrandSla: Puppet driven Infrastructure and Support Developing IT Infrastructures with Puppet (CodeMotion 2012 - Rome) A Holistic approach to Puppet modules (PuppetCamp Dublin and Geneva 2012) “Job Driven” modules development
  3. 3. MODULES PATTERNS...Data Separation Configuration data is defined outside the module (or even Puppet manifests) Module’s behavior is managed via APIs Allow module’s extension and override via external dataReusability Customize behavior without changing module code Do not force author’s idea on how configurations should be provided Support different OS. Easily allow new additionsStandardization Follow PuppetLabs style guidelines (puppet-lint) Have coherent, predictable and intuitive interfaces Provide contextual documentation (puppet-doc)Interoperability Limit cross-module dependencies Allow easy modules’ cherry picking Be self contained, do not interfere with other modules’ resources
  4. 4. ... AND ANTI-PATTERNSData Mixed with Logic Configuration data mixed inside the module’s logic Module’s behavior defined in many different places Module’s logic is rigid and can’t be defined externallyWorks for me (here and now) Module just works for the current setup Can’t easily be re-used in other places for other projects Works just for the currently used OSesCode chaos No layout rules, no standard style Not standardized and predictable parameters Who said “documentation”?Interoperability Who cares.Basically whatever is quick and dirty... but is this really an anti-pattern?* *(IMHO, yes)
  5. 5. Example42 modules: 10 design rulesRule 1 - Provide alternatives for Data SeparationRule 2 - Provide choice on Configuration Files supplyRule 3 - Configure everything but provide OS defaults.Rule 4 - Allow management of general module’s behaviorRule 5 - Allow Custom Options for endless parametersRule 6 - Permit easy extension with custom classesRule 7 - Offer easy removal of the module’s resourcesRule 8 - Limit cross-dependencies. Prerequisites as options.Rule 9 - Automatically monitor and firewall resourcesRule 10 - Puppi integration: Puppet knowledge to the CLI
  6. 6. Example42 DATA SEPARATION ALTERNATIVES Rule #1Set (Top Scope/External Node Classifier) variables and include classes: $::openssh_template = site/openssh/openssh.conf.erb include opensshUse Hiera: hiera(openssh_template) include opensshUse Parametrized Classes: class { openssh:   template => site/openssh/openssh.conf.erb, }Happily mix different patterns: $::monitor = true $::monitor_tool = [ nagios , munin , puppi ] class { openssh:   template => site/openssh/openssh.conf.erb, }
  7. 7. Example42 PARAMS_LOOKUP EVERYWHERE Rule #1 Each parameter is processed by the params_lookup function class openssh ( [...] # openssh module specific parameters ...   $my_class = params_lookup( my_class ),   $source = params_lookup( source ),   $source_dir = params_lookup( source_dir ),   $source_dir_purge = params_lookup( source_dir_purge ),   $template = params_lookup( template ),   $service_autorestart = params_lookup( service_autorestart , global ),   $options = params_lookup( options ),   $version = params_lookup( version ),   $absent = params_lookup( absent ),   $disable = params_lookup( disable ),   $disableboot = params_lookup( disableboot ),   $monitor = params_lookup( monitor , global ),   $monitor_tool = params_lookup( monitor_tool , global ),   $monitor_target = params_lookup( monitor_target , global ), [...] # Other common parameters   ) inherits openssh::params { [...] } Flexibility on booleans: they are sanitized by the any2bool function   You set: $absent => “yes” # (or “1”, ‘Y’, “true”, true ...) The module internally uses:   $bool_absent = any2bool($absent)
  8. 8. Example42 PARAMS LOOKUP ORDER Rule #1The function params_lookup is provided by the Puppi moduleIt allows data to be defined in different ways: Via Hiera, if available As Top Scope variable (as provided by External Node Classifiers) Via defaults set in the module’s params classThe “global” argument is used to define site_wide behavior # If there’s a direct param that’s the value class { ‘openssh’: monitor => true } # Otherwise, If Hiera is available: hiera(“monitor”) # If global lookup is set hiera(“openssh_monitor”) # A specific value overrides the global one # If variable is still not evaluated, Top Scope is looked up: $::monitor # If global lookup is set $::openssh_monitor # If present, overrides $::monitor # Module’s params are used as last option defaults: $openssh::params::monitor
  9. 9. Example42 CUSTOMIZE: CONFIGURATION FILE Rule #2Provide Main Configuration as a static file ... class { openssh:   source => puppet:///modules/site/ssh/sshd.conf }... an array of files looked up on a first match logic ... class { openssh:   source => [ "puppet:///modules/site/ssh/sshd.conf-${fqdn}",               "puppet:///modules/site/ssh/openssh.conf"], }... or an erb template: class { openssh:   template => site/ssh/sshd.conf.erb, } Config File Path is defined in params.pp (can be overriden): config_file => /etc/ssh/sshd_config,
  10. 10. Example42 CUSTOMIZE: CONFIGURATION DIR Rule #2You can manage the whole Configuration Directory: class { openssh:   source_dir => puppet:///modules/site/ssh/sshd/, } This copies all the files in lab42/files/ssh/sshd/* to local config_dirYou can purge any existing file on the destination config_dir which arenot present on the source_dir path: class { openssh:   source_dir => puppet:///modules/site/ssh/sshd/,   source_dir_purge => true, # default is false } WARNING: Use with careConfig Dir Path is defined in params.pp (can be overriden):   config_dir => /etc/ssh,
  11. 11. Example42 CUSTOMIZE: PATHS AND NAMES Rule #3Customize Application Parameters. An example:Use the puppet module to manage pe-puppet! class { puppet:   template => lab42/pe-puppet/puppet.conf.erb,   package => pe-puppet,   service => pe-puppet,   service_status => true,   config_file => /etc/puppetlabs/puppet/puppet.conf,   config_file_owner => root,   config_file_group => root,   config_file_init => /etc/sysconfig/pe-puppet,   process => ruby,   process_args => puppet,   process_user => root,   config_dir => /etc/puppetlabs/puppet/,   pid_file => /var/run/pe-puppet/,   log_file => /var/log/pe-puppet/puppet.log,   log_dir => /var/log/pe-puppet, }
  12. 12. Example42 DEFAULTS IN PARAMS.PP Rule #3 Each module has a params class with defaults for different OS class openssh::params { ### Application related parameters   $package = $::operatingsystem ? {     default => openssh-server,   }   $service = $::operatingsystem ? {     /(?i:Debian|Ubuntu|Mint)/ => ssh,     default => sshd,   }   $process = $::operatingsystem ? {     default => sshd,   } [...] $port = 22   $protocol = tcp # General Settings   $my_class =   $source =   $source_dir =   $source_dir_purge = [...] ### General module variables that can have a site or per module default   $monitor = false   $monitor_tool =   $monitor_target = $::ipaddress   $firewall = false   $firewall_tool =   $firewall_src = [...]
  13. 13. Example42 MANAGE BEHAVIOR Rule #4Enable Auditing: class { openssh:   audit_only => true, # Default: false } No changes to configuration files are actually made and potential changes are auditedManage Service Autorestart: class { openssh:   service_autorestart => false, # Default: true } No automatic service restart when a configuration file / dir changesManage Software Version: class { foo:   version => 1.2.0, # Default: unset } Specify the package version you want to be installed. Set => ‘latest’ to force installation of latest version
  14. 14. Example42 CUSTOM OPTIONS Rule #5 With templates you can provide an hash of custom options: class { openssh:   template => site/ssh/sshd.conf.erb,   options => {     LogLevel => INFO,     UsePAM => yes,   }, } The Hash values can be used in your custom templates: - Allow management of any kind of configuration parameter - Provide endless configuration values without adding new parameters - Works only for parameters used in templates on in custom classes
  15. 15. Example42 CUSTOM OPTIONS IN TEMPLATES Rule #5 Alternative ways to use the options hash in an erb template: Direct but not safe (you must always provide all the used options) UsePAM <%= options[UsePAM] %> Failsafe with defaults (verbose but safe) <% if scope.lookupvar("openssh::options[UsePAM]") then -%> UsePAM <%= options[UsePAM] %> <% else -%>UsePAM no<% end -%> Show what you have (useful for config files has defaults for every option) <% scope.lookupvar("openssh::options").sort_by {|key, value| key}.each do |key, value| -%> <%= key %> <%= value %> <% end -%> The smart way: options_lookup (Use the option value or set a default) UsePAM <%= scope.function_options_lookup([UsePAM,‘no’]) %>
  16. 16. Example42 CUSTOMIZE: CUSTOM CLASS Rule #6Provide added resources in a Custom Class: class { openssh:   my_class => site/my_openssh, } This autoloads: site/manifests/my_openssh.ppCustom class can stay in your site module: class site::my_openssh {   file { motd:     path => /etc/motd,     content => template(site/openssh/motd.erb),   } } You hardly need to inherit openssh: there are parameters for everything Do not call your class site::openssh, naming collisions could happen.
  17. 17. Example42 EASY DECOMMISSIONING Rule #7Disable openssh service: class { openssh:   disable => true }Deactivate openssh service only at boot time: class { openssh:   disableboot => true } Useful when a service is managed by another tool (ie: a cluster suite)Remove openssh (package and files): class { openssh:   absent => true }Monitoring and firewalling resources removal is automatically managed
  18. 18. Example42 CROSS-MODULE INTEGRATIONS Rule #8Integration with other modules sets and conflicts management is not easy. Strategy 1: Provide the option to use the module’s prerequisite resources: class { logstash:   install_prerequisites => false, # Default true } The prerequisites resources for this module are installed automatically BUT can be managed by third-party modules Strategy 2: Use if ! defined when defining common resources if ! defined(Package[git]) {   package { git: ensure => installed } } Not a definitive solution, but better than nothing. Strategy 3: Always define in Modulefile the module’s dependencies dependency example42/puppi, >= 2.0.0 Strategy 4: Never assume your resource defaults are set for others Exec { path => "/bin:/sbin:/usr/bin:/usr/sbin" }
  19. 19. Example42 EXTEND: MONITOR Rule #9Manage Abstract Automatic Monitoring: class { openssh:   monitor => true,   monitor_tool => [ nagios,puppi,monit ],   monitor_target => $::ip_address # Default }Monitoring is based on these parameters defined in params.pp:   port => 22,   protocol => tcp,   service => ssh[d], # According to OS   process => sshd,   process_args => ,   process_user => root,   pid_file => /var/run/,Abstraction is managed in the Example42 monitor module Here “connectors” for different monitoring tools are defined and can be added (also using 3rd party modules).
  20. 20. Example42 EXTEND: FIREWALL Rule #9Manage Automatic Firewalling (host based): class { openssh:   firewall => true,   firewall_tool => iptables,   firewall_src =>,   firewall_dst => $::ipaddress_eth1, # Default is $::ipaddress }Firewalling is based on these parameters defined in params.pp:   port => 22,   protocol => tcp,Abstraction is managed in the Example42 firewall module Currently only the “iptables” firewall_tool is defined, it uses Example42 iptables module to manage local iptables rules
  21. 21. Example42 EXTEND: PUPPI Rule #10Manage Puppi Integration: class { openssh:   puppi => true, # Default: false   puppi_helper => standard, # Default }The Puppi module is a prerequisite for all Example42 modules Is required because it provides common libs, widely used in the modules BUT the actual puppi integration is optional (and disabled by default)Puppi integration allows CLI enrichment commands like: puppi info openssh puppi log openssh puppi check openssh Note: puppi support for info/log commands for NextGen modules is under developmentPuppi helpers allow you to customize Puppi behavior
  22. 22. DOWNLOAD Example42 Puppet Modules Site: GitHub repositories: Git Download: git clone -r example42/puppet-modules-nextgen Note on GitHub repos: puppet-modules-nextgen contains only NextGen modules (as git submodules) puppet-modules contains both NextGen and older modules
  23. 23. One more thing...How to make a NextGen module git clone -r cd puppet-modules-nextgen Example42-tools/ This script creates a skeleton for a new module based on different Example42 foo module templates. Run it from the directory that contains the foo module (moduledir). By default it uses the "foo" module as template. Specify -t <source_module> to use a different template. Example: Example42-tools/ -t foo_webapp Source module template is foo Enter the name of the new module based on foo: mynewmodule E di t my n ewm o dul e / m an i f e st s/ param s.pp t o m an age di f f e re n t OSA new, basic, NextGen module based on the foo template is done.Add features and application specific resources to enrich it
  24. 24. Graphics: ad maiora Questions?@alvagante