Anatomy of a reusable module
Upcoming SlideShare
Loading in...5
×
 

Anatomy of a reusable module

on

  • 876 views

A presentation at PuppetConf 2013 about modules reusability, naming standards and stacks.

A presentation at PuppetConf 2013 about modules reusability, naming standards and stacks.

Statistics

Views

Total Views
876
Views on SlideShare
788
Embed Views
88

Actions

Likes
2
Downloads
6
Comments
0

7 Embeds 88

http://eventifier.com 69
https://twitter.com 7
http://eventifier.co 5
https://www.linkedin.com 3
http://ec2-54-243-189-159.compute-1.amazonaws.com 2
http://www.linkedin.com 1
http://www.example42.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Anatomy of a reusable module Anatomy of a reusable module Presentation Transcript

    • Anatomy of a Reusable Module Alessandro Franceschi github.com/example42 PuppetConf 2013
    • Be
    •   patient please
    • How
    •   do we
    •   use Puppet today? Include classes Set Params & Variables Define Business & Integration Logic Provide Configuration Files Manage Resources
    • How do we use Puppet today Include classes manifests/ site.pp Set Parameters / Variables Integration logic Resources ENC HIERA SITE MODULES SHARED MODULES BAD EDGE Configuration files manifests/ site.pp ENC HIERA manifests/ site.pp BAD? BAD? BAD SITE MODULES SITE MODULES SHARED MODULES SITE MODULES SHARED MODULES manifests/ site.pp BAD? ENC BAD?
    • A
    •   reusable
    •    module is
    •   all
    •   about CHOICE Operating Systems Infrastructures Scales Node classifiers Installation methods Alternative setups
    • The
    •   cons of
    •   a reusable module Harder & Longer Development Complexity Verbosity Not Optimized for performance
    • THE PARAMETERS DILEMMA Managed resources attributes Application specific config options Application logic and behaviour Integration with other modules
    • Parameters: Resources attributes Enough: $package = $redis::params::package, $service = $redis::params::service, $service_ensure = 'running', $service_enable = true, $file = $redis::params::file, $file_notify = "Service['redis']", $file_source = undef, $file_content = undef, Too much? $package_provider = undef, $file_owner = $redis::params::file_owner, $file_group = $redis::params::file_group, $file_mode = $redis::params::file_mode, $file_replace = $redis::params::file_replace, Benefits from: A standard naming convention
    • Parameters: Application options Enough: $puppet_server = “puppet.${::domain}”, $syslog_server = “syslog.${::domain}”, $munin_server = “munin.${::domain}”, $dns_servers = [ '8.8.8.8' , '8.8.4.4' ], Too much! $anonymous_enable = true, $anon_mkdir_write_enable = true, $anon_upload_enable = false, $chroot_list_enable = true, $chroot_list_file = '/etc/vsftpd/chroot_list', $resourcefile = $nagios::params::resourcefile, $statusfile = $nagios::params::statusfile, $commandfile = $nagios::params::commandfile, $resultpath = $nagios::params::resultpath, $retentionfile = $nagios::params::retentionfile, $p1file = $nagios::params::p1file, Benefits from: Template + Options Hash pattern
    • Parameters: Application logic Examples: $install_client = true, $install_stomp_server = true, $install_plugins = true, $use_ssl = false, $munin_autoconfigure = true, $service_autorestart = true, $manage_package_repo = true, $run_initdb = undef, Benefits from: A standard naming convention
    • Parameters: Modules Integrations Examples: $mongo_db_host = $graylog2::params::mongo_db_host, $mongo_db_port = $graylog2::params::mongo_db_port, $mongo_db_name = $graylog2::params::mongo_db_name, $mongo_user = $graylog2::params::mongo_user, $mongo_password = $graylog2::params::mongo_password, $elasticsearch_template = $graylog2::params::elasticsearch_template, $elasticsearch_path = $graylog2::params::elasticsearch_path, $database = $puppetdb::params::database, $manage_redhat_firewall = $puppetdb::params::manage_redhat_firewall, $db_type = 'mysql', Benefits from: Shared Stacks
    • PATTERNS REUSABILITY
    • Managing
    •    files Let user decide how to manage configuration files. Alternatives: source content concat augeas custom types
    • Managing files: source & content redis/manifests/init.pp class redis ( $file = $redis::params::file, $file_source = undef, $file_template = undef, $file_content = undef, ) { [...] $managed_file_content = $file_content ? { undef => $file_template ? { undef => undef, default => template($file_template), }, default => $file_content, } [...] if $redis::file { file { 'redis.conf': path => $redis::file, source => $redis::file_source, content => $redis::managed_file_content, } } } Provide the Puppet path of an erb template class { ‘redis’: file_template => ‘site/redis/ redis.conf.erb’, } Provide directly the content attribute class { ‘redis’: file_content => “template(‘site/redis/ redis.conf.erb’)”, } Provide a fileserver source path class { ‘redis’: file_source => ‘puppet:///modules/site/ redis/redis.conf’, } Manage the configuration file with other methods (augeas, concat...) class { ‘redis’: }
    • Multiple
    •    config
    •   files
    •    Add parameters to main class Use a generic conf define Manage the whole configuration dir
    • Multiple files: Add parameters elasticsearch/manifests/init.pp class elasticsearch ( $file = $elasticsearch::params::file, $file_source = undef, $file_template = undef, $file_content = undef, [...] $init_script_file = '/etc/init.d/elasticsearch', $init_script_file_template = 'elasticsearch/init.erb', $init_options_file = $elasticsearch::params::init_options_file, $init_options_file_template = 'elasticsearch/init_options.erb', Provide custom templates for the main file and the init script class { ‘elasticsearch’: file_template => ‘site/elasticsearch/elasticsearch.yml.erb’, init_script_file_template => ‘site/elasticsearch/elasticsearch.init.erb’, }
    • Multiple files: Generic conf define nova/manifests/conf.pp define nova::conf ( $source = undef, $template = undef, $content = undef, $path = undef, [...] $options_hash = undef, $ensure = present ) { include nova $managed_path = $path ? { undef => "${nova::config_dir}/${name}", default => $path, } [...] file { "nova_conf_${name}": ensure => $ensure, source => $source, content => $managed_content, path => $managed_path, mode => $managed_mode, owner => $managed_owner, group => $managed_group, require => $managed_require, notify => $managed_notify, replace => $managed_replace, } } Provide a custom template for an alternative config file in config_dir nova::conf { ‘rootwrap.conf’: template => ‘site/nova/rootwrap.conf.erb’, }
    • Multiple files: Whole config dir redis/manifests/init.pp class redis ( $dir = $redis::params::dir, $dir_source = undef, $dir_purge = false, $dir_recurse = true, ) { [...] $dir_ensure = $ensure ? { 'absent' => 'absent', 'present' => 'directory', } if $redis::dir_source { file { 'redis.dir': ensure => $redis::dir_ensure, path => $redis::dir, source => $redis::dir_source, recurse => $redis::dir_recurse, purge => $redis::dir_purge, force => $redis::dir_purge, notify => $redis::file_notify, require => $redis::file_require, } } } Provide a custom source for the whole config_dir class { ‘redis’: dir_source => ‘puppet:///modules/site/redis/conf/’, } Provide a custom source for the whole config_dir and purge any not managed config file class { ‘redis’: dir_source => ‘puppet:///modules/site/redis/conf/’, dir_purge => true, }
    • Managing
    •    Users Everyone has his own users... Leave options to decide if, how and where to manage the ones the module requires.
    • Managing Users elasticsearch/manifests/init.pp class elasticsearch { $ensure = 'present', [...] $user = 'elasticsearch', $user_uid = undef, $user_gid = undef, $user_groups = undef, $user_class = 'elasticsearch::user', [...] if $elasticsearch::user_class { require $elasticsearch::user_class } elasticsearch/manifests/user.pp class elasticsearch::user { @user { $elasticsearch::user : ensure => $elasticsearch::ensure, comment => "${elasticsearch::user} user", password => '!', managehome => false, uid => $elasticsearch::user_uid, gid => $elasticsearch::user_gid, groups => $elasticsearch::user_groups, shell => '/bin/bash', } User <| title == $elasticsearch::user |> } Do not create the requested user class { ‘elasticsearch’: user_class => undef, } Provide the user in a different custom class class { ‘elasticsearch’: user_class => 'site::users', } Run elasticsearch with a different user class { ‘elasticsearch’: user => 'apache', }
    • Managing extra
    •    resources Options to specify custom classes Options to pass an hash to create_resources
    • Extra Resources: Custom classes elasticsearch/manifests/init.pp class elasticsearch ( $dependency_class = 'elasticsearch::dependency', $monitor_class = 'elasticsearch::monitor', $firewall_class = 'elasticsearch::firewall', $my_class = undef, ) { [...] if $elasticsearch::dependency_class { include $elasticsearch::dependency_class } if $elasticsearch::monitor and $elasticsearch::monitor_class { include $elasticsearch::monitor_class } if $elasticsearch::firewall and $elasticsearch::firewall_class { include $elasticsearch::firewall_class } if $elasticsearch::my_class { include $elasticsearch::my_class }[...] Provide the modules dependencies with a custom class class { ‘elasticsearch’: dependency_class => 'site::dep_elasticsearch', }
    • Extra Resources: Resources Hash elasticsearch/manifests/init.pp class elasticsearch ( $create_resource = undef, $resources_hash = undef, ) { [...] if $create_resource { create_resources( $create_resource , $resources_hash ) } Alternative: A single hash that includes resources and resources_hash Provide the modules dependencies with a custom class class { ‘elasticsearch’: create_resource => 'file', resources_hash => { path => '/etc/elasticsearch/my_file', content => template('site/elasticsearch/my_file.erb), mode => '0600', }, }
    • Managing Packages and Services Names change Custom packages are common Leave choice, optionally
    • Managing packages openssh/manifests/init.pp class openssh ( $ensure = 'present', $version = undef, $package = $openssh::params::package, [...] ) { if $version and $ensure == 'present' { $managed_package_ensure = $version } else { $managed_package_ensure = $ensure } if $openssh::package { package { $openssh::package: ensure => $openssh::managed_package_ensure, } } openssh/manifests/params.pp class openssh::params { $package = $::osfamily ? { Suse => 'openssh', OpenBSD => '', default => 'openssh-server', } Install a custom company-openssh package class { ‘openssh’: package => 'company-openssh', }
    • Managing services openssh/manifests/init.pp class openssh ( $service = $openssh::params::service, $service_ensure = 'running', $service_enable = true, [...] ) { if $ensure == 'absent' { $managed_service_enable = undef $managed_service_ensure = stopped } else { $managed_service_enable = $service_enable $managed_service_ensure = $service_ensure } if $openssh::service { service { $openssh::service: ensure => $openssh::managed_service_ensure, enable => $openssh::managed_service_enable, } } openssh/manifests/params.pp class openssh::params { $service = $::osfamily ? { Debian => 'ssh', default => 'sshd', } [...] Manage a custom company-openssh service class { ‘openssh’: service => 'company-openssh', }
    • Managing Installation
    •    options Let users decide: OS Packages Upstream tarballs Provider
    • Installation options elasticsearch/manifests/init.pp class elasticsearch ( $package_provider = undef, $install = 'package', $install_base_url = $elasticsearch::params::install_base_url, $install_source = undef, $install_destination = '/opt', ) { [...] $managed_file = $elasticsearch::install ? { package => $elasticsearch::file, default => "${elasticsearch::home_dir}/config/elasticsearch.yml", } [...] case $elasticsearch::install { package: { package { $elasticsearch::package: ensure => $elasticsearch::managed_package_ensure, provider => $elasticsearch::package_provider, } } upstream: { puppi::netinstall { 'netinstall_elasticsearch': url => $elasticsearch::managed_install_source, destination_dir => $elasticsearch::install_destination, owner => $elasticsearch::user, group => $elasticsearch::user, } [...] Install elasticsearch from upstream source class { ‘elasticsearch’: install => 'upstream', install_source => 'https://download.elasticsearch.org/ elasticsearch/elasticsearch/elasticsearch-0.90.3.zip', }
    • Templates and
    •   hashes Managing specific application configs parameters may get out of control A single config hash to show them all A custom template to use them Application specific configs THE PARAMETERS DILEMMA
    • Options Hash: Setup openssh/manifests/init.pp class openssh ( [...] $file_template = undef, $options_hash = undef, site/templates/openssh/sshd_config.erb # File Managed by Puppet [...] Port <%= scope.function_options_lookup(['Port','22']) %> PermitRootLogin <%= scope.function_options_lookup(['PermitRootLogin','yes']) %> UsePAM <%= scope.function_options_lookup(['UsePAM','yes']) %> [...] * Function options_lookup currently in Example42's Puppi module Alternative site/templates/openssh/sshd_config.erb Port <%= scope.lookupvar('openssh::options_hash')['Port'] ||='22' %> PermitRootLogin <%= scope.lookupvar('openssh::options_hash')['PermitRootLogin'] ||='yes' %> UsePAM <%= scope.lookupvar('openssh::options_hash')['UsePAM'] ||='yes' %> [...]
    • Options Hash: Usage Usage (with Hiera): include openssh /etc/puppet/hieradata/global.yml: --- openssh::file_template: 'site/openssh/sshd_config.erb' openssh::file_options_hash: Port: '22222' PermitRootLogin: 'no' Usage (with parametrized class): class { 'openssh': file_template => 'site/openssh/sshd_config.erb' file_options_hash => { Port => '22222', PermitRootLogin => 'no', }
    • STANDARDS NAMING Managed resources attributes THE PARAMETERS DILEMMA
    • The Handy
    •   Grail of
    •   Modules
    •    Standards A blog post* Some discussions on Puppet-Users github.com/stdmod Naming standards for modules parameters Community driven (draft 0.0.2) * http://www.example42.com/?q=The_handy_Grail_of_Modules_Standards
    • Benefits
    •   of
    •    suggested (and
    •   shared) naming
    •    conventions Saner User Experience Better modules Interoperability Reusability Patterns Predictability in usage and development
    • Stdmod Params: Main resources ### General parameters ensure (enable?) version (package_version?) ### Package - Service - Main configuration file package (package_name?) package_ensure package_provider package_* [any relevant package type attribute] service (service_name?) service_ensure service_enable service_subscribe service_* file (file_path? config_file? config?) file_source (source? config_file_source? config_source?) file_template (template? config_file_template? config_template?) file_content (content? config_file_content? config_content?) file_* (config_file_*? config_*?) file_options_hash (options? options_hash? file_options?)
    • Stdmod Params: Extra resources other_package other_package_* client_package client_package_* server_package server_package_* other_service other_service_* log_file log_file_* pid_file pid_file_* init_script_file init_script_file_* init_config_file init_config_file_*
    • Stdmod Params: Installation ### Parameter related parameters install install_url install_base_url install_source install_destination install_pre_exec install_pre_exec_* install_post_exec install_post_exec_* install_script_file install_script_file_* install_response_file install_response_file_*
    • Stdmod Params: Monitor monitor monitor_tool monitor_host monitor_port monitor_protocol monitor_url monitor_process monitor_service monitor_config_hash
    • Stdmod Params: Firewall firewall firewall_src firewall_dst firewall_port firewall_protocol
    • STACKS MODULES Integrations of modules THE PARAMETERS DILEMMA
    • Why
    •    reinventing
    •    the
    •   stack? We always use stacks. We need them to make something useful with modules. What about: Sharing? Best practices? Standardization?
    • Stacks - A Simple Sample class stack::logs ( $ensure = 'present', $syslog_server = false, $syslog_server_port = '5544', $elasticsearch_server = false, $elasticsearch_server_port = '9200', $elasticsearch_cluster = 'logs', $elasticsearch_java_opts = '-Xmx2g -Xms1g', $install_logstash = false, $install_elasticsearch = false, $install_kibana = false, $install_graylog2 = false, $install_graylog2_webinterface = false, $syslog_config_template = 'stack/logs/syslog.conf.erb', $logstash_config_template = 'stack/logs/logstash.conf.erb', $elasticsearch_config_template = 'stack/logs/elasticsearch.yml.erb', $kibana_config_template = 'stack/logs/config.js.erb', $graylog2_config_template = 'stack/logs/graylog2.conf.erb', ) { [... TO BE CONTINUED ...]
    • Stacks - A Simple Sample [...] if $syslog_server { rsyslog::config { 'logstash_stack': content => template($syslog_config_template), } } if $install_logstash { class { 'logstash': template => $logstash_config_template, } } if $install_elasticsearch { class { 'elasticsearch': java_opts => $elasticsearch_java_opts, template => $elasticsearch_config_template, } } [...]
    • Stacks - Usage On any host: stack::logs { 'central': syslog_server => 'syslog.example42.com', } On the Logstash (syslog) server: stack::logs { 'central': syslog_server => 'syslog.example42.com', install_logstash => true, elasticsearch_server => 'el.example42.com', } On the Elasticsearch server(s), with a custom configuration file: stack::logs { 'central': syslog_server => 'syslog.example42.com', install_elasticsearch => true, elasticsearch_server => 'el.example42.com', elasticsearch_config_template => 'site/logs/elasticsearch.yml.erb', } On the Kibana server: stack::logs { 'central': syslog_server => 'syslog.example42.com', install_kibana => true, elasticsearch_server => 'el.example42.com', }
    • The Stacks Logic Stacks are local Modules are shared Higher level interface Integrate different set of modules Preserve modules local change
    • How do we use Puppet today Include classes manifests/ site.pp Set Parameters / Variables Integration logic Resources ENC HIERA SITE MODULES SHARED MODULES BAD EDGE Configuration files manifests/ site.pp ENC HIERA manifests/ site.pp BAD? BAD? BAD SITE MODULES SITE MODULES SHARED MODULES SITE MODULES SHARED MODULES manifests/ site.pp BAD? ENC BAD? STACKS STACKS STACKS
    • Standards for Stacks? Usual benefits: User Experience Interoperability Higher level API exposure Possible GUI Integrations
    • Steps Define stdmod naming conventions Explore Stacks design and approach Create templates for stacks and modules Explore GUI integrations
    • SO
    •   Long and thanks for all the fish! Graphics: www.tatlin.net @alvagante
    • SO
    •   Long and thanks for all the fish! Graphics: www.tatlin.net @alvagante