Your SlideShare is downloading. ×
  • Like
Anatomy of a Reuseable Module
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Anatomy of a Reuseable Module

  • 5,095 views
Published

What is a reusable Puppet module? What are its requirements? Why does it matter and why it's not always the best choice? And, most of all, HOW do you make reusable modules? This session is all about …

What is a reusable Puppet module? What are its requirements? Why does it matter and why it's not always the best choice? And, most of all, HOW do you make reusable modules? This session is all about this: How to make modules written in a way that you can use them in many different environments (also with different OS) without modifying them.

Alessandro Franceschi
Freelance Consultant, Lab42
Founded an Internet Service Provider in 1995. Founded a IT consulting firm in 1999. Decided that it was funnier to do technical stuff rather than to manage companies and became a freelance consultant in 2006. Discovered Puppet in 2007. Passed the last years pursuing an insane passion for fun and profit: Puppet consulting and modules development.

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
5,095
On SlideShare
0
From Embeds
0
Number of Embeds
7

Actions

Shares
Downloads
26
Comments
0
Likes
3

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Anatomy of a Reusable Module Alessandro Franceschi github.com/example42 PuppetConf 2013 Saturday, August 24, 13
  • 2. Be
  • 3.   patient please Saturday, August 24, 13
  • 4. How
  • 5.   do we
  • 6.   use Puppet today? Include classes Set Params & Variables Define Business & Integration Logic Provide Configuration Files Manage Resources Saturday, August 24, 13
  • 7. How do we use Puppet today Include classes manifests/ site.pp Set Parameters / Variables Integration logic Resources ENC HIERA SITE MODULES SHARED MODULES BAD EDGE Configuration files manifests/ site.pp ENC HIERA manifests/ site.pp BAD? BAD? BAD SITE MODULES SITE MODULES SHARED MODULES SITE MODULES SHARED MODULES manifests/ site.pp BAD? ENC BAD? Saturday, August 24, 13
  • 8. A
  • 9.   reusable
  • 10.    module is
  • 11.   all
  • 12.   about CHOICE Operating Systems Infrastructures Scales Node classifiers Installation methods Alternative setups Saturday, August 24, 13
  • 13. The
  • 14.   cons of
  • 15.   a reusable module Harder & Longer Development Complexity Verbosity Not Optimized for performance Saturday, August 24, 13
  • 16. THE PARAMETERS DILEMMA Managed resources attributes Application specific config options Application logic and behaviour Integration with other modules Saturday, August 24, 13
  • 17. Parameters: Resources attributes Enough: $package = $redis::params::package, $service = $redis::params::service, $service_ensure = 'running', $service_enable = true, $file = $redis::params::file, $file_notify = "Service['redis']", $file_source = undef, $file_content = undef, Too much? $package_provider = undef, $file_owner = $redis::params::file_owner, $file_group = $redis::params::file_group, $file_mode = $redis::params::file_mode, $file_replace = $redis::params::file_replace, Benefits from: A standard naming convention Saturday, August 24, 13
  • 18. Parameters: Application options Enough: $puppet_server = “puppet.${::domain}”, $syslog_server = “syslog.${::domain}”, $munin_server = “munin.${::domain}”, $dns_servers = [ '8.8.8.8' , '8.8.4.4' ], Too much! $anonymous_enable = true, $anon_mkdir_write_enable = true, $anon_upload_enable = false, $chroot_list_enable = true, $chroot_list_file = '/etc/vsftpd/chroot_list', $resourcefile = $nagios::params::resourcefile, $statusfile = $nagios::params::statusfile, $commandfile = $nagios::params::commandfile, $resultpath = $nagios::params::resultpath, $retentionfile = $nagios::params::retentionfile, $p1file = $nagios::params::p1file, Benefits from: Template + Options Hash pattern Saturday, August 24, 13
  • 19. Parameters: Application logic Examples: $install_client = true, $install_stomp_server = true, $install_plugins = true, $use_ssl = false, $munin_autoconfigure = true, $service_autorestart = true, $manage_package_repo = true, $run_initdb = undef, Benefits from: A standard naming convention Saturday, August 24, 13
  • 20. Parameters: Modules Integrations Examples: $mongo_db_host = $graylog2::params::mongo_db_host, $mongo_db_port = $graylog2::params::mongo_db_port, $mongo_db_name = $graylog2::params::mongo_db_name, $mongo_user = $graylog2::params::mongo_user, $mongo_password = $graylog2::params::mongo_password, $elasticsearch_template = $graylog2::params::elasticsearch_template, $elasticsearch_path = $graylog2::params::elasticsearch_path, $database = $puppetdb::params::database, $manage_redhat_firewall = $puppetdb::params::manage_redhat_firewall, $db_type = 'mysql', Benefits from: Shared Stacks Saturday, August 24, 13
  • 21. PATTERNS REUSABILITY Saturday, August 24, 13
  • 22. Managing
  • 23.    files Let user decide how to manage configuration files. Alternatives: source content concat augeas custom types Saturday, August 24, 13
  • 24. Managing files: source & content redis/manifests/init.pp class redis ( $file = $redis::params::file, $file_source = undef, $file_template = undef, $file_content = undef, ) { [...] $managed_file_content = $file_content ? { undef => $file_template ? { undef => undef, default => template($file_template), }, default => $file_content, } [...] if $redis::file { file { 'redis.conf': path => $redis::file, source => $redis::file_source, content => $redis::managed_file_content, } } } Provide the Puppet path of an erb template class { ‘redis’: file_template => ‘site/redis/ redis.conf.erb’, } Provide directly the content attribute class { ‘redis’: file_content => “template(‘site/redis/ redis.conf.erb’)”, } Provide a fileserver source path class { ‘redis’: file_source => ‘puppet:///modules/site/ redis/redis.conf’, } Manage the configuration file with other methods (augeas, concat...) class { ‘redis’: } Saturday, August 24, 13
  • 25. Multiple
  • 26.    config
  • 27.   files
  • 28.    Add parameters to main class Use a generic conf define Manage the whole configuration dir Saturday, August 24, 13
  • 29. Multiple files: Add parameters elasticsearch/manifests/init.pp class elasticsearch ( $file = $elasticsearch::params::file, $file_source = undef, $file_template = undef, $file_content = undef, [...] $init_script_file = '/etc/init.d/elasticsearch', $init_script_file_template = 'elasticsearch/init.erb', $init_options_file = $elasticsearch::params::init_options_file, $init_options_file_template = 'elasticsearch/init_options.erb', Provide custom templates for the main file and the init script class { ‘elasticsearch’: file_template => ‘site/elasticsearch/elasticsearch.yml.erb’, init_script_file_template => ‘site/elasticsearch/elasticsearch.init.erb’, } Saturday, August 24, 13
  • 30. Multiple files: Generic conf define nova/manifests/conf.pp define nova::conf ( $source = undef, $template = undef, $content = undef, $path = undef, [...] $options_hash = undef, $ensure = present ) { include nova $managed_path = $path ? { undef => "${nova::config_dir}/${name}", default => $path, } [...] file { "nova_conf_${name}": ensure => $ensure, source => $source, content => $managed_content, path => $managed_path, mode => $managed_mode, owner => $managed_owner, group => $managed_group, require => $managed_require, notify => $managed_notify, replace => $managed_replace, } } Provide a custom template for an alternative config file in config_dir nova::conf { ‘rootwrap.conf’: template => ‘site/nova/rootwrap.conf.erb’, } Saturday, August 24, 13
  • 31. Multiple files: Whole config dir redis/manifests/init.pp class redis ( $dir = $redis::params::dir, $dir_source = undef, $dir_purge = false, $dir_recurse = true, ) { [...] $dir_ensure = $ensure ? { 'absent' => 'absent', 'present' => 'directory', } if $redis::dir_source { file { 'redis.dir': ensure => $redis::dir_ensure, path => $redis::dir, source => $redis::dir_source, recurse => $redis::dir_recurse, purge => $redis::dir_purge, force => $redis::dir_purge, notify => $redis::file_notify, require => $redis::file_require, } } } Provide a custom source for the whole config_dir class { ‘redis’: dir_source => ‘puppet:///modules/site/redis/conf/’, } Provide a custom source for the whole config_dir and purge any not managed config file class { ‘redis’: dir_source => ‘puppet:///modules/site/redis/conf/’, dir_purge => true, } Saturday, August 24, 13
  • 32. Managing
  • 33.    Users Everyone has his own users... Leave options to decide if, how and where to manage the ones the module requires. Saturday, August 24, 13
  • 34. Managing Users elasticsearch/manifests/init.pp class elasticsearch { $ensure = 'present', [...] $user = 'elasticsearch', $user_uid = undef, $user_gid = undef, $user_groups = undef, $user_class = 'elasticsearch::user', [...] if $elasticsearch::user_class { require $elasticsearch::user_class } elasticsearch/manifests/user.pp class elasticsearch::user { @user { $elasticsearch::user : ensure => $elasticsearch::ensure, comment => "${elasticsearch::user} user", password => '!', managehome => false, uid => $elasticsearch::user_uid, gid => $elasticsearch::user_gid, groups => $elasticsearch::user_groups, shell => '/bin/bash', } User <| title == $elasticsearch::user |> } Do not create the requested user class { ‘elasticsearch’: user_class => undef, } Provide the user in a different custom class class { ‘elasticsearch’: user_class => 'site::users', } Run elasticsearch with a different user class { ‘elasticsearch’: user => 'apache', } Saturday, August 24, 13
  • 35. Managing extra
  • 36.    resources Options to specify custom classes Options to pass an hash to create_resources Saturday, August 24, 13
  • 37. Extra Resources: Custom classes elasticsearch/manifests/init.pp class elasticsearch ( $dependency_class = 'elasticsearch::dependency', $monitor_class = 'elasticsearch::monitor', $firewall_class = 'elasticsearch::firewall', $my_class = undef, ) { [...] if $elasticsearch::dependency_class { include $elasticsearch::dependency_class } if $elasticsearch::monitor and $elasticsearch::monitor_class { include $elasticsearch::monitor_class } if $elasticsearch::firewall and $elasticsearch::firewall_class { include $elasticsearch::firewall_class } if $elasticsearch::my_class { include $elasticsearch::my_class }[...] Provide the modules dependencies with a custom class class { ‘elasticsearch’: dependency_class => 'site::dep_elasticsearch', } Saturday, August 24, 13
  • 38. Extra Resources: Resources Hash elasticsearch/manifests/init.pp class elasticsearch ( $create_resource = undef, $resources_hash = undef, ) { [...] if $create_resource { create_resources( $create_resource , $resources_hash ) } Alternative: A single hash that includes resources and resources_hash Provide the modules dependencies with a custom class class { ‘elasticsearch’: create_resource => 'file', resources_hash => { path => '/etc/elasticsearch/my_file', content => template('site/elasticsearch/my_file.erb), mode => '0600', }, } Saturday, August 24, 13
  • 39. Managing Packages and Services Names change Custom packages are common Leave choice, optionally Saturday, August 24, 13
  • 40. Managing packages openssh/manifests/init.pp class openssh ( $ensure = 'present', $version = undef, $package = $openssh::params::package, [...] ) { if $version and $ensure == 'present' { $managed_package_ensure = $version } else { $managed_package_ensure = $ensure } if $openssh::package { package { $openssh::package: ensure => $openssh::managed_package_ensure, } } openssh/manifests/params.pp class openssh::params { $package = $::osfamily ? { Suse => 'openssh', OpenBSD => '', default => 'openssh-server', } Install a custom company-openssh package class { ‘openssh’: package => 'company-openssh', } Saturday, August 24, 13
  • 41. Managing services openssh/manifests/init.pp class openssh ( $service = $openssh::params::service, $service_ensure = 'running', $service_enable = true, [...] ) { if $ensure == 'absent' { $managed_service_enable = undef $managed_service_ensure = stopped } else { $managed_service_enable = $service_enable $managed_service_ensure = $service_ensure } if $openssh::service { service { $openssh::service: ensure => $openssh::managed_service_ensure, enable => $openssh::managed_service_enable, } } openssh/manifests/params.pp class openssh::params { $service = $::osfamily ? { Debian => 'ssh', default => 'sshd', } [...] Manage a custom company-openssh service class { ‘openssh’: service => 'company-openssh', } Saturday, August 24, 13
  • 42. Managing Installation
  • 43.    options Let users decide: OS Packages Upstream tarballs Provider Saturday, August 24, 13
  • 44. Installation options elasticsearch/manifests/init.pp class elasticsearch ( $package_provider = undef, $install = 'package', $install_base_url = $elasticsearch::params::install_base_url, $install_source = undef, $install_destination = '/opt', ) { [...] $managed_file = $elasticsearch::install ? { package => $elasticsearch::file, default => "${elasticsearch::home_dir}/config/elasticsearch.yml", } [...] case $elasticsearch::install { package: { package { $elasticsearch::package: ensure => $elasticsearch::managed_package_ensure, provider => $elasticsearch::package_provider, } } upstream: { puppi::netinstall { 'netinstall_elasticsearch': url => $elasticsearch::managed_install_source, destination_dir => $elasticsearch::install_destination, owner => $elasticsearch::user, group => $elasticsearch::user, } [...] Install elasticsearch from upstream source class { ‘elasticsearch’: install => 'upstream', install_source => 'https://download.elasticsearch.org/ elasticsearch/elasticsearch/elasticsearch-0.90.3.zip', } Saturday, August 24, 13
  • 45. Templates and
  • 46.   hashes Managing specific application configs parameters may get out of control A single config hash to show them all A custom template to use them Application specific configs THE PARAMETERS DILEMMA Saturday, August 24, 13
  • 47. Options Hash: Setup openssh/manifests/init.pp class openssh ( [...] $file_template = undef, $options_hash = undef, site/templates/openssh/sshd_config.erb # File Managed by Puppet [...] Port <%= scope.function_options_lookup(['Port','22']) %> PermitRootLogin <%= scope.function_options_lookup(['PermitRootLogin','yes']) %> UsePAM <%= scope.function_options_lookup(['UsePAM','yes']) %> [...] * Function options_lookup currently in Example42's Puppi module Alternative site/templates/openssh/sshd_config.erb Port <%= scope.lookupvar('openssh::options_hash')['Port'] ||='22' %> PermitRootLogin <%= scope.lookupvar('openssh::options_hash')['PermitRootLogin'] ||='yes' %> UsePAM <%= scope.lookupvar('openssh::options_hash')['UsePAM'] ||='yes' %> [...] Saturday, August 24, 13
  • 48. Options Hash: Usage Usage (with Hiera): include openssh /etc/puppet/hieradata/global.yml: --- openssh::file_template: 'site/openssh/sshd_config.erb' openssh::file_options_hash: Port: '22222' PermitRootLogin: 'no' Usage (with parametrized class): class { 'openssh': file_template => 'site/openssh/sshd_config.erb' file_options_hash => { Port => '22222', PermitRootLogin => 'no', } Saturday, August 24, 13
  • 49. STANDARDS NAMING Managed resources attributes THE PARAMETERS DILEMMA Saturday, August 24, 13
  • 50. The Handy
  • 51.   Grail of
  • 52.   Modules
  • 53.    Standards A blog post* Some discussions on Puppet-Users github.com/stdmod Naming standards for modules parameters Community driven (draft 0.0.2) * http://www.example42.com/?q=The_handy_Grail_of_Modules_Standards Saturday, August 24, 13
  • 54. Benefits
  • 55.   of
  • 56.    suggested (and
  • 57.   shared) naming
  • 58.    conventions Saner User Experience Better modules Interoperability Reusability Patterns Predictability in usage and development Saturday, August 24, 13
  • 59. Stdmod Params: Main resources ### General parameters ensure (enable?) version (package_version?) ### Package - Service - Main configuration file package (package_name?) package_ensure package_provider package_* [any relevant package type attribute] service (service_name?) service_ensure service_enable service_subscribe service_* file (file_path? config_file? config?) file_source (source? config_file_source? config_source?) file_template (template? config_file_template? config_template?) file_content (content? config_file_content? config_content?) file_* (config_file_*? config_*?) file_options_hash (options? options_hash? file_options?) Saturday, August 24, 13
  • 60. Stdmod Params: Extra resources other_package other_package_* client_package client_package_* server_package server_package_* other_service other_service_* log_file log_file_* pid_file pid_file_* init_script_file init_script_file_* init_config_file init_config_file_* Saturday, August 24, 13
  • 61. Stdmod Params: Installation ### Parameter related parameters install install_url install_base_url install_source install_destination install_pre_exec install_pre_exec_* install_post_exec install_post_exec_* install_script_file install_script_file_* install_response_file install_response_file_* Saturday, August 24, 13
  • 62. Stdmod Params: Monitor monitor monitor_tool monitor_host monitor_port monitor_protocol monitor_url monitor_process monitor_service monitor_config_hash Saturday, August 24, 13
  • 63. Stdmod Params: Firewall firewall firewall_src firewall_dst firewall_port firewall_protocol Saturday, August 24, 13
  • 64. STACKS MODULES Integrations of modules THE PARAMETERS DILEMMA Saturday, August 24, 13
  • 65. Why
  • 66.    reinventing
  • 67.    the
  • 68.   stack? We always use stacks. We need them to make something useful with modules. What about: Sharing? Best practices? Standardization? Saturday, August 24, 13
  • 69. Stacks - A Simple Sample class stack::logs ( $ensure = 'present', $syslog_server = false, $syslog_server_port = '5544', $elasticsearch_server = false, $elasticsearch_server_port = '9200', $elasticsearch_cluster = 'logs', $elasticsearch_java_opts = '-Xmx2g -Xms1g', $install_logstash = false, $install_elasticsearch = false, $install_kibana = false, $install_graylog2 = false, $install_graylog2_webinterface = false, $syslog_config_template = 'stack/logs/syslog.conf.erb', $logstash_config_template = 'stack/logs/logstash.conf.erb', $elasticsearch_config_template = 'stack/logs/elasticsearch.yml.erb', $kibana_config_template = 'stack/logs/config.js.erb', $graylog2_config_template = 'stack/logs/graylog2.conf.erb', ) { [... TO BE CONTINUED ...] Saturday, August 24, 13
  • 70. Stacks - A Simple Sample [...] if $syslog_server { rsyslog::config { 'logstash_stack': content => template($syslog_config_template), } } if $install_logstash { class { 'logstash': template => $logstash_config_template, } } if $install_elasticsearch { class { 'elasticsearch': java_opts => $elasticsearch_java_opts, template => $elasticsearch_config_template, } } [...] Saturday, August 24, 13
  • 71. Stacks - Usage On any host: stack::logs { 'central': syslog_server => 'syslog.example42.com', } On the Logstash (syslog) server: stack::logs { 'central': syslog_server => 'syslog.example42.com', install_logstash => true, elasticsearch_server => 'el.example42.com', } On the Elasticsearch server(s), with a custom configuration file: stack::logs { 'central': syslog_server => 'syslog.example42.com', install_elasticsearch => true, elasticsearch_server => 'el.example42.com', elasticsearch_config_template => 'site/logs/elasticsearch.yml.erb', } On the Kibana server: stack::logs { 'central': syslog_server => 'syslog.example42.com', install_kibana => true, elasticsearch_server => 'el.example42.com', } Saturday, August 24, 13
  • 72. The Stacks Logic Stacks are local Modules are shared Higher level interface Integrate different set of modules Preserve modules local change Saturday, August 24, 13
  • 73. How do we use Puppet today Include classes manifests/ site.pp Set Parameters / Variables Integration logic Resources ENC HIERA SITE MODULES SHARED MODULES BAD EDGE Configuration files manifests/ site.pp ENC HIERA manifests/ site.pp BAD? BAD? BAD SITE MODULES SITE MODULES SHARED MODULES SITE MODULES SHARED MODULES manifests/ site.pp BAD? ENC BAD? STACKS STACKS STACKS Saturday, August 24, 13
  • 74. Standards for Stacks? Usual benefits: User Experience Interoperability Higher level API exposure Possible GUI Integrations Saturday, August 24, 13
  • 75. Steps Define stdmod naming conventions Explore Stacks design and approach Create templates for stacks and modules Explore GUI integrations Saturday, August 24, 13
  • 76. SO
  • 77.   Long and thanks for all the fish! Graphics: www.tatlin.net @alvagante Saturday, August 24, 13
  • 78. SO
  • 79.   Long and thanks for all the fish! Graphics: www.tatlin.net @alvagante Saturday, August 24, 13