Puppet Module Reusability - What I Learned from Shipping to the Forge

6,334 views

Published on

A simple search for "puppet-apache" on GitHub returns 70 separate repositories. An awful lot of people are busy reinventing the same configuration wheel. Configuration management tools promise write once, run anywhere code; but writing code that can be used by anyone looks like a lot of work. This presentation aims to show anyone familiar with Puppet how to write reusable modules and importantly how to make them compatible with already shared modules released on the Forge or elsewhere. We'll look at when and why testing a declarative language is actually useful, examples of good and bad modules and how to re-factor puppet code for re-usability. We'll also talk about potential improvements to Puppet that would make reuse easier.

Gareth Rushgrove
Technical Architect, Government Digital Service
Gareth Rushgrove is now a technical architect at the Government Digital Service, part of the UK Government. He is mainly interested in configuration management, infrastructure and platform as a service, deployment and monitoring tooling and the whole devops community. He thinks when used well together these allow you to move really fast, even in tightly controlled environments like Government. When not working, Gareth can be found blogging over on morethanseven.net or uploading code to GitHub. He also curates the Devops Weekly newsletter and occasionally organises community events.

0 Comments
11 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
6,334
On SlideShare
0
From Embeds
0
Number of Embeds
2,098
Actions
Shares
0
Downloads
90
Comments
0
Likes
11
Embeds 0
No embeds

No notes for slide

Puppet Module Reusability - What I Learned from Shipping to the Forge

  1. 1. Puppet Module Reusability Learning from shipping to the Forge Gareth Rushgrove
  2. 2. Who (Who is this person?)
  3. 3. @garethr
  4. 4. UK Government Digital Service
  5. 5. Last code I wrote
  6. 6. 7. Gareth Rushgrove
  7. 7. Last puppet module I wrote
  8. 8. The Problem (Sharing modules is hard)
  9. 9. Not invented here syndrome 1
  10. 10. A search for puppet-apache
  11. 11. 145 Gareth Rushgrove
  12. 12. 145Gareth Rushgrove
  13. 13. Puppetlabs apache
  14. 14. 67 contributors Gareth Rushgrove
  15. 15. 281 forks Gareth Rushgrove
  16. 16. 281Gareth Rushgrove
  17. 17. Vendor everything pattern 2
  18. 18. Not publishing to the Forge 3
  19. 19. Puppet Forge
  20. 20. Undocumented modules 4
  21. 21. Official docs guidelines
  22. 22. (Too) opinionated configuration 5
  23. 23. Sometimes I just want to write nginx configuration Gareth Rushgrove
  24. 24. Duplicate resources 6
  25. 25. Package { 'build-essential’: ensure => installed, } Gareth Rushgrove
  26. 26. Stdlib (Start here)
  27. 27. Standard Library from Puppetlabs
  28. 28. Fail fast Gareth Rushgrove
  29. 29. Validation Gareth Rushgrove
  30. 30. Useful error messages Gareth Rushgrove
  31. 31. fail("${::operatingsystem} not supported") Gareth Rushgrove
  32. 32. validate_string Gareth Rushgrove
  33. 33. validate_string($version) Gareth Rushgrove
  34. 34. <Puppet::Error: true is not a string. Gareth Rushgrove
  35. 35. validate_slength Gareth Rushgrove
  36. 36. validate_re Gareth Rushgrove
  37. 37. validate_hash Gareth Rushgrove
  38. 38. Gareth Rushgrove validate_cmd
  39. 39. validate_bool Gareth Rushgrove
  40. 40. Avoid duplicate resources Gareth Rushgrove
  41. 41. Package { 'build-essential’: ensure => installed, } Gareth Rushgrove
  42. 42. include 'gcc' Gareth Rushgrove
  43. 43. package{[ 'python-pip', 'python-ldap', ]: ensure => installed } Gareth Rushgrove
  44. 44. ensure_packages Gareth Rushgrove
  45. 45. ensure_packages([ 'python-pip', 'python-ldap', ]) Gareth Rushgrove
  46. 46. ensure_resource Gareth Rushgrove
  47. 47. Nicer interfaces Gareth Rushgrove
  48. 48. str2bool Gareth Rushgrove
  49. 49. any2array Gareth Rushgrove
  50. 50. And much more Gareth Rushgrove
  51. 51. Dependency management is everywhere (Else)
  52. 52. NPM, Bundler, Pip, Mix, Leiningen Gareth Rushgrove
  53. 53. Librarian Puppet
  54. 54. r10k
  55. 55. echo 'modules' >> .gitignore Gareth Rushgrove
  56. 56. dependency 'puppetlabs/stdlib' dependency 'garethr/erlang' dependency 'maestrodev/wget' Gareth Rushgrove
  57. 57. forge "http://forge.puppetlabs.com" mod 'puppetlabs/ruby' mod 'puppetlabs/ntp' mod 'puppetlabs/git' mod 'puppetlabs/vcsrepo' mod 'puppetlabs/apt' mod 'puppetlabs/gcc' Gareth Rushgrove
  58. 58. librarian-puppet install Gareth Rushgrove
  59. 59. A Nice Module Pattern (Structuring modules for reuse)
  60. 60. R.I.Pienaar
  61. 61. install, config, service, params Gareth Rushgrove
  62. 62. manifests !"" config.pp !"" init.pp !"" install.pp !"" params.pp #"" service.pp Gareth Rushgrove
  63. 63. class sample ( ) inherits sample::params { class { 'sample::install': } -> class { 'sample::config': } ~> class { 'sample::service': } -> Class['sample'] } Gareth Rushgrove
  64. 64. anchor { 'sample::begin': } -> class { 'sample::install': } -> class { 'sample::config': } class { 'sample::service': } -> anchor { 'sample::end': } Gareth Rushgrove
  65. 65. Anchor['sample::begin'] ~> Class['sample::service'] Class['sample::install'] ~> Class['sample::service'] Class['sample::config'] ~> Class['sample::service'] Gareth Rushgrove
  66. 66. Puppet module tool Gareth Rushgrove
  67. 67. puppet module generate name Gareth Rushgrove
  68. 68. Default module skeleton Gareth Rushgrove
  69. 69. In puppet source code
  70. 70. !"" manifests !"" spec !"" tests !"" Modulefile !"" .gitignore #"" README Gareth Rushgrove
  71. 71. Creating your own module skeleton Gareth Rushgrove
  72. 72. ~/.puppet/var/puppet-module/skeleton Gareth Rushgrove
  73. 73. puppet module skeleton
  74. 74. !"" manifests !"" spec !"" tests !"" templates !"" files !"" lib !"" Gemfile Gareth Rushgrove
  75. 75. !"" Rakefile !"" .nodeset.yml !"" .fixtures.yml !"" .travis.yml !"" Modulefile !"" .gitignore #"" README.md Gareth Rushgrove
  76. 76. Creates install, config, service, params classes Gareth Rushgrove
  77. 77. Dependency management with Bundler Gareth Rushgrove
  78. 78. Better unit testing setup using rspec- puppet-helper Gareth Rushgrove
  79. 79. Adds syntax and lint checking Gareth Rushgrove
  80. 80. Adds Travis CI configuration Gareth Rushgrove
  81. 81. Adds integration tests with rspec-system Gareth Rushgrove
  82. 82. Adds module management with maestrodev blacksmith Gareth Rushgrove
  83. 83. Focus on the interface Gareth Rushgrove
  84. 84. Minimize number of entry points Gareth Rushgrove
  85. 85. Allow overriding provided templates Gareth Rushgrove
  86. 86. Multi-OS Support (Where to abstract the differences)
  87. 87. Vary default parameters Gareth Rushgrove
  88. 88. Keep logic in one place Gareth Rushgrove
  89. 89. case $::osfamily { 'Debian': {   }   'RedHat', 'Amazon': { } default: { fail("${::operatingsystem} not su } Gareth Rushgrove
  90. 90. ARM-9 Data in Modules
  91. 91. garethr-erlang
  92. 92. 0.0.x supported Ubuntu only Gareth Rushgrove
  93. 93. 0.1.x supports Ubuntu, Debian, Redhat, Suse Gareth Rushgrove
  94. 94. Gareth Rushgrove
  95. 95. Pull requests to the rescue
  96. 96. Insist on tests with open source contributions Gareth Rushgrove
  97. 97. Module Testing (Code should have tests)
  98. 98. Why test a declarative language? Gareth Rushgrove
  99. 99. Modules increasingly contain logic Gareth Rushgrove
  100. 100. Modules increasingly take arguments Gareth Rushgrove
  101. 101. Modules increasingly have interfaces with other modules Gareth Rushgrove
  102. 102. Modules increasingly used in many Ruby and Puppet version combinations Gareth Rushgrove
  103. 103. Good news. The tools got better Gareth Rushgrove
  104. 104. puppet-lint Gareth Rushgrove
  105. 105. Puppet style guide
  106. 106. Available as a gem
  107. 107. puppet-lint --with-filename /etc/puppet/modules foo/manifests/bar.pp: trailing whitespace found on line 1 apache/manifests/server.pp: variable not enclosed in {} on line 56 Gareth Rushgrove
  108. 108. puppet-syntax Gareth Rushgrove
  109. 109. Validate Puppet and ERB syntax
  110. 110. require 'puppet-syntax/tasks/puppet-syntax' Gareth Rushgrove
  111. 111. bundle exec rake syntax ---> syntax:manifests ---> syntax:templates Gareth Rushgrove
  112. 112. rspec-puppet Gareth Rushgrove
  113. 113. Unit testing for Puppet
  114. 114. context "epel enabled" do let(:params) {{ :epel_enable => true }} it { should contain_class('epel') } end Gareth Rushgrove
  115. 115. context "epel disabled" do let(:params) {{ :epel_enable => false }} it { should_not contain_class('epel') } end Gareth Rushgrove
  116. 116. Fixtures, matchers
  117. 117. Test the interface not the implementation Gareth Rushgrove
  118. 118. All of the above Gareth Rushgrove
  119. 119. task :test => [ :syntax, :lint, :spec, ] Gareth Rushgrove
  120. 120. bundle exec rake test Gareth Rushgrove
  121. 121. Gareth Rushgrove
  122. 122. Nice continuous integration
  123. 123. Test pull request branches too
  124. 124. --- language: ruby before_install: rm Gemfile.lock || true rvm: - 1.8.7 - 1.9.3 script: bundle exec rake test env: matrix: - PUPPET_VERSION="~> 2.7.0" - PUPPET_VERSION="~> 3.1.0" - PUPPET_VERSION="~> 3.2.0" Gareth Rushgrove
  125. 125. A matrix of possibilities Gareth Rushgrove
  126. 126. rspec-system Gareth Rushgrove
  127. 127. Integration test against real systems
  128. 128. Puppet helpers for rspec-system
  129. 129. Vagrant driver
  130. 130. VSphere provider
  131. 131. Test that Puppet runs without errors Gareth Rushgrove
  132. 132. it 'should run without errors' do pp = "class { 'sample': }" puppet_apply(pp) do |r| r.exit_code.should == 2 end end Gareth Rushgrove
  133. 133. Test runs are idempotent Gareth Rushgrove
  134. 134. it 'should run without errors' do pp = "class { 'sample': }" puppet_apply(pp) do |r| r.exit_code.should == 2 r.refresh r.exit_code.should be_zero end end Gareth Rushgrove
  135. 135. Test that the module installs relevant binaries Gareth Rushgrove
  136. 136. it 'should install the erl binary' do shell 'which erl' do |r| r.stdout.should =~ //usr/bin/e r.stderr.should be_empty r.exit_code.should be_zero end end Gareth Rushgrove
  137. 137. Test against different operating systems Gareth Rushgrove
  138. 138. default_set: 'centos-64-x64' sets: 'centos-64-x64': nodes: "main.foo.vm": prefab: 'centos-64-x64' 'fedora-18-x64': nodes: "main.foo.vm": prefab: 'fedora-18-x64' 'debian-607-x64': nodes: "main.foo.vm": Gareth Rushgrove
  139. 139. Room for improvements (Making reuse easier in Puppet)
  140. 140. Run your own Forge 1
  141. 141. A more secure Forge 2
  142. 142. Bless a dependency management tool 3
  143. 143. Optional dependencies 4
  144. 144. dependency 'puppetlabs/stdlib' optional_dependency 'puppetlabs/apache' Gareth Rushgrove
  145. 145. Private classes 5
  146. 146. private class elixir::install { Gareth Rushgrove
  147. 147. Parameter naming conventions 6
  148. 148. example42 Proposal
  149. 149. Interfaces in Puppet 7
  150. 150. interface packageandservice { $version $enable $package_name $template } Gareth Rushgrove
  151. 151. class diamond ( $version, $enable, $package_name, $template, ) { implements packageandservice Gareth Rushgrove
  152. 152. Questions? (And thanks for listening)

×