Testing for Ops: Going Beyond the Manifest - PuppetConf 2013

5,597 views

Published on

"Testing for Ops: Going Beyond the Manifest" by Christopher Webber, Infrastructure Engineer, Demand Media.

Presentation Overview: This talk aims to show the value of rspec-puppet for those who come from a more Ops-centric background. The focus will be on using tests to go beyond just rewriting manifests in rspec. Instead the focus will be on scenarios like: - Are the baseline security measures in place? - Do the differences between dev and prod get reflected? - Are the config elements that are core to the application present? In addition, tests will help to be a place to help document the oddities of our configurations and ensuring that minor changes don't result in catastrophe.

Speaker Bio: After beginning his career at UC Riverside supporting enterprise operations and bioinformatics research, Chris is now rocking being an infrastructure engineer at Demand Media in Santa Monica. He currently supports large high-traffic sites like eHow.com, LiveSTRONG.com, and Cracked.com. Chris enjoys attending local meetups, writing new Puppet modules, and creating small tools to make his team's lives a little easier. Find him on Twitter as @cwebber.

Published in: Technology
  • Be the first to comment

Testing for Ops: Going Beyond the Manifest - PuppetConf 2013

  1. 1. TESTING FOR OPS: GOING BEYOND THE MANIFEST Christopher Webber (@cwebber) Infrastructure Engineer, Demand Media
  2. 2. ABOUT ME • Infrastructure Engineer at Demand Media • Always been in Support or Operations Roles • Been a Puppet user since v 0.24.7 (2008)
  3. 3. ROLE MODULES VS LIBRARY MODULES Post By Craig Dunn (http://www.craigdunn.org/2012/05/239/)
  4. 4. OPS IMPLICITLY GETS TESTING
  5. 5. THEN WE TRY TO DO IT • And it is dumb • Might as well write a script to transform the code from one format to another • Start with unit testing frameworks, looking for integration testing
  6. 6. TYPES OF TESTING • Syntax checking (puppet parser validate) • Linting • Unit Tests • Integration Tests
  7. 7. SETTING UP OUR ENVIRONMENT • Ruby • Use the version you use in prod • Gems • puppet (use the version you use in prod) • rspec-puppet • puppet-lint • puppetlabs_spec_helper
  8. 8. THE TRIFECTA 1 # == Class: ssh 2 # 3 class ssh { 4 5 package { 6 'openssh-server': 7 ensure => installed 8 } 9 10 file { 11 '/etc/ssh/sshd_config': 12 ensure => present, 13 owner => 'root', 14 group => 'root', 15 mode => '0644', 16 content => template('ssh/sshd_config.erb'), 17 require => Package['openssh-server'] 18 } 19 20 service { 21 'sshd': 22 ensure => running, 23 enable => true, 24 hasstatus => true, 25 hasrestart => true, 26 subscribe => File['/etc/ssh/sshd_config'] 27 } 28 29 }
  9. 9. THE TESTS 1 require 'spec_helper' 2 3 describe 'ssh' do 4 5 it { should contain_package('openssh-server') } 6 7 it do 8 should contain_file('/etc/ssh/sshd_config') 9 .with_ensure('present') 10 .with_owner('root') 11 .with_group('root') 12 .with_mode('0644') 13 .with_require('Package[openssh-server]') 14 end 15 16 it do 17 should contain_service('sshd') 18 .with_ensure('running') 19 .with_enable(true) 20 .with_hasstatus(true) 21 .with_hasrestart(true) 22 .with_subscribe('File[/etc/ssh/sshd_config]') 23 end 24 end
  10. 10. IS THIS USEFUL? At first, I said NO
  11. 11. BUT WHY? It really doesn’t capture the things we care about.
  12. 12. ENTER INFOSEC • SSH MUST HAVE THE FOLLOWING SETTINGS: • PermitRootLogin no • X11Forwarding no
  13. 13. ADD USEFUL TESTS 25 context "InfoSec Requirements" do 26 27 it do 28 should contain_file('/etc/ssh/sshd_config') 29 .with_content(%r{^PermitRootLogin no$}) 30 end 31 32 it do 33 should contain_file('/etc/ssh/sshd_config') 34 .with_content(%r{^X11Forwarding no$}) 35 end 36 37 end
  14. 14. AND FAILFailures: 1) ssh InfoSec Requirements Failure/Error: .with_content(%r{^PermitRootLogin no$}) expected that the catalogue would contain File[/etc/ssh/sshd_config] with content matching `/^PermitRootLogin no$/` but its value of `"<file contents>"` does not # ./spec/classes/ssh_spec.rb:29:in `block (3 levels) in <top (required)>' 2) ssh InfoSec Requirements Failure/Error: .with_content(%r{^X11Forwarding no$}) expected that the catalogue would contain File[/etc/ssh/sshd_config] with content matching `/^X11Forwarding no$/` but its value of `"<file contents>"` does not # ./spec/classes/ssh_spec.rb:34:in `block (3 levels) in <top (required)>' Finished in 0.47736 seconds 5 examples, 2 failures Failed examples: rspec ./spec/classes/ssh_spec.rb:27 # ssh InfoSec Requirements rspec ./spec/classes/ssh_spec.rb:32 # ssh InfoSec Requirements
  15. 15. WHO CARES? • Can now fix and validate • Becomes one more check and balance • Safety in changes
  16. 16. OPERATIONAL EXAMPLES
  17. 17. LIBRARY MODULE EXAMPLES https://github.com/cwebberOps/puppetconf-ssh
  18. 18. CONTINUING WITH SSH • New Requirements • We have a system that other systems ssh to and drop info • We have determined that we need to increase the MaxStartups to 40
  19. 19. NEW TEST CODE 25 it do 26 should contain_file('/etc/ssh/sshd_config') 27 .with_content(%r{^MaxStartups 10$}) 28 end 29 30 context "maxstartups => 40" do 31 32 let (:params) {{ :maxstartups => 40 }} 33 34 it do 35 should contain_file('/etc/ssh/sshd_config') 36 .with_content(%r{^MaxStartups 40$}) 37 end 38 39 end https://github.com/cwebberOps/puppetconf-ssh
  20. 20. NEW PUPPET CODE 1 # == Class: ssh 2 # 3 class ssh ( 4 $maxstartups = 10 5 ){ https://github.com/cwebberOps/puppetconf-ssh
  21. 21. FEEDBACK LOOP • Did you remember to update the template with your new variable? • Much faster than vagrant destroy && vagrant up or even vagrant provision
  22. 22. MOVING ON TO THE ROLE • In the end, it is the module that matters the most • Should test that the config has the right config for the app https://github.com/cwebberOps/puppetconf-infra
  23. 23. TEST CODE 1 require 'spec_helper' 2 3 describe 'infra::collector' do 4 5 it do 6 should contain_file('/etc/ssh/sshd_config') 7 .with_content(%r{^MaxStartups 40$}) 8 end 9 10 end https://github.com/cwebberOps/puppetconf-infra
  24. 24. PUPPET CODE 1 # Class for setting up the infrastructure collector 2 class infra::collector { 3 4 class { 5 'ssh': 6 maxstartups => 40 7 } 8 9 } https://github.com/cwebberOps/puppetconf-infra
  25. 25. BUT THE TESTS... THEY DON’T WORK
  26. 26. MORE SETUP • Rakefile • puppet_rspec_helper • .fixtures.yml
  27. 27. RAKEFILE & SPEC_HELPER 1 require 'rake' 2 3 require 'rspec/core/rake_task' 4 require 'puppetlabs_spec_helper/rake_tasks' New 1 require 'rake' 2 3 require 'rspec/core/rake_task' 4 5 RSpec::Core::RakeTask.new(:spec) do |t| 6 t.pattern = 'spec/*/*_spec.rb' 7 end Old
  28. 28. .FIXTURES.YML 1 fixtures: 2 repositories: 3 ssh: "https://github.com/cwebberOps/puppetconf-ssh.git" 4 symlinks: 5 infra: "#{ source_dir }"
  29. 29. FUN EXAMPLES
  30. 30. INFRA::JENKINS 1 require 'spec_helper' 2 3 describe 'infra::jenkins' do 4 5 let (:facts) {{ 6 :osfamily => 'RedHat', 7 :operatingsystem => 'CentOS' 8 }} 9 10 it { should include_class('java') } 11 it { should include_class('jenkins') } 12 13 end https://github.com/cwebberOps/puppetconf-infra
  31. 31. INFRA::JENKINS 1 # Class for standing up a jenkins box 2 class infra::jenkins { 3 4 class { 5 'java': 6 } -> 7 8 class { 9 'jenkins': 10 } 11 12 } https://github.com/cwebberOps/puppetconf-infra
  32. 32. WHAT’S BROKE? http://projects.puppetlabs.com/issues/2053 The Bug The Tweet
  33. 33. INFRA::NODEJS & INFRA::NODEJS::DEV1 require 'spec_helper' 2 3 describe 'infra::nodejs' do 4 5 let (:facts) {{ 6 :osfamily => 'RedHat', 7 :operatingsystem => 'CentOS' 8 }} 9 10 it { should include_class('nodejs') } 11 12 it { should_not contain_package('nodejs-dev') } 13 14 it { should contain_file('/app_specific_stuff') } 15 16 end https://github.com/cwebberOps/puppetconf-infra
  34. 34. INFRA::NODEJS & INFRA::NODEJS::DEV 1 require 'spec_helper' 2 3 describe 'infra::nodejs::dev' do 4 5 let (:facts) {{ 6 :osfamily => 'RedHat', 7 :operatingsystem => 'CentOS' 8 }} 9 10 it { should include_class('nodejs') } 11 it { should contain_package('nodejs-dev') } 12 13 end https://github.com/cwebberOps/puppetconf-infra
  35. 35. INFRA::NODEJS & INFRA::NODEJS::DEV 1 # Demo class that sets up nodejs 2 class infra::nodejs { 3 4 class { 5 '::nodejs': 6 dev_package => false 7 } 8 9 file { 10 '/app_specific_stuff': 11 ensure => directory 12 } 13 14 } https://github.com/cwebberOps/puppetconf-infra
  36. 36. INFRA::NODEJS & INFRA::NODEJS::DEV 1 # Override class 2 class infra::nodejs::dev inherits infra::nodejs { 3 4 Class['::nodejs'] { 5 dev_package => true 6 } 7 8 } https://github.com/cwebberOps/puppetconf-infra
  37. 37. WHAT’S BROKE? http://projects.puppetlabs.com/issues/5517 The Bug
  38. 38. WHY DO THESE EXAMPLES MATTER?
  39. 39. WHAT’S NEXT • Integration testing using rspec-system • Continuous Integration
  40. 40. PREVIEW
  41. 41. RSPEC-SYSTEM1 require 'spec_helper_system' 2 3 describe 'ssh' do 4 5 it 'class should converge' do 6 pp = <<-EOS 7 class { 'ssh': } 8 EOS 9 10 puppet_apply(pp) do |r| 11 r.exit_code.should_not == 1 12 r.refresh 13 r.exit_code.should be_zero 14 end 15 end 16 17 describe service('sshd') do 18 it { should be_enabled } 19 it { should be_running } 20 end 21 end
  42. 42. JENKINS
  43. 43. QUESTIONS?

×