Automated Puppet Testing - PuppetCamp Chicago '12 - Scott Nottingham
Upcoming SlideShare
Loading in...5
×
 

Automated Puppet Testing - PuppetCamp Chicago '12 - Scott Nottingham

on

  • 3,728 views

How IMC has implemented a CI/CD using Cobbler and Puppet, from a presentation given by Scott Nottingham & Eric Pogrelis at PuppetCamp Chicago '12.

How IMC has implemented a CI/CD using Cobbler and Puppet, from a presentation given by Scott Nottingham & Eric Pogrelis at PuppetCamp Chicago '12.

www.puppetlabs.com

Statistics

Views

Total Views
3,728
Views on SlideShare
3,616
Embed Views
112

Actions

Likes
9
Downloads
32
Comments
0

6 Embeds 112

http://mindreframer.github.com 39
http://mindreframer.github.io 32
http://localhost 27
http://puppetlabs.com 9
http://devopsbox.es 4
https://puppetlabs.com 1

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Automated Puppet Testing - PuppetCamp Chicago '12 - Scott Nottingham Automated Puppet Testing - PuppetCamp Chicago '12 - Scott Nottingham Presentation Transcript

  • Automated Puppet Testing “All code is guilty, until proven innocent.” – AnonymousScott Nottingham July 23rd, 2012
  • Before and After Testing• Before Testing – Unit tests • Cucumber-puppet • Rspec-puppet• After Testing – User Space tests • Did everything that was supposed to get done actually get done?
  • Unit Tests - frameworks• Cucumber-puppet – Tests the outcome of the entire module as opposed to just the manifests – Overly complicated by trying to be simple (imho)• Rspec-puppet – Unit tests for the manifests – Truly simple (and effective)
  • RSpec-puppet• Written by Tim Sharpe (rodjek)• Available on github – https://github.com/rodjek/rspec-puppet/
  • RSpec-puppet• PROs – Simple language and structure – Easy to maintain – Fast to execute – Execution easily automated• CONs – Only tests what you tell it to • Easy to make changes to manifests and forget to update RSpec tests
  • Writing RSpec Tests• Directory Tree module | +-- manifests | +-- files | +-- templates | +-- lib | +-- spec | +-- spec_helper.rb | +-- classes | +-- <class_name>_spec.rb
  • Writing RSpec Tests• spec_helper file – Required for RSpec to know where to find the module path – Resides within: <module_name>/spec/spec_helper.rb – contents: require puppet require rubygems require rspec-puppet RSpec.configure do |c| c.module_path = File.join(File.dirname(__FILE__), ../../) c.manifest = ../../../../manifests/site.pp end
  • Writing RSpec Tests• Spec file – Contains test logic – Naming convention: • <classname>_<pp_filename>_spec.rb • Ex. nagios_init_spec.rb – Contents • Requires – require ‘spec_helper’ • Tests
  • Writing RSpec Tests• Defining Tests – Tests are created using the ‘describe’ method. – The first test must describe the class defined in your manifest. describe ‘nagios’ do … … end
  • Writing RSpec Tests• Accounting for class dependencies – Some modules define class dependencies – Example: • Class*‘yum’+ -> Class*‘nagios’+ – These are handled in RSpec test with ‘:pre_condition’ let(:pre_condition) , class ,”yum":- -
  • Writing RSpec Tests• Setting parameters used by puppet manifests – Optional unless manifest needs them to compile – Handled with ‘:params’ – Example (nested parameters): let(:params) { {:key1 => value1, :key2 => value2, :key3 => {keyA => valA}, :key4 => {keyB => {keyX => valX} } } }
  • Writing RSpec Tests• Setting a node name for the tests (optional) – Might use if your manifests has node name dependent logic let(:node) { host.example.com }
  • Writing RSpec Tests• Setting custom fact values – Probably the most commonly used – Example: let(:facts) { {:fact1 => val1, :fact2 => val2} }
  • Writing RSpec Tests• Sub-tests – These are the tests for each condition that you want to test for a given manifest. describe nagios do describe with valid parameters passed do .... end describe with invalid parameters passed do .... end end
  • Writing RSpec Tests• Testing the resources• Generally speaking, the tests you will be writing will be to confirm that the class contains some set of resources. This is done with the ‘contain_<resource_type>’• Each resource will generally have attributes that you will want to test are present. This is done with the ‘.with_<attribute>(‘value’)’ method.• Example: it { should contain_file(foo).with_ensure(present).with_owner(root).with_group(root) }
  • Writing RSpec Tests• Cascading .with_ methods can get ugly!• Cascading should only be used for resources with 1 or 2 attributes defined.• Alternatively, you can use block notation for resources with many attributes it do should contain_file(/var/spool/squid/cache).with( require => Package[squid], ensure => directory, owner => squid, group => squid ) end
  • Writing RSpec Tests• Sometimes multiple values are assigned to a single attribute• These are handled like so: it do should contain_mount(/var/spool/squid/cache).with( require => [Logical_volume[lv_cache], Package[xfsprogs] ], .... .... ) end
  • Writing RSpec Tests• Templates – If your file content is generated by a template, rspec can test for this too. Here’s how: • Test that a template generates any content it should generate valid content for squid.conf do content = catalogue.resource(file, /etc/squid/squid.conf).send(:paramters)[:content] content.should_not be_empty End • Test that a template generates specific content it should generate valid content for squid.conf do content = catalogue.resource(file, /etc/squid/squid.conf).send(:paramters)[:content] content.should match(some_string) end
  • Writing RSpec Tests• Test for an expected error• Sometimes, you want to test that an error is raised. – For example, if not passing a parameter when one is needed, or passing an invalid parameter. – Example test it { expect { should contain_class(nagios::server) }.to raise_error(Puppet::Error) }
  • require spec_helperdescribe sendmail do let(:params) { {:mail_server => foobarbaz} } it { should contain_package(sendmail).with_ensure(installed) } it do should contain_file(/etc/mail/sendmail.cf).with( require => Package[sendmail], notify => Exec[makemap] ) end it should generate /etc/mail/sendmail.cf from a template do content = catalogue.resource(file, /etc/mail/sendmail.cf).send(:parameters)[:content] content.should match(foobarbaz) end it should generate /etc/mail/submit.cf from a template do content = catalogue.resource(file, /etc/mail/submit.cf).send(:parameters)[:content] content.should match(foobarbaz) end it do should contain_file(/etc/mail/submit.cf).with( require => Package[sendmail], notify => Exec[makemap] ) end it { should contain_file(/etc/sysconfig/sendmail).with_source(puppet:///sendmail/sendmail) } it do should contain_service(sendmail).with( require => Package[sendmail], enable => true, ensure => running ) endend
  • Running RSpec Tests• rspec –color –format documentation /etc/puppet/modules/sendmail/spec/classes/sendmail_init_spec.rb sendmail should contain Package[sendmail] with ensure => "installed" should contain File[/etc/mail/sendmail.cf] with notify => "Exec[makemap]" and require => "Package[sendmail]" should generate /etc/mail/sendmail.cf from a template should generate /etc/mail/submit.cf from a template should contain File[/etc/mail/submit.cf] with notify => "Exec[makemap]" and require => "Package[sendmail]" should contain File[/etc/sysconfig/sendmail] with source => "puppet:///sendmail/sendmail" should contain Service[sendmail] with enable => "true", ensure => "running" and require => "Package[sendmail]" Finished in 1.98 seconds 7 examples, 0 failures
  • Running RSpec Tests• Oops! Someone put a comma where they meant to put a semi-colon!SendmailFailures: 1) sendmail Failure/Error: it { should contain_package(sendmail).with_ensure(installed) } Puppet::Error: Syntax error at /etc/sysconfig/sendmail; expected } at/etc/puppetmaster/modules/sendmail/spec/../../sendmail/manifests/init.pp:19 on node neodymium.trading.imc.intra # ./sendmail_init_spec.rb:6 2) sendmail Failure/Error: it { should contain_package(sendmail).with_ensure(installed) } Puppet::Error: Syntax error at /etc/sysconfig/sendmail; expected } at/etc/puppetmaster/modules/sendmail/spec/../../sendmail/manifests/init.pp:19 on node neodymium.trading.imc.intra # ./sendmail_init_spec.rb:6Finished in 0.19808 seconds2 examples, 2 failures
  • Automating RSpec Tests• Tests can be run by anything capable of executing a system command• We integrate ours into Team City – Much like Jenkins – Allows for event driven execution of tests• Any time a commit occurs in version control, tests are executed – If tests pass, manifests are wrapped inside an RPM and pushed to puppet masters – If tests fail, we are notified
  • Automating RSpec Tests• RSpec tests do a great job of making sure manifests are syntactically correct and, generally speaking, function as expected.• But….Things aren’t always as they seem!
  • ‘After’ Tests• To ensure our environment really is clean we must test in user space land.• Probably many solutions out there – We wrote our own
  • User Space Testing• Post Provision Checks (PPC) is a framework for testing stuff in user space. – Originally written as health checks for testing a machine immediately after provisioning it and as verification testing after maintenance. – Quickly grew to an extensible framework capable of integrating with Nagios/Icinga and Teamcity, or running as stand alone application. – Contains libraries of functions for commonly used tasks. – Easily customize runtime options with profiles – Can be extended with plugins
  • PPC functionality
  • puppet_ust plugin• The puppet user space tests plugin was written to extend the framework for specifically testing puppet applied configurations. – Auto detects the puppet classes that are applied to a given box – Determines the parameters defined for the host being tested and utilizes them within tests. • Also uses facter facts in the same manner
  • That’s great, but puppet reports will tell youwhat worked and what didn’t…
  • misses• You wouldn’t believe the things we have seen go wrong with applying configurations (that the puppet logs/reports didn’t indicate). – PPC tests can be written to catch all of these! • Mistakes only happen once • Winning! – A few examples • Version of package specified not what got installed – not puppet’s fault, but still a problem. • Package got installed, but some key files were zero length • Drivers upgraded, but not properly loaded • Machines joined to wrong domain • Services running (per status) but not functioning properly • Yum repos configured but not functioning • Sysctl settings still not loaded after sysctl –p
  • PPC sample output• Stand Alone mode
  • PPC Sample Output• Team City Mode
  • Automating PPC Tests• Team City + multiplexed SSH (for parallelization)• Each version control commit results in a puppet run + user space test run in our dev and qa environments.• Over 20,000 tests performed in ~20 minutes – This means every 20 minutes new, fully tested changes are pushed from the development environment to QA – Each stage of the pipeline is tested in a similar fashion