SlideShare a Scribd company logo
1 of 35
Automated Puppet Testing
      “All code is guilty, until proven
         innocent.” – Anonymous


Scott Nottingham                  July 23rd, 2012
Before and After Testing
• Before Testing
  – Unit tests
     • Cucumber-puppet
     • Rspec-puppet
• After Testing
  – User Space tests
     • Did everything that was supposed to get done actually
       get done?
Unit Tests - frameworks
• Cucumber-puppet
  – Tests the outcome of the entire module as
    opposed to just the manifests
  – Overly complicated by trying to be simple (imho)


• Rspec-puppet
  – Unit tests for the manifests
  – Truly simple (and effective)
RSpec-puppet
• Written by Tim Sharpe (rodjek)
• Available on github
  – https://github.com/rodjek/rspec-puppet/
RSpec-puppet
• PROs
  – Simple language and structure
  – Easy to maintain
  – Fast to execute
  – Execution easily automated

• CONs
  – Only tests what you tell it to
     • Easy to make changes to manifests and forget to update
       RSpec tests
Writing RSpec Tests
•   Directory Tree
     module
      |
      +-- manifests
      |
      +-- files
      |
      +-- templates
      |
      +-- lib
      |
      +-- spec
         |
         +-- spec_helper.rb
         |
         +-- classes
              |
              +-- <class_name>_spec.rb
Writing RSpec Tests
• spec_helper file
  – Required for RSpec to know where to find the module
    path
  – Resides within: <module_name>/spec/spec_helper.rb
  – contents:
        require 'puppet'
        require 'rubygems'
        require 'rspec-puppet'

        RSpec.configure do |c|
         c.module_path = File.join(File.dirname(__FILE__), '../../')
         c.manifest = '../../../../manifests/site.pp'
        end
Writing RSpec Tests
• Spec file
  – Contains test logic
  – Naming convention:
     • <classname>_<pp_filename>_spec.rb
     • Ex. nagios_init_spec.rb
  – Contents
     • Requires
        – require ‘spec_helper’
     • Tests
Writing RSpec Tests
• Defining Tests
  – Tests are created using the ‘describe’ method.
  – The first test must describe the class defined in
    your manifest.

        describe ‘nagios’ do
          …
          …
        end
Writing RSpec Tests
• Accounting for class dependencies
  – Some modules define class dependencies
  – Example:
     • Class*‘yum’+ -> Class*‘nagios’+
  – These are handled in RSpec test with
    ‘:pre_condition’

        let(:pre_condition) , 'class ,”yum":-' -
Writing RSpec Tests
• Setting parameters used by puppet manifests
  – Optional unless manifest needs them to compile
  – Handled with ‘:params’
  – Example (nested parameters):
  let(:params) { {:key1 => 'value1', :key2 => 'value2', :key3 => {'keyA' => 'valA'},
  :key4 => {'keyB' => {'keyX' => 'valX'} } } }
Writing RSpec Tests
• Setting a node name for the tests (optional)
  – Might use if your manifests has node name
    dependent logic

     let(:node) { host.example.com }
Writing RSpec Tests
• Setting custom fact values
  – Probably the most commonly used
  – Example:

     let(:facts) { {:fact1 => 'val1', :fact2 => 'val2'} }
Writing RSpec Tests
• Sub-tests
  – These are the tests for each condition that you want
    to test for a given manifest.
  describe 'nagios' do
   describe 'with valid parameters passed' do
     ....
   end

   describe 'with invalid parameters passed' do
     ....
   end
  end
Writing RSpec Tests
• Testing the resources
• Generally speaking, the tests you will be writing will be to confirm that the
  class contains some set of resources. This is done with the
  ‘contain_<resource_type>’
• Each resource will generally have attributes that you will want to test are
  present. This is done with the ‘.with_<attribute>(‘value’)’ method.
• Example:

    it { should contain_file('foo').with_ensure('present').with_owner('root').with_group('root') }
Writing RSpec Tests
• Cascading .with_ methods can get ugly!
• Cascading should only be used for resources with 1 or 2
  attributes defined.
• Alternatively, you can use block notation for resources with
  many attributes

   it do should contain_file('/var/spool/squid/cache').with(
        'require' => 'Package[squid]',
        'ensure' => 'directory',
        'owner' => 'squid',
        'group' => 'squid'
   ) end
Writing RSpec Tests
• Sometimes multiple values are assigned to a
  single attribute
• These are handled like so:
  it do should contain_mount('/var/spool/squid/cache').with(
     'require' => ['Logical_volume[lv_cache]', 'Package[xfsprogs]' ],
     ....
     ....
  ) end
Writing RSpec Tests
• Templates
   – If your file content is generated by a template, rspec can test for this
     too. Here’s how:
       • Test that a template generates any content
       it 'should generate valid content for squid.conf' do
             content = catalogue.resource('file', '/etc/squid/squid.conf').send(:paramters)[:content]
             content.should_not be_empty
       End


       • Test that a template generates specific content
       it 'should generate valid content for squid.conf' do
             content = catalogue.resource('file', '/etc/squid/squid.conf').send(:paramters)[:content]
             content.should match('some_string')
       end
Writing RSpec Tests
• Test for an expected error
• Sometimes, you want to test that an error is
  raised.
  – For example, if not passing a parameter when one
    is needed, or passing an invalid parameter.
  – Example test
     it { expect { should contain_class('nagios::server') }.to raise_error(Puppet::Error) }
require 'spec_helper'

describe 'sendmail' do
  let(:params) { {:mail_server => 'foobarbaz'} }

  it { should contain_package('sendmail').with_ensure('installed') }

  it do should contain_file('/etc/mail/sendmail.cf').with(
     'require' => 'Package[sendmail]',
     'notify' => 'Exec[makemap]'
  ) end

  it 'should generate /etc/mail/sendmail.cf from a template' do
      content = catalogue.resource('file', '/etc/mail/sendmail.cf').send(:parameters)[:content]
      content.should match('foobarbaz')
  end

  it 'should generate /etc/mail/submit.cf from a template' do
      content = catalogue.resource('file', '/etc/mail/submit.cf').send(:parameters)[:content]
      content.should match('foobarbaz')
  end

  it do should contain_file('/etc/mail/submit.cf').with(
     'require' => 'Package[sendmail]',
     'notify' => 'Exec[makemap]'
  ) end

  it { should contain_file('/etc/sysconfig/sendmail').with_source('puppet:///sendmail/sendmail') }

  it do should contain_service('sendmail').with(
     'require' => 'Package[sendmail]',
     'enable' => 'true',
     'ensure' => 'running'
  ) end
end
Running RSpec Tests
• rspec –color –format documentation
  /etc/puppet/modules/sendmail/spec/classes/sendmail_init_spec.rb


    sendmail
         should contain Package[sendmail] with ensure => "installed"
         should contain File[/etc/mail/sendmail.cf] with notify => "Exec[makemap]" and require =>
        "Package[sendmail]"
         should generate /etc/mail/sendmail.cf from a template
         should generate /etc/mail/submit.cf from a template
         should contain File[/etc/mail/submit.cf] with notify => "Exec[makemap]" and require =>
        "Package[sendmail]"
         should contain File[/etc/sysconfig/sendmail] with source => "puppet:///sendmail/sendmail"
         should contain Service[sendmail] with enable => "true", ensure => "running" and require =>
        "Package[sendmail]"
   Finished in 1.98 seconds
   7 examples, 0 failures
Running RSpec Tests
• Oops! Someone put a comma where they meant to put a semi-colon!

Sendmail
Failures:

 1) sendmail
   Failure/Error: it { should contain_package('sendmail').with_ensure('installed') }
   Puppet::Error:
    Syntax error at '/etc/sysconfig/sendmail'; expected '}' at
/etc/puppetmaster/modules/sendmail/spec/../../sendmail/manifests/init.pp:19 on node neodymium.trading.imc.intra
   # ./sendmail_init_spec.rb:6

 2) sendmail
   Failure/Error: it { should contain_package('sendmail').with_ensure('installed') }
   Puppet::Error:
    Syntax error at '/etc/sysconfig/sendmail'; expected '}' at
/etc/puppetmaster/modules/sendmail/spec/../../sendmail/manifests/init.pp:19 on node neodymium.trading.imc.intra
   # ./sendmail_init_spec.rb:6

Finished in 0.19808 seconds
2 examples, 2 failures
Automating RSpec Tests
• Tests can be run by anything capable of executing a
  system command
• We integrate ours into Team City
   – Much like Jenkins
   – Allows for event driven execution of tests
• Any time a commit occurs in version control, tests
  are executed
   – If tests pass, manifests are wrapped inside an RPM and
     pushed to puppet masters
   – If tests fail, we are notified
Automating RSpec Tests
• RSpec tests do a great job of making sure manifests
  are syntactically correct and, generally speaking,
  function as expected.
• But….Things aren’t
  always as they seem!
‘After’ Tests
• To ensure our environment really is clean we
  must test in user space land.
• Probably many solutions out there
  – We wrote our own
User Space Testing
• Post Provision Checks (PPC) is a framework for
  testing stuff in user space.
  – Originally written as health checks for testing a
    machine immediately after provisioning it and as
    verification testing after maintenance.
  – Quickly grew to an extensible framework capable of
    integrating with Nagios/Icinga and Teamcity, or
    running as stand alone application.
  – Contains libraries of functions for commonly used
    tasks.
  – Easily customize runtime options with profiles
  – Can be extended with plugins
PPC functionality
puppet_ust plugin
• The puppet user space tests plugin was
  written to extend the framework for
  specifically testing puppet applied
  configurations.
  – Auto detects the puppet classes that are applied
    to a given box
  – Determines the parameters defined for the host
    being tested and utilizes them within tests.
     • Also uses facter facts in the same manner
That’s great, but puppet reports will tell you
what worked and what didn’t…
misses
• You wouldn’t believe the things we have seen go wrong
  with applying configurations (that the puppet logs/reports
  didn’t indicate).
   – PPC tests can be written to catch all of these!
       • Mistakes only happen once
       • Winning!
   – A few examples
       • Version of package specified not what got installed – not puppet’s
         fault, but still a problem.
       • Package got installed, but some key files were zero length
       • Drivers upgraded, but not properly loaded
       • Machines joined to wrong domain
       • Services running (per status) but not functioning properly
       • Yum repos configured but not functioning
       • Sysctl settings still not loaded after sysctl –p
PPC sample output
• Stand Alone mode
PPC Sample Output
• Team City Mode
Automating PPC Tests
• Team City + multiplexed SSH (for parallelization)
• Each version control commit results in a puppet
  run + user space test run in our dev and qa
  environments.
• Over 20,000 tests performed in ~20 minutes
   – This means every 20 minutes new, fully tested
     changes are pushed from the development
     environment to QA
   – Each stage of the pipeline is tested in a similar fashion
Automated Puppet Testing - PuppetCamp Chicago '12 - Scott Nottingham

More Related Content

What's hot

Puppet Camp Seattle 2014: Puppet: Cloud Infrastructure as Code
Puppet Camp Seattle 2014: Puppet: Cloud Infrastructure as CodePuppet Camp Seattle 2014: Puppet: Cloud Infrastructure as Code
Puppet Camp Seattle 2014: Puppet: Cloud Infrastructure as Code
Puppet
 

What's hot (19)

BASH Variables Part 1: Basic Interpolation
BASH Variables Part 1: Basic InterpolationBASH Variables Part 1: Basic Interpolation
BASH Variables Part 1: Basic Interpolation
 
Puppet for Sys Admins
Puppet for Sys AdminsPuppet for Sys Admins
Puppet for Sys Admins
 
Django REST Framework
Django REST FrameworkDjango REST Framework
Django REST Framework
 
Keeping objects healthy with Object::Exercise.
Keeping objects healthy with Object::Exercise.Keeping objects healthy with Object::Exercise.
Keeping objects healthy with Object::Exercise.
 
Anatomy of a reusable module
Anatomy of a reusable moduleAnatomy of a reusable module
Anatomy of a reusable module
 
Object Trampoline: Why having not the object you want is what you need.
Object Trampoline: Why having not the object you want is what you need.Object Trampoline: Why having not the object you want is what you need.
Object Trampoline: Why having not the object you want is what you need.
 
Rest api with Python
Rest api with PythonRest api with Python
Rest api with Python
 
Replacing "exec" with a type and provider: Return manifests to a declarative ...
Replacing "exec" with a type and provider: Return manifests to a declarative ...Replacing "exec" with a type and provider: Return manifests to a declarative ...
Replacing "exec" with a type and provider: Return manifests to a declarative ...
 
The $path to knowledge: What little it take to unit-test Perl.
The $path to knowledge: What little it take to unit-test Perl.The $path to knowledge: What little it take to unit-test Perl.
The $path to knowledge: What little it take to unit-test Perl.
 
Lies, Damn Lies, and Benchmarks
Lies, Damn Lies, and BenchmarksLies, Damn Lies, and Benchmarks
Lies, Damn Lies, and Benchmarks
 
Refactor Dance - Puppet Labs 'Best Practices'
Refactor Dance - Puppet Labs 'Best Practices'Refactor Dance - Puppet Labs 'Best Practices'
Refactor Dance - Puppet Labs 'Best Practices'
 
Metadata-driven Testing
Metadata-driven TestingMetadata-driven Testing
Metadata-driven Testing
 
Hacking ansible
Hacking ansibleHacking ansible
Hacking ansible
 
Findbin libs
Findbin libsFindbin libs
Findbin libs
 
Test-Driven Puppet Development - PuppetConf 2014
Test-Driven Puppet Development - PuppetConf 2014Test-Driven Puppet Development - PuppetConf 2014
Test-Driven Puppet Development - PuppetConf 2014
 
Puppet modules for Fun and Profit
Puppet modules for Fun and ProfitPuppet modules for Fun and Profit
Puppet modules for Fun and Profit
 
Puppet Camp Seattle 2014: Puppet: Cloud Infrastructure as Code
Puppet Camp Seattle 2014: Puppet: Cloud Infrastructure as CodePuppet Camp Seattle 2014: Puppet: Cloud Infrastructure as Code
Puppet Camp Seattle 2014: Puppet: Cloud Infrastructure as Code
 
Oliver hookins puppetcamp2011
Oliver hookins puppetcamp2011Oliver hookins puppetcamp2011
Oliver hookins puppetcamp2011
 
Puppet @ Seat
Puppet @ SeatPuppet @ Seat
Puppet @ Seat
 

Viewers also liked

Ops Meta-Metrics: The Currency You Pay For Change
Ops Meta-Metrics: The Currency You Pay For ChangeOps Meta-Metrics: The Currency You Pay For Change
Ops Meta-Metrics: The Currency You Pay For Change
John Allspaw
 
Final startup grind
Final startup grindFinal startup grind
Final startup grind
Mark Suster
 

Viewers also liked (12)

Ops Meta-Metrics: The Currency You Pay For Change
Ops Meta-Metrics: The Currency You Pay For ChangeOps Meta-Metrics: The Currency You Pay For Change
Ops Meta-Metrics: The Currency You Pay For Change
 
Puppet Camp Sydney Feb 2014 - A Build Engineering Team’s Journey of Infrastru...
Puppet Camp Sydney Feb 2014 - A Build Engineering Team’s Journey of Infrastru...Puppet Camp Sydney Feb 2014 - A Build Engineering Team’s Journey of Infrastru...
Puppet Camp Sydney Feb 2014 - A Build Engineering Team’s Journey of Infrastru...
 
Innovating faster with SBT, Continuous Delivery, and LXC
Innovating faster with SBT, Continuous Delivery, and LXCInnovating faster with SBT, Continuous Delivery, and LXC
Innovating faster with SBT, Continuous Delivery, and LXC
 
CI/CD Pipeline to Deploy and Maintain an OpenStack IaaS Cloud
CI/CD Pipeline to Deploy and Maintain an OpenStack IaaS CloudCI/CD Pipeline to Deploy and Maintain an OpenStack IaaS Cloud
CI/CD Pipeline to Deploy and Maintain an OpenStack IaaS Cloud
 
Considerations for Alert Design
Considerations for Alert DesignConsiderations for Alert Design
Considerations for Alert Design
 
Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls
Velocity EU 2012 Escalating Scenarios: Outage Handling PitfallsVelocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls
Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls
 
Why docker | OSCON 2013
Why docker | OSCON 2013Why docker | OSCON 2013
Why docker | OSCON 2013
 
Dev and Ops Collaboration and Awareness at Etsy and Flickr
Dev and Ops Collaboration and Awareness at Etsy and FlickrDev and Ops Collaboration and Awareness at Etsy and Flickr
Dev and Ops Collaboration and Awareness at Etsy and Flickr
 
Final startup grind
Final startup grindFinal startup grind
Final startup grind
 
Docker and Go: why did we decide to write Docker in Go?
Docker and Go: why did we decide to write Docker in Go?Docker and Go: why did we decide to write Docker in Go?
Docker and Go: why did we decide to write Docker in Go?
 
10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
10+ Deploys Per Day: Dev and Ops Cooperation at Flickr10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
 
Scalability, Availability & Stability Patterns
Scalability, Availability & Stability PatternsScalability, Availability & Stability Patterns
Scalability, Availability & Stability Patterns
 

Similar to Automated Puppet Testing - PuppetCamp Chicago '12 - Scott Nottingham

modern module development - Ken Barber 2012 Edinburgh Puppet Camp
modern module development - Ken Barber 2012 Edinburgh Puppet Campmodern module development - Ken Barber 2012 Edinburgh Puppet Camp
modern module development - Ken Barber 2012 Edinburgh Puppet Camp
Puppet
 
20140406 loa days-tdd-with_puppet_tutorial
20140406 loa days-tdd-with_puppet_tutorial20140406 loa days-tdd-with_puppet_tutorial
20140406 loa days-tdd-with_puppet_tutorial
garrett honeycutt
 
20090514 Introducing Puppet To Sasag
20090514 Introducing Puppet To Sasag20090514 Introducing Puppet To Sasag
20090514 Introducing Puppet To Sasag
garrett honeycutt
 
Testing NodeJS with Mocha, Should, Sinon, and JSCoverage
Testing NodeJS with Mocha, Should, Sinon, and JSCoverageTesting NodeJS with Mocha, Should, Sinon, and JSCoverage
Testing NodeJS with Mocha, Should, Sinon, and JSCoverage
mlilley
 

Similar to Automated Puppet Testing - PuppetCamp Chicago '12 - Scott Nottingham (20)

modern module development - Ken Barber 2012 Edinburgh Puppet Camp
modern module development - Ken Barber 2012 Edinburgh Puppet Campmodern module development - Ken Barber 2012 Edinburgh Puppet Camp
modern module development - Ken Barber 2012 Edinburgh Puppet Camp
 
Learning Puppet basic thing
Learning Puppet basic thing Learning Puppet basic thing
Learning Puppet basic thing
 
Puppet
PuppetPuppet
Puppet
 
20140408 tdd puppetcamp-paris
20140408 tdd puppetcamp-paris20140408 tdd puppetcamp-paris
20140408 tdd puppetcamp-paris
 
Puppet Camp Paris 2014: Test Driven Development
Puppet Camp Paris 2014: Test Driven DevelopmentPuppet Camp Paris 2014: Test Driven Development
Puppet Camp Paris 2014: Test Driven Development
 
Unit testing presentation
Unit testing presentationUnit testing presentation
Unit testing presentation
 
Puppet meetup testing
Puppet meetup testingPuppet meetup testing
Puppet meetup testing
 
Testing for Ops: Going Beyond the Manifest - PuppetConf 2013
Testing for Ops: Going Beyond the Manifest - PuppetConf 2013Testing for Ops: Going Beyond the Manifest - PuppetConf 2013
Testing for Ops: Going Beyond the Manifest - PuppetConf 2013
 
Orchestrated Functional Testing with Puppet-spec and Mspectator
Orchestrated Functional Testing with Puppet-spec and MspectatorOrchestrated Functional Testing with Puppet-spec and Mspectator
Orchestrated Functional Testing with Puppet-spec and Mspectator
 
Orchestrated Functional Testing with Puppet-spec and Mspectator - PuppetConf ...
Orchestrated Functional Testing with Puppet-spec and Mspectator - PuppetConf ...Orchestrated Functional Testing with Puppet-spec and Mspectator - PuppetConf ...
Orchestrated Functional Testing with Puppet-spec and Mspectator - PuppetConf ...
 
20140406 loa days-tdd-with_puppet_tutorial
20140406 loa days-tdd-with_puppet_tutorial20140406 loa days-tdd-with_puppet_tutorial
20140406 loa days-tdd-with_puppet_tutorial
 
Introduction to InSpec and 1.0 release update
Introduction to InSpec and 1.0 release updateIntroduction to InSpec and 1.0 release update
Introduction to InSpec and 1.0 release update
 
Writing and Publishing Puppet Modules - PuppetConf 2014
Writing and Publishing Puppet Modules - PuppetConf 2014Writing and Publishing Puppet Modules - PuppetConf 2014
Writing and Publishing Puppet Modules - PuppetConf 2014
 
Quick tour to front end unit testing using jasmine
Quick tour to front end unit testing using jasmineQuick tour to front end unit testing using jasmine
Quick tour to front end unit testing using jasmine
 
20090514 Introducing Puppet To Sasag
20090514 Introducing Puppet To Sasag20090514 Introducing Puppet To Sasag
20090514 Introducing Puppet To Sasag
 
BuildStuff.LT 2018 InSpec Workshop
BuildStuff.LT 2018 InSpec WorkshopBuildStuff.LT 2018 InSpec Workshop
BuildStuff.LT 2018 InSpec Workshop
 
DevOpsDays InSpec Workshop
DevOpsDays InSpec WorkshopDevOpsDays InSpec Workshop
DevOpsDays InSpec Workshop
 
Logstash
LogstashLogstash
Logstash
 
Model-Driven Testing For Agile Teams
Model-Driven Testing For Agile TeamsModel-Driven Testing For Agile Teams
Model-Driven Testing For Agile Teams
 
Testing NodeJS with Mocha, Should, Sinon, and JSCoverage
Testing NodeJS with Mocha, Should, Sinon, and JSCoverageTesting NodeJS with Mocha, Should, Sinon, and JSCoverage
Testing NodeJS with Mocha, Should, Sinon, and JSCoverage
 

More from Puppet

Puppet camp2021 testing modules and controlrepo
Puppet camp2021 testing modules and controlrepoPuppet camp2021 testing modules and controlrepo
Puppet camp2021 testing modules and controlrepo
Puppet
 
2021 04-15 operational verification (with notes)
2021 04-15 operational verification (with notes)2021 04-15 operational verification (with notes)
2021 04-15 operational verification (with notes)
Puppet
 
Enforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automationEnforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automation
Puppet
 

More from Puppet (20)

Puppet camp2021 testing modules and controlrepo
Puppet camp2021 testing modules and controlrepoPuppet camp2021 testing modules and controlrepo
Puppet camp2021 testing modules and controlrepo
 
Puppetcamp r10kyaml
Puppetcamp r10kyamlPuppetcamp r10kyaml
Puppetcamp r10kyaml
 
2021 04-15 operational verification (with notes)
2021 04-15 operational verification (with notes)2021 04-15 operational verification (with notes)
2021 04-15 operational verification (with notes)
 
Puppet camp vscode
Puppet camp vscodePuppet camp vscode
Puppet camp vscode
 
Modules of the twenties
Modules of the twentiesModules of the twenties
Modules of the twenties
 
Applying Roles and Profiles method to compliance code
Applying Roles and Profiles method to compliance codeApplying Roles and Profiles method to compliance code
Applying Roles and Profiles method to compliance code
 
KGI compliance as-code approach
KGI compliance as-code approachKGI compliance as-code approach
KGI compliance as-code approach
 
Enforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automationEnforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automation
 
Keynote: Puppet camp compliance
Keynote: Puppet camp complianceKeynote: Puppet camp compliance
Keynote: Puppet camp compliance
 
Automating it management with Puppet + ServiceNow
Automating it management with Puppet + ServiceNowAutomating it management with Puppet + ServiceNow
Automating it management with Puppet + ServiceNow
 
Puppet: The best way to harden Windows
Puppet: The best way to harden WindowsPuppet: The best way to harden Windows
Puppet: The best way to harden Windows
 
Simplified Patch Management with Puppet - Oct. 2020
Simplified Patch Management with Puppet - Oct. 2020Simplified Patch Management with Puppet - Oct. 2020
Simplified Patch Management with Puppet - Oct. 2020
 
Accelerating azure adoption with puppet
Accelerating azure adoption with puppetAccelerating azure adoption with puppet
Accelerating azure adoption with puppet
 
Puppet catalog Diff; Raphael Pinson
Puppet catalog Diff; Raphael PinsonPuppet catalog Diff; Raphael Pinson
Puppet catalog Diff; Raphael Pinson
 
ServiceNow and Puppet- better together, Kevin Reeuwijk
ServiceNow and Puppet- better together, Kevin ReeuwijkServiceNow and Puppet- better together, Kevin Reeuwijk
ServiceNow and Puppet- better together, Kevin Reeuwijk
 
Take control of your dev ops dumping ground
Take control of your  dev ops dumping groundTake control of your  dev ops dumping ground
Take control of your dev ops dumping ground
 
100% Puppet Cloud Deployment of Legacy Software
100% Puppet Cloud Deployment of Legacy Software100% Puppet Cloud Deployment of Legacy Software
100% Puppet Cloud Deployment of Legacy Software
 
Puppet User Group
Puppet User GroupPuppet User Group
Puppet User Group
 
Continuous Compliance and DevSecOps
Continuous Compliance and DevSecOpsContinuous Compliance and DevSecOps
Continuous Compliance and DevSecOps
 
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick MaludyThe Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
 

Recently uploaded

Recently uploaded (20)

Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
Your enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jYour enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4j
 
Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream Processing
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The InsideCollecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 

Automated Puppet Testing - PuppetCamp Chicago '12 - Scott Nottingham

  • 1. Automated Puppet Testing “All code is guilty, until proven innocent.” – Anonymous Scott Nottingham July 23rd, 2012
  • 2. Before and After Testing • Before Testing – Unit tests • Cucumber-puppet • Rspec-puppet • After Testing – User Space tests • Did everything that was supposed to get done actually get done?
  • 3. Unit Tests - frameworks • Cucumber-puppet – Tests the outcome of the entire module as opposed to just the manifests – Overly complicated by trying to be simple (imho) • Rspec-puppet – Unit tests for the manifests – Truly simple (and effective)
  • 4. RSpec-puppet • Written by Tim Sharpe (rodjek) • Available on github – https://github.com/rodjek/rspec-puppet/
  • 5. RSpec-puppet • PROs – Simple language and structure – Easy to maintain – Fast to execute – Execution easily automated • CONs – Only tests what you tell it to • Easy to make changes to manifests and forget to update RSpec tests
  • 6. Writing RSpec Tests • Directory Tree module | +-- manifests | +-- files | +-- templates | +-- lib | +-- spec | +-- spec_helper.rb | +-- classes | +-- <class_name>_spec.rb
  • 7. Writing RSpec Tests • spec_helper file – Required for RSpec to know where to find the module path – Resides within: <module_name>/spec/spec_helper.rb – contents: require 'puppet' require 'rubygems' require 'rspec-puppet' RSpec.configure do |c| c.module_path = File.join(File.dirname(__FILE__), '../../') c.manifest = '../../../../manifests/site.pp' end
  • 8. Writing RSpec Tests • Spec file – Contains test logic – Naming convention: • <classname>_<pp_filename>_spec.rb • Ex. nagios_init_spec.rb – Contents • Requires – require ‘spec_helper’ • Tests
  • 9. Writing RSpec Tests • Defining Tests – Tests are created using the ‘describe’ method. – The first test must describe the class defined in your manifest. describe ‘nagios’ do … … end
  • 10. Writing RSpec Tests • Accounting for class dependencies – Some modules define class dependencies – Example: • Class*‘yum’+ -> Class*‘nagios’+ – These are handled in RSpec test with ‘:pre_condition’ let(:pre_condition) , 'class ,”yum":-' -
  • 11. Writing RSpec Tests • Setting parameters used by puppet manifests – Optional unless manifest needs them to compile – Handled with ‘:params’ – Example (nested parameters): let(:params) { {:key1 => 'value1', :key2 => 'value2', :key3 => {'keyA' => 'valA'}, :key4 => {'keyB' => {'keyX' => 'valX'} } } }
  • 12. Writing RSpec Tests • Setting a node name for the tests (optional) – Might use if your manifests has node name dependent logic let(:node) { host.example.com }
  • 13. Writing RSpec Tests • Setting custom fact values – Probably the most commonly used – Example: let(:facts) { {:fact1 => 'val1', :fact2 => 'val2'} }
  • 14. Writing RSpec Tests • Sub-tests – These are the tests for each condition that you want to test for a given manifest. describe 'nagios' do describe 'with valid parameters passed' do .... end describe 'with invalid parameters passed' do .... end end
  • 15. Writing RSpec Tests • Testing the resources • Generally speaking, the tests you will be writing will be to confirm that the class contains some set of resources. This is done with the ‘contain_<resource_type>’ • Each resource will generally have attributes that you will want to test are present. This is done with the ‘.with_<attribute>(‘value’)’ method. • Example: it { should contain_file('foo').with_ensure('present').with_owner('root').with_group('root') }
  • 16. Writing RSpec Tests • Cascading .with_ methods can get ugly! • Cascading should only be used for resources with 1 or 2 attributes defined. • Alternatively, you can use block notation for resources with many attributes it do should contain_file('/var/spool/squid/cache').with( 'require' => 'Package[squid]', 'ensure' => 'directory', 'owner' => 'squid', 'group' => 'squid' ) end
  • 17. Writing RSpec Tests • Sometimes multiple values are assigned to a single attribute • These are handled like so: it do should contain_mount('/var/spool/squid/cache').with( 'require' => ['Logical_volume[lv_cache]', 'Package[xfsprogs]' ], .... .... ) end
  • 18. Writing RSpec Tests • Templates – If your file content is generated by a template, rspec can test for this too. Here’s how: • Test that a template generates any content it 'should generate valid content for squid.conf' do content = catalogue.resource('file', '/etc/squid/squid.conf').send(:paramters)[:content] content.should_not be_empty End • Test that a template generates specific content it 'should generate valid content for squid.conf' do content = catalogue.resource('file', '/etc/squid/squid.conf').send(:paramters)[:content] content.should match('some_string') end
  • 19. Writing RSpec Tests • Test for an expected error • Sometimes, you want to test that an error is raised. – For example, if not passing a parameter when one is needed, or passing an invalid parameter. – Example test it { expect { should contain_class('nagios::server') }.to raise_error(Puppet::Error) }
  • 20. require 'spec_helper' describe 'sendmail' do let(:params) { {:mail_server => 'foobarbaz'} } it { should contain_package('sendmail').with_ensure('installed') } it do should contain_file('/etc/mail/sendmail.cf').with( 'require' => 'Package[sendmail]', 'notify' => 'Exec[makemap]' ) end it 'should generate /etc/mail/sendmail.cf from a template' do content = catalogue.resource('file', '/etc/mail/sendmail.cf').send(:parameters)[:content] content.should match('foobarbaz') end it 'should generate /etc/mail/submit.cf from a template' do content = catalogue.resource('file', '/etc/mail/submit.cf').send(:parameters)[:content] content.should match('foobarbaz') end it do should contain_file('/etc/mail/submit.cf').with( 'require' => 'Package[sendmail]', 'notify' => 'Exec[makemap]' ) end it { should contain_file('/etc/sysconfig/sendmail').with_source('puppet:///sendmail/sendmail') } it do should contain_service('sendmail').with( 'require' => 'Package[sendmail]', 'enable' => 'true', 'ensure' => 'running' ) end end
  • 21. Running RSpec Tests • rspec –color –format documentation /etc/puppet/modules/sendmail/spec/classes/sendmail_init_spec.rb sendmail should contain Package[sendmail] with ensure => "installed" should contain File[/etc/mail/sendmail.cf] with notify => "Exec[makemap]" and require => "Package[sendmail]" should generate /etc/mail/sendmail.cf from a template should generate /etc/mail/submit.cf from a template should contain File[/etc/mail/submit.cf] with notify => "Exec[makemap]" and require => "Package[sendmail]" should contain File[/etc/sysconfig/sendmail] with source => "puppet:///sendmail/sendmail" should contain Service[sendmail] with enable => "true", ensure => "running" and require => "Package[sendmail]" Finished in 1.98 seconds 7 examples, 0 failures
  • 22. Running RSpec Tests • Oops! Someone put a comma where they meant to put a semi-colon! Sendmail Failures: 1) sendmail Failure/Error: it { should contain_package('sendmail').with_ensure('installed') } Puppet::Error: Syntax error at '/etc/sysconfig/sendmail'; expected '}' at /etc/puppetmaster/modules/sendmail/spec/../../sendmail/manifests/init.pp:19 on node neodymium.trading.imc.intra # ./sendmail_init_spec.rb:6 2) sendmail Failure/Error: it { should contain_package('sendmail').with_ensure('installed') } Puppet::Error: Syntax error at '/etc/sysconfig/sendmail'; expected '}' at /etc/puppetmaster/modules/sendmail/spec/../../sendmail/manifests/init.pp:19 on node neodymium.trading.imc.intra # ./sendmail_init_spec.rb:6 Finished in 0.19808 seconds 2 examples, 2 failures
  • 23. Automating RSpec Tests • Tests can be run by anything capable of executing a system command • We integrate ours into Team City – Much like Jenkins – Allows for event driven execution of tests • Any time a commit occurs in version control, tests are executed – If tests pass, manifests are wrapped inside an RPM and pushed to puppet masters – If tests fail, we are notified
  • 24. Automating RSpec Tests • RSpec tests do a great job of making sure manifests are syntactically correct and, generally speaking, function as expected. • But….Things aren’t always as they seem!
  • 25. ‘After’ Tests • To ensure our environment really is clean we must test in user space land. • Probably many solutions out there – We wrote our own
  • 26.
  • 27. User Space Testing • Post Provision Checks (PPC) is a framework for testing stuff in user space. – Originally written as health checks for testing a machine immediately after provisioning it and as verification testing after maintenance. – Quickly grew to an extensible framework capable of integrating with Nagios/Icinga and Teamcity, or running as stand alone application. – Contains libraries of functions for commonly used tasks. – Easily customize runtime options with profiles – Can be extended with plugins
  • 29. puppet_ust plugin • The puppet user space tests plugin was written to extend the framework for specifically testing puppet applied configurations. – Auto detects the puppet classes that are applied to a given box – Determines the parameters defined for the host being tested and utilizes them within tests. • Also uses facter facts in the same manner
  • 30. That’s great, but puppet reports will tell you what worked and what didn’t…
  • 31. misses • You wouldn’t believe the things we have seen go wrong with applying configurations (that the puppet logs/reports didn’t indicate). – PPC tests can be written to catch all of these! • Mistakes only happen once • Winning! – A few examples • Version of package specified not what got installed – not puppet’s fault, but still a problem. • Package got installed, but some key files were zero length • Drivers upgraded, but not properly loaded • Machines joined to wrong domain • Services running (per status) but not functioning properly • Yum repos configured but not functioning • Sysctl settings still not loaded after sysctl –p
  • 32. PPC sample output • Stand Alone mode
  • 33. PPC Sample Output • Team City Mode
  • 34. Automating PPC Tests • Team City + multiplexed SSH (for parallelization) • Each version control commit results in a puppet run + user space test run in our dev and qa environments. • Over 20,000 tests performed in ~20 minutes – This means every 20 minutes new, fully tested changes are pushed from the development environment to QA – Each stage of the pipeline is tested in a similar fashion