This document discusses using Puppet-spec and Mspectator to orchestrate functional testing of Puppet configurations. Puppet-spec allows running unit and integration tests as part of Puppet runs, while Mspectator provides RSpec matchers to run functional tests across nodes using MCollective. The tests validate resources, packages, files and more, failing runs when tests don't pass to ensure configurations meet standards.

1. ORCHESTRATED FUNCTIONAL
TESTING WITH PUPPET-SPEC
AND MSPECTATOR
RAPHAËL PINSON

2. Who am I?
Raphaël Pinson (@raphink)
■ Infrastructure Developer & Trainer @ Camptocamp
■ Augeas & Augeasproviders developer
■ Various contributions to Puppet & ecosystem
www.camptocamp.com / 2/32

3. Monitoring vs. Functional Tests
Complementary or redundant?
www.camptocamp.com / 3/32

4. Conformity Tests
■ Check if machines comply to standards
■ Avoid permanent heavy monitoring checks
■ Tests must be inter-dependent
■ Focus on getting sysadmins to fix one thing at a time to converge
toward standards
www.camptocamp.com / 4/32

5. Treetester
■ Back in 2008
■ Written in Perl
■ Orchestrate conformity tests on a 4k+ server fleet
www.camptocamp.com / 5/32

6. Treetester: modules output
■ For all hosts/modules
■ Number of hosts filtered per module
■ Modules dependency tree
■ Colors by priority
www.camptocamp.com / 6/32

7. Treetester: host output
■ For each host
■ Failed steps in the module tree
■ Green: OK, Red: KO, Purple: Ignored
www.camptocamp.com / 7/32

8. Treetester architecture
■ All data in a database (MySQL)
■ Tests scripts output YAML
■ Tests scripts can be local (hosts as STDIN) or remote (ssh or http)
■ Tests are inter-dependent
■ Generate filtered data as a tree
■ Generate graphs (graphviz)
www.camptocamp.com / 8/32

9. Treetester filters
■ For each test/module
■ Based on data in MySQL (joins and additional SQL conditions)
■ Allows to link tests to each other
■ Like multiple sieves
www.camptocamp.com / 9/32

10. Treetester: future?
■ Not open-sourced :'-(
■ Too monolithic/not flexible enough
■ Heavily linked to specific architecture
■ Needed a rewrite
www.camptocamp.com / 10/32

11. Adding specs to Puppet runs
■ Testing the catalog before it gets applied
■ Testing the node after the catalog is applied
Enter the Puppet-spec module
www.camptocamp.com / 11/32

12. Rspec-puppet
■ http://rspec-puppet.com
■ Now the standard to unit test Puppet manifests
■ Generates catalogs in clean environments
■ Asserts catalogs for resources/classes
require 'spec_helper'
describe 'logrotate::rule' do
let(:title) { 'nginx' }
it { should compile.with_all_deps }
it { should contain_class('logrotate::setup') }
end
www.camptocamp.com / 12/32

13. Puppet-spec
■ Runs tests from within Puppet runs
■ Test catalogs using rspec-puppet
■ Test hosts using serverspec
www.camptocamp.com / 13/32

14. Puppet-spec: Unit testing
■ Catalog exposed by PuppetSpec::Catalog.instance.catalog
■ Uses rspec-puppet matchers
■ Asserts real catalogs
■ Runs on the master or agent side (as catalog indirection terminii)
describe 'puppet' do
subject { PuppetSpec::Catalog.instance.catalog }
it { should contain_package('puppet') }
it { should contain_package('ppet') }
it { should include_class('puppet') }
it { should include_class('puppet::client::base') }
end
www.camptocamp.com / 14/32

15. Puppet-spec: Unit tests output
# puppet agent -t
info: Retrieving plugin
err: Could not retrieve catalog from remote server: Unit tests failed:
F..
Failures:
1) package
Failure/Error: it { should contain_package('augeas') }
expected that the catalogue would contain Package[augeas]
# /var/lib/puppet/lib/spec/class/augeas/package_spec.rb:3
# /var/lib/puppet/lib/puppet/indirector/catalog/rest_spec.rb:31:in `find'
Finished in 0.00092 seconds
3 examples, 1 failure
Failed examples:
rspec /var/lib/puppet/lib/spec/class/augeas/package_spec.rb:3 # package
info: Not using expired catalog for foo.example.com from cache; expired at Tue Apr 02 17:40:21 +0200 2013
notice: Using cached catalog
www.camptocamp.com / 15/32

16. Puppet-spec: Deploying unit tests
■ On the master side:
○ Tests are located in the spec/catalog/class directory of the
environment
○ Only the directories named after classes declared in the catalog
are tested
■ On the agent side:
○ Deploy tests using pluginsync
○ Tests are located in the lib/spec/catalog/class directory of each
module
○ Only the directories named after classes declared in the catalog
are tested
www.camptocamp.com / 16/32

17. Puppet-spec: Unit tests limits
■ When to apply the tests (currently based on class names)
■ Tests on master, or need to deploy all tests with pluginsync
■ Redundant with existing unit tests, or additional security?
www.camptocamp.com / 17/32

18. Puppet-spec: Setting up Unit testing
■ Tests achieved from catalog indirection terminii
■ Plugins (terminii) deployed with pluginsync
■ Setup done in routes.yaml:
agent:
catalog:
# Either on the agent side
terminus: rest_spec
cache: yaml
master:
catalog:
# Or on the master side
terminus: compiler_spec
www.camptocamp.com / 18/32

19. Serverspec
■ http://serverspec.org
■ Provides RSpec matchers for local functional tests (packages,
users, services, ports, etc.)
■ Independant from configuration management tools
require 'spec_helper'
describe service('httpd') do
it { should be_enabled }
it { should be_running }
end
describe port(80) do
it { should be_listening }
end
describe file('/etc/httpd/conf/httpd.conf') do
it { should be_file }
its(:content) { should match /ServerName www.example.jp/ }
end
www.camptocamp.com / 19/32

20. Serverspec backends
Allows to use various means of launching tests:
■ SSH (default)
■ Exec
■ Puppet (RAL, removed from core)
$ serverspec-init
Select OS type:
1) UN*X
2) Windows
Select number: 1
Select a backend type:
1) SSH
2) Exec (local)
Select number: 1
www.camptocamp.com / 20/32

21. Puppet-spec: Functional testing
■ Uses serverspec/specinfra matchers
■ Tests the machine state (not the catalog)
require 'spec_helper'
describe service('httpd') do
it { should be_enabled }
it { should be_running }
end
describe port(80) do
it { should be_listening }
end
describe file('/etc/httpd/conf/httpd.conf') do
it { should be_file }
its(:content) { should match /ServerName www.example.jp/ }
end
www.camptocamp.com / 21/32

22. Puppet-spec: Function tests output
# puppet agent -t
info: Retrieving plugin
info: Caching catalog for foo.example.com
info: Applying configuration version 'raphink/a2c8e0f [+]'
... Applying changes ...
notice: Finished catalog run in 59.19 seconds
err: Could not send report: Unit tests failed:
FF
Failures:
1) augeas
Failure/Error: it { should be_installed }
expected "augeas" to be installed
# /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:2
# /var/lib/puppet/lib/puppet/indirector/report/rest_spec.rb:45:in `save'
2) /usr/share/augeas/lenses/dist
Failure/Error: it { should be_file }
expected "/usr/share/augeas/lenses/dist" to be file
# /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:6
# /var/lib/puppet/lib/puppet/indirector/report/rest_spec.rb:45:in `save'
Finished in 0.06033 seconds
2 examples, 2 failures
Failed examples:
rspec /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:2 # augeas
rspec /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:6 # /usr/share/augeas/lenses/www.camptocamp.com / 22/32

23. Puppet-spec: Deploying functional
tests
■ Tests are run after catalog application
■ Tests can be distributed via pluginsync (in the spec/server/class)
directory of each module
■ Tests can be distributed with file Puppet resources, optionally
using the spec::serverspec defined resource type
www.camptocamp.com / 23/32

24. Puppet-spec: MCollective agent
■ Communicates with distant nodes
■ Sends action and values to specinfra check commands
■ Does not implement serverspec syntax
■ Returns true/false
■ Uses MCollective as transport (instead of SSH)
Examples:
$ mco rpc spec check action=user values=rpinson
$ mco rpc spec check action=file values=/etc/passwd
$ mco rpc spec check action=resolvable values=google.fr,A
$ mco rpc spec check action=listening values=80
$ mco rpc spec check action=process values=mcollectived
$ mco rpc spec check action=file_contain values=/etc/passwd,rpinson
www.camptocamp.com / 24/32

25. Mspectator
■ https://github.com/raphink/mspectator
■ RSpec matchers
■ Calls MCollective to achieve tests
■ Uses MCollective spec agent (among others)
www.camptocamp.com / 25/32

26. Mspectator architecture
■ Client runs RSpec
■ RSpec calls MCollective
■ MCollective calls distant spec agent
■ spec agent calls specinfra backend
www.camptocamp.com / 26/32

27. Mspectator syntax
Own matchers, mapping to specinfra backend methods:
require 'mspectator'
describe 'apache' do
it { should find_nodes(100).or_less } # Counts discovered nodes
it { should pass_puppet_spec } # Runs the `spec` agent
it { should have_certificate.signed } # Uses the `puppetca` agent
context 'when on Debian',
:facts => { :operatingsystem => 'Debian' } do # Filter by facts
it { should find_nodes(5).with_agent('spec') }
it { should have_package('apache2.2-common') }
it { should_not have_package('httpd') }
it { should have_service('apache2').with(
:ensure => 'running'
) }
it { should have_file('/etc/apache2/apache2.conf') }
it { should have_directory('/etc/apache2/conf.d') }
it { should have_user('www-data') }
end
context 'when using SSL', :classes => ['apache::ssl'] do # Filter by classes
it { should find_nodes(50).or_more }
it { should have_package('ca-certificates') }
end
end
www.camptocamp.com / 27/32

28. Mspectator output
$ rake spec SPEC=apache_spec.rb
/home/rpinson/.rvm/rubies/ruby-1.8.7-p371/bin/ruby -S rspec apache_spec.rb
apache
should find nodes 100
should pass puppet spec (FAILED - 1)
should have certificate
when on Debian
should find nodes 5 (FAILED - 2)
...
when using SSL
should find nodes 50 (FAILED - 3)
No request sent, we did not discover any nodes. should have package "ca-certificates"
Failures:
1) apache
Failure/Error: it { should pass_puppet_spec }
expected that all hosts would pass tests, the following didn't:
soekris01.wrk.cby.camptocamp.com:
soekris02.wrk.cby.camptocamp.com:
# ./apache_spec.rb:5
...
www.camptocamp.com / 28/32

29. Mspectator demo
www.camptocamp.com / 29/32

30. Contribute
On GitHub:
■ puppet-spec: https://github.com/raphink/puppet-spec
■ mspectator: https://github.com/raphink/mspectator
www.camptocamp.com / 30/32

31. Thank you!
■ raphael.pinson@camptocamp.com
■ @raphink on Twitter/Github
■ raphink on Freenode
■ Slides: slideshare.net/raphink
www.camptocamp.com / 31/32