Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Orchestrated Functional Testing with Puppet-spec and Mspectator - PuppetConf 2014

2,398 views

Published on

Orchestrated Functional Testing with Puppet-spec and Mspectator - Raphaël Pinson, Camptocamp

Published in: Technology
  • Be the first to comment

Orchestrated Functional Testing with Puppet-spec and Mspectator - PuppetConf 2014

  1. 1. ORCHESTRATED FUNCTIONAL TESTING WITH PUPPET-SPEC AND MSPECTATOR RAPHAËL PINSON
  2. 2. Who am I? Raphaël Pinson (@raphink) ■ Infrastructure Developer & Trainer @ Camptocamp ■ Augeas & Augeasproviders developer ■ Various contributions to Puppet & ecosystem www.camptocamp.com / 2/32
  3. 3. Monitoring vs. Functional Tests Complementary or redundant? www.camptocamp.com / 3/32
  4. 4. Conformity Tests ■ Check if machines comply to standards ■ Avoid permanent heavy monitoring checks ■ Tests must be inter-dependent ■ Focus on getting sysadmins to fix one thing at a time to converge toward standards www.camptocamp.com / 4/32
  5. 5. Treetester ■ Back in 2008 ■ Written in Perl ■ Orchestrate conformity tests on a 4k+ server fleet www.camptocamp.com / 5/32
  6. 6. Treetester: modules output ■ For all hosts/modules ■ Number of hosts filtered per module ■ Modules dependency tree ■ Colors by priority www.camptocamp.com / 6/32
  7. 7. Treetester: host output ■ For each host ■ Failed steps in the module tree ■ Green: OK, Red: KO, Purple: Ignored www.camptocamp.com / 7/32
  8. 8. Treetester architecture ■ All data in a database (MySQL) ■ Tests scripts output YAML ■ Tests scripts can be local (hosts as STDIN) or remote (ssh or http) ■ Tests are inter-dependent ■ Generate filtered data as a tree ■ Generate graphs (graphviz) www.camptocamp.com / 8/32
  9. 9. Treetester filters ■ For each test/module ■ Based on data in MySQL (joins and additional SQL conditions) ■ Allows to link tests to each other ■ Like multiple sieves www.camptocamp.com / 9/32
  10. 10. Treetester: future? ■ Not open-sourced :'-( ■ Too monolithic/not flexible enough ■ Heavily linked to specific architecture ■ Needed a rewrite www.camptocamp.com / 10/32
  11. 11. Adding specs to Puppet runs ■ Testing the catalog before it gets applied ■ Testing the node after the catalog is applied Enter the Puppet-spec module www.camptocamp.com / 11/32
  12. 12. Rspec-puppet ■ http://rspec-puppet.com ■ Now the standard to unit test Puppet manifests ■ Generates catalogs in clean environments ■ Asserts catalogs for resources/classes require 'spec_helper' describe 'logrotate::rule' do let(:title) { 'nginx' } it { should compile.with_all_deps } it { should contain_class('logrotate::setup') } end www.camptocamp.com / 12/32
  13. 13. Puppet-spec ■ Runs tests from within Puppet runs ■ Test catalogs using rspec-puppet ■ Test hosts using serverspec www.camptocamp.com / 13/32
  14. 14. Puppet-spec: Unit testing ■ Catalog exposed by PuppetSpec::Catalog.instance.catalog ■ Uses rspec-puppet matchers ■ Asserts real catalogs ■ Runs on the master or agent side (as catalog indirection terminii) describe 'puppet' do subject { PuppetSpec::Catalog.instance.catalog } it { should contain_package('puppet') } it { should contain_package('ppet') } it { should include_class('puppet') } it { should include_class('puppet::client::base') } end www.camptocamp.com / 14/32
  15. 15. Puppet-spec: Unit tests output # puppet agent -t info: Retrieving plugin err: Could not retrieve catalog from remote server: Unit tests failed: F.. Failures: 1) package Failure/Error: it { should contain_package('augeas') } expected that the catalogue would contain Package[augeas] # /var/lib/puppet/lib/spec/class/augeas/package_spec.rb:3 # /var/lib/puppet/lib/puppet/indirector/catalog/rest_spec.rb:31:in `find' Finished in 0.00092 seconds 3 examples, 1 failure Failed examples: rspec /var/lib/puppet/lib/spec/class/augeas/package_spec.rb:3 # package info: Not using expired catalog for foo.example.com from cache; expired at Tue Apr 02 17:40:21 +0200 2013 notice: Using cached catalog www.camptocamp.com / 15/32
  16. 16. Puppet-spec: Deploying unit tests ■ On the master side: ○ Tests are located in the spec/catalog/class directory of the environment ○ Only the directories named after classes declared in the catalog are tested ■ On the agent side: ○ Deploy tests using pluginsync ○ Tests are located in the lib/spec/catalog/class directory of each module ○ Only the directories named after classes declared in the catalog are tested www.camptocamp.com / 16/32
  17. 17. Puppet-spec: Unit tests limits ■ When to apply the tests (currently based on class names) ■ Tests on master, or need to deploy all tests with pluginsync ■ Redundant with existing unit tests, or additional security? www.camptocamp.com / 17/32
  18. 18. Puppet-spec: Setting up Unit testing ■ Tests achieved from catalog indirection terminii ■ Plugins (terminii) deployed with pluginsync ■ Setup done in routes.yaml: agent: catalog: # Either on the agent side terminus: rest_spec cache: yaml master: catalog: # Or on the master side terminus: compiler_spec www.camptocamp.com / 18/32
  19. 19. Serverspec ■ http://serverspec.org ■ Provides RSpec matchers for local functional tests (packages, users, services, ports, etc.) ■ Independant from configuration management tools require 'spec_helper' describe service('httpd') do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end describe file('/etc/httpd/conf/httpd.conf') do it { should be_file } its(:content) { should match /ServerName www.example.jp/ } end www.camptocamp.com / 19/32
  20. 20. Serverspec backends Allows to use various means of launching tests: ■ SSH (default) ■ Exec ■ Puppet (RAL, removed from core) $ serverspec-init Select OS type: 1) UN*X 2) Windows Select number: 1 Select a backend type: 1) SSH 2) Exec (local) Select number: 1 www.camptocamp.com / 20/32
  21. 21. Puppet-spec: Functional testing ■ Uses serverspec/specinfra matchers ■ Tests the machine state (not the catalog) require 'spec_helper' describe service('httpd') do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end describe file('/etc/httpd/conf/httpd.conf') do it { should be_file } its(:content) { should match /ServerName www.example.jp/ } end www.camptocamp.com / 21/32
  22. 22. Puppet-spec: Function tests output # puppet agent -t info: Retrieving plugin info: Caching catalog for foo.example.com info: Applying configuration version 'raphink/a2c8e0f [+]' ... Applying changes ... notice: Finished catalog run in 59.19 seconds err: Could not send report: Unit tests failed: FF Failures: 1) augeas Failure/Error: it { should be_installed } expected "augeas" to be installed # /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:2 # /var/lib/puppet/lib/puppet/indirector/report/rest_spec.rb:45:in `save' 2) /usr/share/augeas/lenses/dist Failure/Error: it { should be_file } expected "/usr/share/augeas/lenses/dist" to be file # /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:6 # /var/lib/puppet/lib/puppet/indirector/report/rest_spec.rb:45:in `save' Finished in 0.06033 seconds 2 examples, 2 failures Failed examples: rspec /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:2 # augeas rspec /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:6 # /usr/share/augeas/lenses/www.camptocamp.com / 22/32
  23. 23. Puppet-spec: Deploying functional tests ■ Tests are run after catalog application ■ Tests can be distributed via pluginsync (in the spec/server/class) directory of each module ■ Tests can be distributed with file Puppet resources, optionally using the spec::serverspec defined resource type www.camptocamp.com / 23/32
  24. 24. Puppet-spec: MCollective agent ■ Communicates with distant nodes ■ Sends action and values to specinfra check commands ■ Does not implement serverspec syntax ■ Returns true/false ■ Uses MCollective as transport (instead of SSH) Examples: $ mco rpc spec check action=user values=rpinson $ mco rpc spec check action=file values=/etc/passwd $ mco rpc spec check action=resolvable values=google.fr,A $ mco rpc spec check action=listening values=80 $ mco rpc spec check action=process values=mcollectived $ mco rpc spec check action=file_contain values=/etc/passwd,rpinson www.camptocamp.com / 24/32
  25. 25. Mspectator ■ https://github.com/raphink/mspectator ■ RSpec matchers ■ Calls MCollective to achieve tests ■ Uses MCollective spec agent (among others) www.camptocamp.com / 25/32
  26. 26. Mspectator architecture ■ Client runs RSpec ■ RSpec calls MCollective ■ MCollective calls distant spec agent ■ spec agent calls specinfra backend www.camptocamp.com / 26/32
  27. 27. Mspectator syntax Own matchers, mapping to specinfra backend methods: require 'mspectator' describe 'apache' do it { should find_nodes(100).or_less } # Counts discovered nodes it { should pass_puppet_spec } # Runs the `spec` agent it { should have_certificate.signed } # Uses the `puppetca` agent context 'when on Debian', :facts => { :operatingsystem => 'Debian' } do # Filter by facts it { should find_nodes(5).with_agent('spec') } it { should have_package('apache2.2-common') } it { should_not have_package('httpd') } it { should have_service('apache2').with( :ensure => 'running' ) } it { should have_file('/etc/apache2/apache2.conf') } it { should have_directory('/etc/apache2/conf.d') } it { should have_user('www-data') } end context 'when using SSL', :classes => ['apache::ssl'] do # Filter by classes it { should find_nodes(50).or_more } it { should have_package('ca-certificates') } end end www.camptocamp.com / 27/32
  28. 28. Mspectator output $ rake spec SPEC=apache_spec.rb /home/rpinson/.rvm/rubies/ruby-1.8.7-p371/bin/ruby -S rspec apache_spec.rb apache should find nodes 100 should pass puppet spec (FAILED - 1) should have certificate when on Debian should find nodes 5 (FAILED - 2) ... when using SSL should find nodes 50 (FAILED - 3) No request sent, we did not discover any nodes. should have package "ca-certificates" Failures: 1) apache Failure/Error: it { should pass_puppet_spec } expected that all hosts would pass tests, the following didn't: soekris01.wrk.cby.camptocamp.com: soekris02.wrk.cby.camptocamp.com: # ./apache_spec.rb:5 ... www.camptocamp.com / 28/32
  29. 29. Mspectator demo www.camptocamp.com / 29/32
  30. 30. Contribute On GitHub: ■ puppet-spec: https://github.com/raphink/puppet-spec ■ mspectator: https://github.com/raphink/mspectator www.camptocamp.com / 30/32
  31. 31. Thank you! ■ raphael.pinson@camptocamp.com ■ @raphink on Twitter/Github ■ raphink on Freenode ■ Slides: slideshare.net/raphink www.camptocamp.com / 31/32

×