Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Testing servers like software

875 views

Published on

It's easy enough to test the correctness of the infracode we write with unit-tests and parsers, but testing it does what it's supposed to do in the enviornment itself is a little more challenging. In this talk, I'm going to talk about some of the tools and approaches to use to test your configuration automation tool of choice.

Published in: Technology

Testing servers like software

  1. 1. TESTING SERVERS LIKE SOFTWARE
  2. 2. ME DEVELOPER TURNED ops guy WORKING AT KAINOS, CONTRACTING ON GOVERNMENT PROJECTS
  3. 3. PREVIOUSLY ON THE IER PROJECT NOW LIVE! HTTPS://WWW.GOV.UK/ REGISTER-TO-VOTE
  4. 4. NOW ON THE DEFRA CAPD PROJECT
  5. 5. FOREWARNING ▸ Ruby and Puppet Biased ▸ They're the tools I use the most! ▸ But most tools mentioned are system and tool agnostic
  6. 6. SO YOU'VE MADE A CONFIGURATION CODE CHANGE...
  7. 7. THE UNIT TESTS PASS...
  8. 8. IT'S BEEN CODE REVIEWED
  9. 9. SO YOU PUSH TO PRODUCTION!
  10. 10. IT DOESNT WORK...
  11. 11. WHAT'S THE MISSING STEP?
  12. 12. ACCEPTANCE TESTING: Serverspec
  13. 13. "Serverspec tests your servers'actual state through SSH access"
  14. 14. CHECK YOUR SERVER ▸ Is $package installed? ▸ Does file contain $foo? ▸ Does the firewall have the correct rules? ▸ Is service running? ▸ etc...
  15. 15. RESOURCE TYPES cgroup, command, cron, default_gateway, file, group, host, iis_app_pool, iis_website, interface, ipfilter, ipnat, iptables, kernel_module, linux_kernel_parameter, lxc, mail_alias, package, php_config, port, ppa, process, routing_table, selinux, service, user, windows_feature, windows_registry_key, yumrepo, zfsfs
  16. 16. MOST ARE FAIRLY SELF EXPLANATORY
  17. 17. return_stdout describe command('cat /etc/resolv.conf') do it { should return_stdout /8.8.8.8/ } end content describe file('/etc/httpd/conf/httpd.conf') do its(:content) { should match /ServerName www.example.jp/ } end
  18. 18. MY BREAD AND BUTTER
  19. 19. BE_RESOLVABLE describe host('serverspec.org') do it { should be_resolvable } end describe host('serverspec.org') do it { should be_resolvable.by('hosts') } end describe host('serverspec.org') do it { should be_resolvable.by('dns') } end
  20. 20. BE_REACHABLE describe host('target.example.jp') do # ping it { should be_reachable } # tcp port 22 it { should be_reachable.with( :port => 22 ) } # set protocol explicitly it { should be_reachable.with( :port => 22, :proto => 'tcp' ) } # udp port 53 it { should be_reachable.with( :port => 53, :proto => 'udp' ) } # timeout setting (default is 5 seconds) it { should be_reachable.with( :port => 22, :proto => 'tcp', :timeout => 1 ) } end
  21. 21. FULL SPECS FOR A WEB SERVER require 'spec_helper' describe package('apache2') do it { should be_installed } end describe service('apache2') do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end
  22. 22. EXAMPLE:HTTPS://GITHUB.COM/JVOORHIS/ VAGRANT-SERVERSPEC
  23. 23. LIVE DEMO TIME: SERVERSPEC!
  24. 24. OUR WORKFLOW:CHANGE IN PUPPET MADE => TESTED IN VAGRANT INSTANCE CODE REVIEWED, PASSES CI AND MERGED PUSHED TO INTEGRATION ENVIRONMENT SERVERSPEC TESTS RUN ON INTEGRATION ENVIRONMENT ANY ISSUES: FIX OR IF BIG ENOUGH, REVERT CONTINUES DOWN THE PIPELINE TO PRODUCTION
  25. 25. SERVERSPEC: PRETTY NEAT BUT THERE'S ALSO LANGUAGE SPECIFIC TOOLS!
  26. 26. TEST KITCHEN
  27. 27. ▸ Uses serverspec as it's core ▸ It's basically helper wrappers to install chef ▸ And run the cookbooks given ▸ Used for cookbook acceptance testing
  28. 28. EXAMPLE:HTTPS://GITHUB.COM/OPSCODE- COOKBOOKS/APT/
  29. 29. LIVE DEMO TIME: TEST-KITCHEN!
  30. 30. BEAKER
  31. 31. ▸ Again: serverspec as it's core ▸ But specs customised with Puppet references ▸ Specific rspec grammer around running manifests etc.
  32. 32. EXAMPLE:HTTPS://GITHUB.COM/PETEMS/PUPPET- SWAP_FILE
  33. 33. require 'spec_helper_acceptance' describe 'swap_file class', :unless => UNSUPPORTED_PLATFORMS.include?(fact('osfamily')) do context 'swap_file' do context 'ensure => present' do it 'should work with no errors' do pp = <<-EOS class { 'swap_file': } EOS # Run it twice and test for idempotency expect(apply_manifest(pp).exit_code).to_not eq(1) expect(apply_manifest(pp).exit_code).to eq(0) end
  34. 34. context 'custom parameters' do it 'should work with no errors' do pp = <<-EOS class { 'swap_file': swapfile => '/tmp/swapfile', swapfilesize => '5 MB', } EOS it 'should contain the given swapfile' do shell('/sbin/swapon -s | grep /tmp/swapfile', :acceptable_exit_codes => [0]) shell('/sbin/swapon -s | grep 5116', :acceptable_exit_codes => [0]) end end end end
  35. 35. I KNOW WHAT YOU MIGHT BE THINKING...
  36. 36. Wait, whats the difference between this and monitoring?
  37. 37. MONITORING KEEP THE LIGHTS ON ▸ "Fix me now!" issues ▸ Dynamic changes - Things crashing, hard drives are full
  38. 38. ACCEPTANCE "MY CHANGES DIDN'T BREAK ANYTHING" ▸ Smoke tests ▸ Human readable ▸ One-off for edge-cases ▸ Auditing
  39. 39. HOWEVER
  40. 40. USING SERVERSPEC AS MONITORING IS POSSIBLE! AND HAS SOME PRETTY SWEET BENEFITS HTTP://WWW.SLIDESHARE.NET/M_RICHARDSON/SERVERSPEC-AND-SENSU-TESTING-AND-MONITORING- COLLIDE
  41. 41. Q&A
  42. 42. LINKShttp://serverspec.org/ http://vincent.bernat.im/en/blog/2014-serverspec-test- infrastructure.html https://github.com/serverspec/serverspec http://www.debian-administration.org/article/703/ A_brief_introduction_to_server-testing_with_serverspec "ChefConf 2014: Gosuke Miyashita, "Serverspec: The Simplest Server Testing Tool Ever" - http://www.youtube.com/watch? v=6GvlHImeloo*

×