Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this presentation? Why not share!

Zombie chaser 0.1

on

  • 1,023 views

Zombie-chaser 0.1: Going beyond skin deep. A description of zombie-chaser and mutation testing.

Zombie-chaser 0.1: Going beyond skin deep. A description of zombie-chaser and mutation testing.

Statistics

Views

Total Views
1,023
Views on SlideShare
1,023
Embed Views
0

Actions

Likes
0
Downloads
1
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Zombie chaser 0.1 Zombie chaser 0.1 Presentation Transcript

  • Zombie-chaser 0.1 Going beyond skin deep Andrew Grimm University of New South Wales
  • Overview
    • Gratuitous violence and copyright violation
    • Unit testing
    • Mutation testing
    • Heckle
    • Chaser
    • Zombie-chaser
    • Gratuitous violence and copyright violation
    • Gratuitous violence and copyright violation
    • See above
  • We don't test code
    • I get a specification from my boss
    • I spend a couple of weeks writing on my typewriter
    • Fax it to the client
    • They scan it in
    • It runs first time
    • And does what they wanted
    • This DOESN'T happen
  • We all test code somehow
    • SFTP code.php, make a few edits, SFTP it back to the production server, view it in your browser
      • This DOES happen
    • svn update, make a few changes, try it on a development web server, don't spot anything wrong, push to production
      • Better
    • git clone, add new test, add new code, check all tests pass, push to production
      • Yay!!
  • Automated testing
    • Term “Unit testing” sometimes misused
    • Technically means testing a small portion of your code
    • Nowadays used to mean automated testing
    • Sometimes automated testing is the lazier option than manual testing even in the short-term
  • Automated testing = exomemory
    • Your code is thousands of lines long
      • You can't memorize your code
    • Requirements change from the original
      • You can't write once, try it out, and never touch it again
    • You can't email past-self
      • The code itself needs to know what's right and wrong
  • An example of a test
    • Given I am a fake Canadian motorcade at the 2007 APEC summit
    • When I supply obviously bogus credentials to security
    • Then I should not be let in to the green zone
  • Mutation testing
    • Evaluates thoroughness of testing
    • If your code is modified, it ought to fail some of its tests
  • Example of a mutation
    • Modifying an APEC credential verification method, such that it regards an invalid pass as valid and/or vice versa
  • Heckle
    • Main mutation tester in ruby
    • Name derived from Jester (Java mutation tester)
    • Uses ParseTree and ruby2ruby
    • Mutates literals (eg strings) and branches (eg swaps “if”s with “unless”s)
    • Parses the program, makes changes to the parse tree, and builds the new program
  • Heckle pros and cons
    • Best mutator out there
    • Except when it's not available
      • Doesn't work with default version of ruby on Windows
      • ParseTree doesn't work on ruby 1.9
      • Plans to replace ParseTree with RubyParser, which does work on ruby 1.9
  • Chaser
    • Modified version of heckle
    • Named after the ABC comedy team, best known for a penetration test that went horribly right
    • Created to deal with unavailability of heckle on Windows (which I no longer need)
    • Uses metaprogramming to modify methods
    • Intended to run under anything that's ruby
    • Runs test/unit and my fork of exemplor
  • Chaser metaprogramming def modify_instance_method chaser = self @mutated = true @old_method = @klass.instance_method(@method_name) @klass.class_eval do remove_method(chaser.method_name) define_method(chaser.method_name) do (*args) original_value = chaser.old_method.bind(self).call(*args) chaser.mutate_value(original_value) end end end
  • Chaser disadvantages
    • Only modifies return values (and yielded parameters)
    • Methods with side-effects will work as normal
    • Ability to not call a method at all could be added
    • Doesn't mutate every line
    • Currently doesn't describe the mutations being done
      • May be difficult task
  • Availability of chaser
    • Chaser (actually zombie-chaser) works on:
    • *nix 1.8 and 1.9
    • Windows 1.8 and 1.9
    • JRuby 1.8 only
      • Haven't got to the bottom yet for 1.9 (see next slide)
    • MacRuby: We broke it good! (see following slide)
    • Haven't tried IronRuby, Rubinius, or other platforms
  • Availability of chaser: JRuby 1.9 Andrew-Grimms-MacBook-Pro:zombie-chaser agrimm$ jruby --1.9 -Ilib bin/zombie-chaser /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:11:in `included' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `call' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `set_attributes' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `each' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `set_attributes' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `each' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `set_attributes' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:42:in `attribute' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `each' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `attribute' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:76:in `register_fixture' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:24:in `setup' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:9:in `included' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:8:in `class_eval' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:8:in `included' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/testcase.rb:82 /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/testcase.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit.rb:1 /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit- 2.0.7/lib/test/unit/ui/testrunnermediator.rb:7 /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/ui/testrunnermediator.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' /Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:4 /Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' bin/zombie-chaser:4 /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:11:in `included': undefined method `unregister_setup_method' for nil:NilClass (NoMethodError) from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `call' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `set_attributes' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `each' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `set_attributes' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `each' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `set_attributes' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:42:in `attribute' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `each' ... 16 levels... from /Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:31:in `require' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' from bin/zombie-chaser:4
  • Availability of chaser: MacRuby Not even close! [1] % ruby -Ilib -Itest -rrubygems test/test_unit.rb ~/Code/zombie-chaser /Users/pete/Code/zombie-chaser/test/test_unit.rb:1:in `<main>': unrecognized runtime type `' (TypeError) [1] % ruby -v ~/Code/zombie-chaser MacRuby version 0.6 (ruby 1.9.0) [universal-darwin10.0, x86_64] And no, we don't recognize the runtime type either! Thanks to Peter Hollows for running zombie-chaser on his box
  • Zombie-chaser
    • Graphic(al) user interface for mutation testing
    • Incorporated code from Chris Lloyd (et al)'s brains
    • Ryan Bigg demonstrated zombino, a brains-based game he created because brains wasn't working
    • psDooM, a Doom interface to sysadmin, was also an inspiration
  • Gooey metaphor
    • Human running across the screen represents running of your unit tests with unmodified code
      • Equivalent to a “control” in a scientific experiment
      • Fail any tests, and the human dies before zombies appear (boring!)
    • Zombies represent mutations to your code
      • Each successful test is represented with a step closer
      • Kill them with failing unit tests
  • Interfaces available
    • Nethack like representation
      • @ human
      • Z zombie
      • * dying zombie
      • + dead zombie
    • GUI interface
      • Gosu used
  • Disadvantages with Zombie-chaser
    • Issues shared with chaser:
      • Only modifies return values and yielded parameters
      • Doesn't describe the mutations
    • Additional issues
      • Doesn't explicitly describe which test it failed, or how
      • Stops running after the first failed test
  • Possible TODOs
    • Add color to console based output (make dying zombies red)
    • Make it work on more platforms
    • Make it work with more testing frameworks
    • Fix performance and threading bugs
    • Provide information that's currently not provided
    • Improve GUI metaphor
    • Sound and music
    • Gratuitous violence and copyright violation
    • See above