Zombie-chaser 0.1: Going beyond skin deep. A description of zombie-chaser and mutation testing.

1. Zombie-chaser 0.1
Going beyond skin deep
Andrew Grimm
University of New
South Wales

2. Overview
● Gratuitous violence and copyright violation
● Unit testing
● Mutation testing
● Heckle
● Chaser
● Zombie-chaser
● Gratuitous violence and copyright violation

3. Gratuitous violence and copyright violation
● See above

4. We don't test code
● I get a specification from my boss
● I spend a couple of weeks writing on my typewriter
● Fax it to the client
● They scan it in
● It runs first time
● And does what they wanted
● This DOESN'T happen

5. We all test code somehow
● SFTP code.php, make a few edits, SFTP it back to
the production server, view it in your browser
– This DOES happen
● svn update, make a few changes, try it on a
development web server, don't spot anything wrong,
push to production
– Better
● git clone, add new test, add new code, check all tests
pass, push to production
– Yay!!

6. Automated testing
● Term “Unit testing” sometimes misused
● Technically means testing a small portion of your
code
● Nowadays used to mean automated testing
● Sometimes automated testing is the lazier option
than manual testing even in the short-term

7. Automated testing = exomemory
● Your code is thousands of lines long
– You can't memorize your code
● Requirements change from the original
– You can't write once, try it out, and never touch it
again
● You can't email past-self
– The code itself needs to know what's right and wrong

8. An example of a test
● Given I am a fake Canadian motorcade at the 2007
APEC summit
● When I supply obviously bogus credentials to
security
● Then I should not be let in to the green zone

9. Mutation testing
● Evaluates thoroughness of testing
● If your code is modified, it ought to fail some of its
tests

10. Example of a mutation
● Modifying an APEC credential verification method,
such that it regards an invalid pass as valid and/or
vice versa

11. Heckle
● Main mutation tester in ruby
● Name derived from Jester (Java mutation tester)
● Uses ParseTree and ruby2ruby
● Mutates literals (eg strings) and branches (eg swaps
“if”s with “unless”s)
● Parses the program, makes changes to the parse
tree, and builds the new program

12. Heckle pros and cons
● Best mutator out there
● Except when it's not available
– Doesn't work with default version of ruby on
Windows
– ParseTree doesn't work on ruby 1.9
– Plans to replace ParseTree with RubyParser, which
does work on ruby 1.9

13. Chaser
● Modified version of heckle
● Named after the ABC comedy team, best known for
a penetration test that went horribly right
● Created to deal with unavailability of heckle on
Windows (which I no longer need)
● Uses metaprogramming to modify methods
● Intended to run under anything that's ruby
● Runs test/unit and my fork of exemplor

14. Chaser metaprogramming
def modify_instance_method
chaser = self
@mutated = true
@old_method = @klass.instance_method(@method_name)
@klass.class_eval do
remove_method(chaser.method_name)
define_method(chaser.method_name) do (*args)
original_value = chaser.old_method.bind(self).call(*args)
chaser.mutate_value(original_value)
end
end
end

15. Chaser disadvantages
● Only modifies return values (and yielded
parameters)
● Methods with side-effects will work as normal
● Ability to not call a method at all could be added
● Doesn't mutate every line
● Currently doesn't describe the mutations being done
– May be difficult task

16. Availability of chaser
● Chaser (actually zombie-chaser) works on:
● *nix 1.8 and 1.9
● Windows 1.8 and 1.9
● JRuby 1.8 only
– Haven't got to the bottom yet for 1.9 (see next slide)
● MacRuby: We broke it good! (see following slide)
● Haven't tried IronRuby, Rubinius, or other platforms

17. Availability of chaser: JRuby 1.9
Andrew-Grimms-MacBook-Pro:zombie-chaser agrimm$ jruby --1.9 -Ilib bin/zombie-chaser
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:11:in `included'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `call'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `set_attributes'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `each'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `set_attributes'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `each'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `set_attributes'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:42:in `attribute'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `each'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `attribute'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:76:in `register_fixture'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:24:in `setup'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:9:in `included'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:8:in `class_eval'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:8:in `included'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/testcase.rb:82
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/testcase.rb:31:in `require'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit.rb:1
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit.rb:31:in `require'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/ui/testrunnermediator.rb:7
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/ui/testrunnermediator.rb:31:in `require'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
/Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:4
/Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:31:in `require'
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
bin/zombie-chaser:4
/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:11:in `included': undefined method `unregister_setup_method' for nil:NilClass (NoMethodError)
from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `call'
from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `set_attributes'
from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `each'
from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `set_attributes'
from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `each'
from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `set_attributes'
from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:42:in `attribute'
from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `each'
... 16 levels...
from /Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:31:in `require'
from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from bin/zombie-chaser:4

18. Availability of chaser: MacRuby
Not even close!
[1] % ruby -Ilib -Itest -rrubygems test/test_unit.rb ~/Code/zombie-chaser
/Users/pete/Code/zombie-chaser/test/test_unit.rb:1:in `<main>': unrecognized runtime type `'
(TypeError)
[1] % ruby -v ~/Code/zombie-chaser
MacRuby version 0.6 (ruby 1.9.0) [universal-darwin10.0, x86_64]
And no, we don't recognize the runtime type either!
Thanks to Peter Hollows for running zombie-chaser on his box

19. Zombie-chaser
● Graphic(al) user interface for mutation testing
● Incorporated code from Chris Lloyd (et al)'s brains
● Ryan Bigg demonstrated zombino, a brains-based
game he created because brains wasn't working
● psDooM, a Doom interface to sysadmin, was also an
inspiration

20. Gooey metaphor
● Human running across the screen represents
running of your unit tests with unmodified code
– Equivalent to a “control” in a scientific experiment
– Fail any tests, and the human dies before zombies
appear (boring!)
● Zombies represent mutations to your code
– Each successful test is represented with a step closer
– Kill them with failing unit tests

21. Interfaces available
● Nethack like representation
– @ human
– Z zombie
– * dying zombie
– + dead zombie
● GUI interface
– Gosu used

22. Disadvantages with Zombie-chaser
● Issues shared with chaser:
– Only modifies return values and yielded parameters
– Doesn't describe the mutations
● Additional issues
– Doesn't explicitly describe which test it failed, or
how
– Stops running after the first failed test

23. Possible TODOs
● Add color to console based output (make dying
zombies red)
● Make it work on more platforms
● Make it work with more testing frameworks
● Fix performance and threading bugs
● Provide information that's currently not provided
● Improve GUI metaphor
● Sound and music

24. Gratuitous violence and copyright violation
● See above