Zombie-chaser 0.1
Going beyond skin deep
Andrew Grimm
University of New
South Wales
Overview
● Gratuitous violence and copyright violation
● Unit testing
● Mutation testing
● Heckle
● Chaser
● Zombie-chaser...
Gratuitous violence and copyright violation
● See above
We don't test code
● I get a specification from my boss
● I spend a couple of weeks writing on my typewriter
● Fax it to t...
We all test code somehow
● SFTP code.php, make a few edits, SFTP it back to
the production server, view it in your browser...
Automated testing
● Term “Unit testing” sometimes misused
● Technically means testing a small portion of your
code
● Nowad...
Automated testing = exomemory
● Your code is thousands of lines long
– You can't memorize your code
● Requirements change ...
An example of a test
● Given I am a fake Canadian motorcade at the 2007
APEC summit
● When I supply obviously bogus creden...
Mutation testing
● Evaluates thoroughness of testing
● If your code is modified, it ought to fail some of its
tests
Example of a mutation
● Modifying an APEC credential verification method,
such that it regards an invalid pass as valid an...
Heckle
● Main mutation tester in ruby
● Name derived from Jester (Java mutation tester)
● Uses ParseTree and ruby2ruby
● M...
Heckle pros and cons
● Best mutator out there
● Except when it's not available
– Doesn't work with default version of ruby...
Chaser
● Modified version of heckle
● Named after the ABC comedy team, best known for
a penetration test that went horribl...
Chaser metaprogramming
def modify_instance_method
chaser = self
@mutated = true
@old_method = @klass.instance_method(@meth...
Chaser disadvantages
● Only modifies return values (and yielded
parameters)
● Methods with side-effects will work as norma...
Availability of chaser
● Chaser (actually zombie-chaser) works on:
● *nix 1.8 and 1.9
● Windows 1.8 and 1.9
● JRuby 1.8 on...
Availability of chaser: JRuby 1.9
Andrew-Grimms-MacBook-Pro:zombie-chaser agrimm$ jruby --1.9 -Ilib bin/zombie-chaser
/Use...
Availability of chaser: MacRuby
Not even close!
[1] % ruby -Ilib -Itest -rrubygems test/test_unit.rb ~/Code/zombie-chaser
...
Zombie-chaser
● Graphic(al) user interface for mutation testing
● Incorporated code from Chris Lloyd (et al)'s brains
● Ry...
Gooey metaphor
● Human running across the screen represents
running of your unit tests with unmodified code
– Equivalent t...
Interfaces available
● Nethack like representation
– @ human
– Z zombie
– * dying zombie
– + dead zombie
● GUI interface
–...
Disadvantages with Zombie-chaser
● Issues shared with chaser:
– Only modifies return values and yielded parameters
– Doesn...
Possible TODOs
● Add color to console based output (make dying
zombies red)
● Make it work on more platforms
● Make it wor...
Gratuitous violence and copyright violation
● See above
Upcoming SlideShare
Loading in...5
×

Zombie chaser 0.1

767

Published on

Zombie-chaser 0.1: Going beyond skin deep. A description of zombie-chaser and mutation testing.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
767
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Zombie chaser 0.1"

  1. 1. Zombie-chaser 0.1 Going beyond skin deep Andrew Grimm University of New South Wales
  2. 2. Overview ● Gratuitous violence and copyright violation ● Unit testing ● Mutation testing ● Heckle ● Chaser ● Zombie-chaser ● Gratuitous violence and copyright violation
  3. 3. Gratuitous violence and copyright violation ● See above
  4. 4. We don't test code ● I get a specification from my boss ● I spend a couple of weeks writing on my typewriter ● Fax it to the client ● They scan it in ● It runs first time ● And does what they wanted ● This DOESN'T happen
  5. 5. We all test code somehow ● SFTP code.php, make a few edits, SFTP it back to the production server, view it in your browser – This DOES happen ● svn update, make a few changes, try it on a development web server, don't spot anything wrong, push to production – Better ● git clone, add new test, add new code, check all tests pass, push to production – Yay!!
  6. 6. Automated testing ● Term “Unit testing” sometimes misused ● Technically means testing a small portion of your code ● Nowadays used to mean automated testing ● Sometimes automated testing is the lazier option than manual testing even in the short-term
  7. 7. Automated testing = exomemory ● Your code is thousands of lines long – You can't memorize your code ● Requirements change from the original – You can't write once, try it out, and never touch it again ● You can't email past-self – The code itself needs to know what's right and wrong
  8. 8. An example of a test ● Given I am a fake Canadian motorcade at the 2007 APEC summit ● When I supply obviously bogus credentials to security ● Then I should not be let in to the green zone
  9. 9. Mutation testing ● Evaluates thoroughness of testing ● If your code is modified, it ought to fail some of its tests
  10. 10. Example of a mutation ● Modifying an APEC credential verification method, such that it regards an invalid pass as valid and/or vice versa
  11. 11. Heckle ● Main mutation tester in ruby ● Name derived from Jester (Java mutation tester) ● Uses ParseTree and ruby2ruby ● Mutates literals (eg strings) and branches (eg swaps “if”s with “unless”s) ● Parses the program, makes changes to the parse tree, and builds the new program
  12. 12. Heckle pros and cons ● Best mutator out there ● Except when it's not available – Doesn't work with default version of ruby on Windows – ParseTree doesn't work on ruby 1.9 – Plans to replace ParseTree with RubyParser, which does work on ruby 1.9
  13. 13. Chaser ● Modified version of heckle ● Named after the ABC comedy team, best known for a penetration test that went horribly right ● Created to deal with unavailability of heckle on Windows (which I no longer need) ● Uses metaprogramming to modify methods ● Intended to run under anything that's ruby ● Runs test/unit and my fork of exemplor
  14. 14. Chaser metaprogramming def modify_instance_method chaser = self @mutated = true @old_method = @klass.instance_method(@method_name) @klass.class_eval do remove_method(chaser.method_name) define_method(chaser.method_name) do (*args) original_value = chaser.old_method.bind(self).call(*args) chaser.mutate_value(original_value) end end end
  15. 15. Chaser disadvantages ● Only modifies return values (and yielded parameters) ● Methods with side-effects will work as normal ● Ability to not call a method at all could be added ● Doesn't mutate every line ● Currently doesn't describe the mutations being done – May be difficult task
  16. 16. Availability of chaser ● Chaser (actually zombie-chaser) works on: ● *nix 1.8 and 1.9 ● Windows 1.8 and 1.9 ● JRuby 1.8 only – Haven't got to the bottom yet for 1.9 (see next slide) ● MacRuby: We broke it good! (see following slide) ● Haven't tried IronRuby, Rubinius, or other platforms
  17. 17. Availability of chaser: JRuby 1.9 Andrew-Grimms-MacBook-Pro:zombie-chaser agrimm$ jruby --1.9 -Ilib bin/zombie-chaser /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:11:in `included' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `call' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `set_attributes' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `each' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `set_attributes' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `each' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `set_attributes' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:42:in `attribute' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `each' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `attribute' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:76:in `register_fixture' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:24:in `setup' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:9:in `included' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:8:in `class_eval' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:8:in `included' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/testcase.rb:82 /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/testcase.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit.rb:1 /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/ui/testrunnermediator.rb:7 /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/ui/testrunnermediator.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' /Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:4 /Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' bin/zombie-chaser:4 /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:11:in `included': undefined method `unregister_setup_method' for nil:NilClass (NoMethodError) from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `call' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `set_attributes' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `each' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `set_attributes' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `each' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `set_attributes' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:42:in `attribute' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `each' ... 16 levels... from /Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:31:in `require' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' from bin/zombie-chaser:4
  18. 18. Availability of chaser: MacRuby Not even close! [1] % ruby -Ilib -Itest -rrubygems test/test_unit.rb ~/Code/zombie-chaser /Users/pete/Code/zombie-chaser/test/test_unit.rb:1:in `<main>': unrecognized runtime type `' (TypeError) [1] % ruby -v ~/Code/zombie-chaser MacRuby version 0.6 (ruby 1.9.0) [universal-darwin10.0, x86_64] And no, we don't recognize the runtime type either! Thanks to Peter Hollows for running zombie-chaser on his box
  19. 19. Zombie-chaser ● Graphic(al) user interface for mutation testing ● Incorporated code from Chris Lloyd (et al)'s brains ● Ryan Bigg demonstrated zombino, a brains-based game he created because brains wasn't working ● psDooM, a Doom interface to sysadmin, was also an inspiration
  20. 20. Gooey metaphor ● Human running across the screen represents running of your unit tests with unmodified code – Equivalent to a “control” in a scientific experiment – Fail any tests, and the human dies before zombies appear (boring!) ● Zombies represent mutations to your code – Each successful test is represented with a step closer – Kill them with failing unit tests
  21. 21. Interfaces available ● Nethack like representation – @ human – Z zombie – * dying zombie – + dead zombie ● GUI interface – Gosu used
  22. 22. Disadvantages with Zombie-chaser ● Issues shared with chaser: – Only modifies return values and yielded parameters – Doesn't describe the mutations ● Additional issues – Doesn't explicitly describe which test it failed, or how – Stops running after the first failed test
  23. 23. Possible TODOs ● Add color to console based output (make dying zombies red) ● Make it work on more platforms ● Make it work with more testing frameworks ● Fix performance and threading bugs ● Provide information that's currently not provided ● Improve GUI metaphor ● Sound and music
  24. 24. Gratuitous violence and copyright violation ● See above
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×