Your SlideShare is downloading. ×
Zombie chaser 0.1
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Zombie chaser 0.1

721
views

Published on

Zombie-chaser 0.1: Going beyond skin deep. A description of zombie-chaser and mutation testing.

Zombie-chaser 0.1: Going beyond skin deep. A description of zombie-chaser and mutation testing.

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
721
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Zombie-chaser 0.1 Going beyond skin deep Andrew Grimm University of New South Wales
  • 2. Overview ● Gratuitous violence and copyright violation ● Unit testing ● Mutation testing ● Heckle ● Chaser ● Zombie-chaser ● Gratuitous violence and copyright violation
  • 3. Gratuitous violence and copyright violation ● See above
  • 4. We don't test code ● I get a specification from my boss ● I spend a couple of weeks writing on my typewriter ● Fax it to the client ● They scan it in ● It runs first time ● And does what they wanted ● This DOESN'T happen
  • 5. We all test code somehow ● SFTP code.php, make a few edits, SFTP it back to the production server, view it in your browser – This DOES happen ● svn update, make a few changes, try it on a development web server, don't spot anything wrong, push to production – Better ● git clone, add new test, add new code, check all tests pass, push to production – Yay!!
  • 6. Automated testing ● Term “Unit testing” sometimes misused ● Technically means testing a small portion of your code ● Nowadays used to mean automated testing ● Sometimes automated testing is the lazier option than manual testing even in the short-term
  • 7. Automated testing = exomemory ● Your code is thousands of lines long – You can't memorize your code ● Requirements change from the original – You can't write once, try it out, and never touch it again ● You can't email past-self – The code itself needs to know what's right and wrong
  • 8. An example of a test ● Given I am a fake Canadian motorcade at the 2007 APEC summit ● When I supply obviously bogus credentials to security ● Then I should not be let in to the green zone
  • 9. Mutation testing ● Evaluates thoroughness of testing ● If your code is modified, it ought to fail some of its tests
  • 10. Example of a mutation ● Modifying an APEC credential verification method, such that it regards an invalid pass as valid and/or vice versa
  • 11. Heckle ● Main mutation tester in ruby ● Name derived from Jester (Java mutation tester) ● Uses ParseTree and ruby2ruby ● Mutates literals (eg strings) and branches (eg swaps “if”s with “unless”s) ● Parses the program, makes changes to the parse tree, and builds the new program
  • 12. Heckle pros and cons ● Best mutator out there ● Except when it's not available – Doesn't work with default version of ruby on Windows – ParseTree doesn't work on ruby 1.9 – Plans to replace ParseTree with RubyParser, which does work on ruby 1.9
  • 13. Chaser ● Modified version of heckle ● Named after the ABC comedy team, best known for a penetration test that went horribly right ● Created to deal with unavailability of heckle on Windows (which I no longer need) ● Uses metaprogramming to modify methods ● Intended to run under anything that's ruby ● Runs test/unit and my fork of exemplor
  • 14. Chaser metaprogramming def modify_instance_method chaser = self @mutated = true @old_method = @klass.instance_method(@method_name) @klass.class_eval do remove_method(chaser.method_name) define_method(chaser.method_name) do (*args) original_value = chaser.old_method.bind(self).call(*args) chaser.mutate_value(original_value) end end end
  • 15. Chaser disadvantages ● Only modifies return values (and yielded parameters) ● Methods with side-effects will work as normal ● Ability to not call a method at all could be added ● Doesn't mutate every line ● Currently doesn't describe the mutations being done – May be difficult task
  • 16. Availability of chaser ● Chaser (actually zombie-chaser) works on: ● *nix 1.8 and 1.9 ● Windows 1.8 and 1.9 ● JRuby 1.8 only – Haven't got to the bottom yet for 1.9 (see next slide) ● MacRuby: We broke it good! (see following slide) ● Haven't tried IronRuby, Rubinius, or other platforms
  • 17. Availability of chaser: JRuby 1.9 Andrew-Grimms-MacBook-Pro:zombie-chaser agrimm$ jruby --1.9 -Ilib bin/zombie-chaser /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:11:in `included' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `call' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `set_attributes' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `each' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `set_attributes' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `each' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `set_attributes' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:42:in `attribute' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `each' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `attribute' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:76:in `register_fixture' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:24:in `setup' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:9:in `included' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:8:in `class_eval' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:8:in `included' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/testcase.rb:82 /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/testcase.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit.rb:1 /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/ui/testrunnermediator.rb:7 /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/ui/testrunnermediator.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' /Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:4 /Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:31:in `require' /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' bin/zombie-chaser:4 /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:11:in `included': undefined method `unregister_setup_method' for nil:NilClass (NoMethodError) from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `call' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `set_attributes' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `each' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `set_attributes' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `each' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `set_attributes' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:42:in `attribute' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `each' ... 16 levels... from /Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:31:in `require' from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' from bin/zombie-chaser:4
  • 18. Availability of chaser: MacRuby Not even close! [1] % ruby -Ilib -Itest -rrubygems test/test_unit.rb ~/Code/zombie-chaser /Users/pete/Code/zombie-chaser/test/test_unit.rb:1:in `<main>': unrecognized runtime type `' (TypeError) [1] % ruby -v ~/Code/zombie-chaser MacRuby version 0.6 (ruby 1.9.0) [universal-darwin10.0, x86_64] And no, we don't recognize the runtime type either! Thanks to Peter Hollows for running zombie-chaser on his box
  • 19. Zombie-chaser ● Graphic(al) user interface for mutation testing ● Incorporated code from Chris Lloyd (et al)'s brains ● Ryan Bigg demonstrated zombino, a brains-based game he created because brains wasn't working ● psDooM, a Doom interface to sysadmin, was also an inspiration
  • 20. Gooey metaphor ● Human running across the screen represents running of your unit tests with unmodified code – Equivalent to a “control” in a scientific experiment – Fail any tests, and the human dies before zombies appear (boring!) ● Zombies represent mutations to your code – Each successful test is represented with a step closer – Kill them with failing unit tests
  • 21. Interfaces available ● Nethack like representation – @ human – Z zombie – * dying zombie – + dead zombie ● GUI interface – Gosu used
  • 22. Disadvantages with Zombie-chaser ● Issues shared with chaser: – Only modifies return values and yielded parameters – Doesn't describe the mutations ● Additional issues – Doesn't explicitly describe which test it failed, or how – Stops running after the first failed test
  • 23. Possible TODOs ● Add color to console based output (make dying zombies red) ● Make it work on more platforms ● Make it work with more testing frameworks ● Fix performance and threading bugs ● Provide information that's currently not provided ● Improve GUI metaphor ● Sound and music
  • 24. Gratuitous violence and copyright violation ● See above