• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Mutation Testing
 

Mutation Testing

on

  • 813 views

Brown Bag lightning talk I gave on 31st May 2012

Brown Bag lightning talk I gave on 31st May 2012

Statistics

Views

Total Views
813
Views on SlideShare
813
Embed Views
0

Actions

Likes
0
Downloads
14
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Difficult to identify equivalent mutants. There are some papers which suggest methods (but I didn’t have time to read them).
  • Since most of the team do Ruby, I’ve had a look into that too
  • Bytecode is faster to mutate as it avoids recompilationsJumble and Jester also seem quite popular
  • Exclude 3rd party frameworks
  • Looked into Heckle – since that was what the original topic of this talk was. Turns out it’s been dead for a long time.
  • Largely based on Heckle, rewritten on top of RubiniusOnly supports RSpec, but is that what’s used in the team? Author is looking to extend to other frameworks.Not sure if you need rubinius-head any more, but you did as of February 2012 (perhaps there’s a more stable version with support now)
  • If I’ve not hit the time limit, are there any questions?

Mutation Testing Mutation Testing Presentation Transcript

  • Mutation Testing Chris Sinjakli
  • Testing is a good thingBut how do we know our tests are good?
  • Code coverage is a startBut it can give a “good” score with really dreadful tests
  • Really dreadful testspublic int addTwoNumbers(int a, int b) { return a – b;}...@Testpublic void shouldAddTwoNumbers() { int result = addTwoNumbers(1, 1); assertTrue(true);} Coverage: 100% Usefulness: 0
  • A contrived exampleBut how could we detect it?
  • Mutation Testing!“Who watches the watchmen?”
  • If you can change the code, and atest doesn’t fail, either the code is never run or the tests are wrong.
  • Going with our previous examplepublic int addTwoNumbers(int a, int b) { return a – b;} Let’s change something...@Testpublic void shouldAddTwoNumbers() { int result = addTwoNumbers(1, 1); assertTrue(true);}
  • Going with our previous examplepublic int addTwoNumbers(int a, int b) { return a + b;}... This still passes@Testpublic void shouldAddTwoNumbers() { int result = addTwoNumbers(1, 1); assertTrue(true);}
  • So it caught a really rubbish test How about something slightly less obvious?
  • Slightly less obvious (and I mean slightly)public int checkConditions(boolean a, boolean b) { if (a && b) { return 42; } else { return 0; }}@Testpublic void testBothFalse() { int result = checkConditions(false, false); assertEquals(0, result);}@Testpublic void testBothTrue () { int result = checkConditions(true, true); assertEquals(42, result);} Coverage: 100% Usefulness: >0 But still wrong
  • Slightly less obvious (and I mean slightly)public int checkConditions(boolean a, boolean b) { if (a && b) { return 42; } Mutate else { return 0; }}@Testpublic void testBothFalse() { int result = checkConditions(false, false); assertEquals(0, result);}@Testpublic void testBothTrue () { int result = checkConditions(true, true); assertEquals(42, result);}
  • Slightly less obvious (and I mean slightly)public int checkConditions(boolean a, boolean b) { if (a || b) { return 42; } else { return 0; }}@Testpublic void testBothFalse() { Passing tests int result = checkConditions(false, false); assertEquals(0, result);}@Testpublic void testBothTrue () { int result = checkConditions(true, true); assertEquals(42, result);}
  • Mutation testing caught our mistake :D
  • Useful technique But still has its flaws
  • The downfall of mutation (Equivalent Mutants)int index = 0while (someCondition) { doStuff(); index++; if (index == 100) { break; }} Mutates toint index = 0while (someCondition) { doStuff(); index++; if (index >= 100) { break; }} But the programs are equivalent, so no test will fail
  • ToolsSome Java, then some Ruby
  • Java• Loads of tools to choose from• Bytecode vs source mutation• Will look at PIT (seems like one of the better ones)
  • PIT - pitest.org• Works with “everything” – Command line – Ant – Maven• Bytecode level mutations (faster)• Very customisable – Exclude classes/packages from mutation – Choose which mutations you want – Timeouts• Makes pretty HTML reports (line/mutation coverage)
  • Ruby
  • Ruby• Mutant seems to be the new favourite• Runs in Rubinius (1.8 or 1.9 mode)• Only supports RSpec• Easy to set up rvm install rbx-head rvm use rbx-head gem install mutant• And easy to use mutate “ClassName#method_to_test” spec
  • Summary• Seems like it could identify areas of weakness in our tests• At the same time, could be very noisy• Might be worth just trying it against an existing project and seeing what happens
  • Questions?