Prezentacja wygłoszona na 16tym spotkaniu KraQA, 12.01.2016

1. MUTATION TESTING

3. Woot?
EXAMPLE 1
public void testMethod() {
Something something = new Something();
something.executeSomeMethod();
}
EXAMPLE 2
public void testConstructor() {
Something something = new Something();
}

4. LITTLE HISTORY
FIRST PROPOSED in 1971 by Richard Lipton
A. Jefferson Offutt and R. H. Untch “Mutation 2000: Uniting the Orthogonal”
FIRST DEFINED in 1978
R. A. De Millo, R. J. Lipton, F. G. Sayward “Hints on test data selection”
FIRST IMPLEMENTED in 1980 by Timothy Budd
T. A. Budd, “Mutation Analysis of Program Test Data”

5. PROBLEM

6. SOLUTION

7. EXPECTATIONS

8. TESTS - TDD?

9. TESTS - NO TDD?

10. TEST

11. MUTATION

12. TESTING MUTATED CODE...

13. EXPECTED RESULT

14. SOMETHING WRONG?

16. FLOW
1. Run tests
2. Mutate production code
3. Run tests again
4. Generate report

17. KILL THE MUTANT!

18. TESTS RESULTS?
PASSED (SURVIVED) FAILED (KILLED)
other results applicable (e. g. timeout, memory error)

19. EXAMPLE
ORIGINAL
if (i >= 0) {
return "foo";
} else {
return "bar";
}
MUTATED
if (i > 0) {
return "foo";
} else {
return "bar";
}
conditionals boundary mutation

20. COMMON MUTATORS #1
CONDITIONALS MUTATORS
< -> <=
<= -> <
> -> >=
>= -> >
== -> !=
!= -> ==
MATH MUTATORS
+ -> -
- -> +
>> -> <<
<< -> >>
* -> /
/ -> *

21. COMMON MUTATORS #2
RETURN VALUES MUTATOR
returns true if original returns false
returns false if original returns true
returns 0 if original returns int != 0
returns 1 if original returns 0
METHOD CALLS MUTATOR
removes void method call
removes non-void method call, returns default
value for the return type
replaces members with null in constructors

22. SURVIVORS STORIES

23. HOW TO GET STARTED?
PITEST for JAVA http://pitest.org
Parasoft INSURE++ for C/C++ https://www.parasoft.com/product/insure/
MutPy for Python https://pypi.python.org/pypi/MutPy/0.4.0
Maven: mvn org.pitest:pitest-maven:mutationCoverage

24. PERFORMANCE
100 classes
10 tests / class
1 ms / test
tTOTAL = 100 x 10 x 1ms = 1s
10 mutations / class
tALL = 10 x 100 x 1s ~ 17min
tSMART = 10 x 100 x (10 x 1ms) = 10s
1000 classes
10 tests / class
1 ms / test
tTOTAL = 1000 x 10 x 1ms = 10s
10 mutations / class
tALL = 10 x 1000 x 10s ~ 28h
tSMART = 10 x 1000 x (10 x 1ms) ~ 2min
real life example: 481 test cases fully mutated on 16GB i7 -> 03:24 min

26. ATTRIBUTIONS
Hands Up - Avel Chuklanov - https://unsplash.com/photos/9cx4-QowgLc (CC0 1.0)
Bridge - Franc - https://thenounproject.com/search/?q=bridge&i=122542 (CC BY 3.0 US)
Cargo Truck - No way - https://thenounproject.com/search/?q=truck&i=154864 (CC0 1.0)
Philosoraptor - OnlyOneKenobi - http://wallpaperswide.com/philosoraptor-wallpapers.html
Chainsaw - James Keuning - https://thenounproject.com/search/?q=chainsaw&i=11328 (CC0 1.0)
Refugees - Gerald Wildmoser - https://thenounproject.com/search/?q=drown&i=208787 (CC BY 3.0 US)
Resurrection - Isabel Martinez Isabel - https://thenounproject.com/search/?q=survived&i=202842 (CC BY 3.0 US)
Pac Man - iconoci - https://thenounproject.com/search/?q=pac+man&i=9215 (CC BY 3.0 US)
Stopwatch - Nick Holroyd - https://thenounproject.com/search/?q=stopwatch&i=12666 (CC BY 3.0 US)
Zombie analysis - clement127 - https://www.flickr.com/photos/clement127/16089737804/ (CC BY-NC-ND 2.0)
in order of appearance

27. THANK YOU!
Mateusz Bryła