The presentation of my talk at JUG Padua 72 meeting, 08 October 2016.

1. PIT: state of the art of mutation testing system
Revised edition
MUTANTS KILLER
by Tàrin Gamberìni - CC BY-NC-SA 4.0

2. Lavora da sempre con tecnologie
Java-based free e Open Source
presso la Regione Emilia-Romagna
Si interessa di Object Oriented Design,
Design Patterns, Build Automation,
Testing e Continuous Integration
TÀRIN GAMBERÌNI
Partecipa attivamente
al Java User Group
Padova

3. Unit Testing
Test-Driven
Development (TDD)
Continuous Integration
Code Coverage
Mutation Testing
DEVELOPMENT

4. UNIT TESTING & TDD
The proof of working code
Refactor with confidence (regression)
Improve code design
Living documentation
TDD extremely valuable in some context

5. CONTINUOUS INTEGRATION
Maintain a single source repository
Automate the build
Cheap statics for code quality
Automate unit & integration tests
Catch problems fast

6. LINE COVERAGE
Is a measure of tested source code
line, statement, branch, … , coverage

7. LINE COVERAGE 100%
Only able to identify not tested code
Doesn't check tests are able to detect faults
Measures only which code is executed by tests

8. WEAK TESTS
public static String foo( boolean b ) {
if ( b ) {
performVitallyImportantBusinessFunction();
return "OK";
}
return "FAIL";
}
@Test
public void shouldFailWhenGivenFalse() {
assertEquals("FAIL", foo( false ));
}
@Test
public void shouldBeOkWhenGivenTrue() {
assertEquals("OK", foo( true ));
}
The untested side effect

9. WEAK TESTS
public static String foo( boolean b ) {
if ( b ) {
performVitallyImportantBusinessFunction();
return "OK";
}
return "FAIL";
}
@Test
public void shouldFailWhenGivenFalse() {
assertEquals("FAIL", foo( false ));
}
@Test
public void shouldBeOkWhenGivenTrue() {
assertEquals("OK", foo( true ));
}
The untested side effect

10. WEAK TESTS
public static String foo( int i ) {
if ( i >= 0 ) {
return "foo";
} else {
return "bar";
}
}
@Test
public void shouldReturnFooWhenGiven1() {
assertEquals("foo", foo( 1 ));
}
@Test
public void shouldReturnBarWhenGivenMinus1() {
assertEquals("bar", foo( -1 ));
}
The missing boundary test

11. WEAK TESTS
public static String foo( int i ) {
if ( i >= 0 ) {
return "foo";
} else {
return "bar";
}
}
@Test
public void shouldReturnBarWhenGiven1() {
assertEquals("bar", foo( 1 ));
}
@Test
public void shouldReturnFooWhenGivenZero() {
assertEquals("foo", foo( 0 ));
}
@Test
public void shouldReturnFooWhenGivenMinus1() {
assertEquals("foo", foo( -1 ));
}
The missing boundary test

12. WEAK TESTS
public static String foo(Collaborator c, boolean b) {
if ( b ) {
return c.performAction();
}
return "FOO";
} @Test
public void shouldPerformActionWhenGivenTrue() {
foo(mockCollaborator,true);
verify(mockCollaborator).performAction();
}
@Test
public void shouldNotPerformActionWhenGivenFalse() {
foo(mockCollaborator,false);
verify(never(),mockCollaborator).performAction();
}
The myopic mockist

13. WEAK TESTS
public static String foo(Collaborator c, boolean b) {
if ( b ) {
return c.performAction();
}
return "FOO";
} @Test
public void shouldPerformActionWhenGivenTrue() {
String result = foo(mockCollaborator,true);
verify(mockCollaborator).performAction();
assertEquals("MOCK", result);
}
@Test
public void shouldNotPerformActionWhenGivenFalse() {
String result = foo(mockCollaborator,false);
verify(never(),mockCollaborator).performAction();
assertEquals("FOO", result);
}
The myopic mockist

14. WHO TESTS THE TESTS ?

15. MUTATION TESTING
●
originally proposed by Richard Lipton
as a student in 1971
●
1st implementation tool was a PhD
work in 1980
●
recent availability of higher computing
power has led to a resurgence
●
expose weaknesses in tests (quality)

16. MUTATION TESTING
●
originally proposed by Richard Lipton
as a student in 1971
●
1st implementation tool was a PhD
work in 1980
●
recent availability of higher computing
power has led to a resurgence
●
expose weaknesses in tests (quality)

17. MUTATION
Is a small change in your program
original mutated

18. MUTANT
A
mutated
version of
your
program

19. MUTATION TESTING
generate lots of
mutants
run all tests against all mutants
check mutants:
killed or lived?

20. MUTANT KILLED
Proof of detected mutated code
Test is testing source code properly
If the unit test fails

21. MUTANT LIVED
Proof of not detected mutated code
If the unit test succeed
Test isn't testing source code

23. PIT: BASIC CONCEPTS
Applies a configurable set of mutators
Generates reports with test results
Manipulates the byte code generated
by compiling your code

24. PIT: MUTATORS
Conditionals Boundary
< → <=
<= → <
> → >=
>= → >

25. PIT: MUTATORS
Negate Conditionals
== → !=
!= → ==
<= → >
>= → <
< → >=
> → <=

26. PIT: MUTATORS
Math Mutator
+ → -
- → +
* → /
/ → *
% → *
& → |
| → &
^ → &
<< → >>
>> → <<
>>> → <<

27. PIT: MUTATORS
Remove Conditionals
if (a == b) {
// do something
}
if (true) {
// do something
}
→

28. PIT: MUTATORS
Return Values Mutator Conditionals
public Object foo() {
return new Object();
}
public Object foo() {
new Object();
return null;
}
→

29. PIT: MUTATORS
Void Method Call Mutator
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void
doSomething(int i) {
// does something
}
→
public int foo() {
int i = 5;
return i;
}
public void
doSomething(int i) {
// does something
}

30. PIT: MUTATORS
Many others
Activated by default
Conditionals Boundary Mutator - Increments Mutator -
Invert Negatives Mutator - Math Mutator - Negate
Conditionals Mutator - Return Values Mutator - Void
Method Calls Mutator
Deactivated by default
Constructor Calls Mutator - Inline Constant Mutator - Non
Void Method Calls Mutator - Remove Conditionals Mutator
- Experimental Member Variable Mutator - Experimental
Switch Mutator

31. PIT: MUTATORS
Application field
Traditional and class
PIT is suitable for "ordinary" java programs because
mutators change "ordinary" java language instructions.
The right mutator for the right application field
Not the best option if you are interesting testing:
●
Aspect Java programs, try AjMutator
●
Concurrent programs, try ExMan
●
Mutators for: reflection, java annotations, generics, … ?

32. HOW PIT WORKS
Running the tests
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
Your program

33. HOW PIT WORKS
Running the tests
PIT runs traditional line coverage analysis

34. HOW PIT WORKS
Running the tests
Generates mutant by applaying mutators
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
mutant 2
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
mutant 1
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
mutant 3

35. HOW PIT WORKS
Brute force
x = 9
100 tests x 1.2ms = 0.12s
0.12s x 10000 mutants = 20min !!!
1000 class x 10 mutants per class = 10000 mutants

36. HOW PIT WORKS
Running the tests
PIT leverages line coverage to discard tests
that do not exercise the mutated line of code
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
mutant 3

37. HOW PIT WORKS
Extremely fast
x = 1
x = 1
x = 2 4 < 9

38. HOW PIT WORKS
Outcomes
● Killed, Lived
● No coverage: the same as Lived except there were no
tests that exercised the mutated line of code
● Non viable: bytecode was in some way invalid
● Timed Out: infinite loop
● Memory error: mutation that increases the amount of
memory used
● Run error: a large number of run errors probably
means something went wrong

39. PIT ECOSYSTEM
Tools
Command line
Other IDEs as a
Java application
( )

40. EXAMPLE
class Ticket

41. EXAMPLE
testGetPrice_withAge11

42. EXAMPLE
testGetPrice_withAge71

43. EXAMPLE
testGetPrice_withAge18

44. EXAMPLE
mvn org.pitest:pitest-maven:mutationCoverage
>> Generated 8 mutations Killed 6 (75%)
>> Ran 11 tests (1.38 tests per mutation)
=================================================================
- Mutators
=================================================================
> org.pitest...ConditionalsBoundaryMutator
>> Generated 2 Killed 0 (0%)
> KILLED 0 SURVIVED 2 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
-----------------------------------------------------------------
> org.pitest...ReturnValsMutator
>> Generated 3 Killed 3 (100%)
> KILLED 3 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0

45. EXAMPLE

46. EXAMPLE

47. EXAMPLE
testGetPrice_withAge12

48. EXAMPLE
Helps discovering better test data
Well known techniques(1)
for creating data
Input space partitioning, boundary values, error
guessing.
Cartesian product of test data leads to a combinatorial
explosion.
PIT helps but YOU have to find
PIT tells you when a mutant has survived, but you are in
charge to find out the right test data which will kill it.
(1) Introduction to Software Testing, 2008 – by P. Ammann, J. Offutt

49. EXAMPLE
mvn org.pitest:pitest-maven:mutationCoverage
>> Generated 8 mutations Killed 7 (88%)
>> Ran 10 tests (1.25 tests per mutation)
=================================================================
- Mutators
=================================================================
> org.pitest...ConditionalsBoundaryMutator
>> Generated 2 Killed 1 (50%)
> KILLED 1 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
-----------------------------------------------------------------
> org.pitest...ReturnValsMutator
>> Generated 3 Killed 3 (100%)
> KILLED 3 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0

50. EXAMPLE

51. EXAMPLE

52. EXAMPLE
testGetPrice_withAge70

53. EXAMPLE
?!?!?!?!?

54. EXAMPLE
testGetPrice_withAge70

55. QUESTIONS ?
PIT: state of the art of mutation testing system
by
Tàrin Gamberìni
CC BY-NC-SA 4.0

56. CONTACT
Tàrin Gamberìni
tarin.gamberini@jugpadova.it
www.taringamberini.com

57. SOURCE CODE
https://github.com/taringamberini/linuxday2015.git

58. CREDITS
Minion with gun - by jpartwork (no license available)
https://nanookofthenerd.wordpress.com/2015/01/30/the-magnificent-maddening-marvelous-
minion-episode-032/
Minion Hands on - by Ceb1031 (CC BY-SA 3.0)
http://parody.wikia.com/wiki/File:Bob_minions_hands.jpg
Canon XTi components after disassembly - by particlem (CC BY 2.0)
https://www.flickr.com/photos/particlem/3860278043/
Continuous Integration - by New Media Labs (no license available)
http://newmedialabs.co.za/approach
Rally car - by mmphotography.it (CC BY-NC 2.0)
https://www.flickr.com/photos/grantuking/9705677549/
Crash Test dummies for sale - by Jack French (CC BY-NC 2.0)
https://www.flickr.com/photos/jackfrench/34523572/
Minion (no license available)
http://thisandthatforkidsblog.com/2013/09/17/easy-to-make-diy-minion-halloween-pumpkin/
Images

59. Mutant - by minions fans-d6txvph.jpg (no license available)
http://despicableme.wikia.com/wiki/File:Evil_minions_by_minions_fans-d6txvph.jpg
Mutant killed - by Patricia Uribe (no license available)
http://weheartit.com/entry/70334753
Mutant lived - by Carlos Hernández (no license available)
https://gifcept.com/VUQrTOr.gif
PIT logo - by Ling Yeung (CC BY-NC-SA 3.0)
http://pitest.org/about/
Circled Minions - by HARDEEP BHOGAL - (no license available)
https://plus.google.com/109649247879792393053/posts/ghH6tiuUZni?
pid=6004685782480998482&oid=109649247879792393053
Finger puppets – by Gary Leeming (CC BY-NC-SA 2.0)
https://www.flickr.com/photos/grazulis/4754781005/
Questions - by Shinichi Izumi (CC BY-SA)
http://despicableme.wikia.com/wiki/File:Purple_Minion.png
CREDITS
Images