This document discusses mutation testing and how it can be used to evaluate test quality. Mutation testing works by artificially inserting bugs called "mutants" into code and seeing if tests can detect the mutations. If tests fail on a mutant, it is considered "killed", otherwise the mutant is said to have "survived". The document demonstrates how to use the PIT mutation testing tool on simple calculator code and a larger codebase. Adoption of mutation testing has increased in recent decades as tools have improved performance. Mutation testing can help find gaps in test coverage and ensure tests are properly evaluating the logic of the code.

1. Mutation Testing with PIT
Rafał Leszko
@RafalLeszko
rafalleszko.com
Hazelcast

2. About me
● Cloud-Native Tech Lead at Hazelcast
● Worked at Google and CERN
● Author of the book "Continuous Delivery
with Docker and Jenkins"
● Trainer and conference speaker
● Live in Kraków, Poland

3. About Hazelcast
● Distributed Company
● Open Source Software
● 140+ Employees
@Hazelcast

6. I wrote code for the
spacecraft.

7. How do you that it
works?

8. Hmm...

9. I just know

10. I feel it

11. I feel it

12. I feel it

13. I feel it

16. $327.6 million

23. I wrote code for the
spacecraft.

24. How do you that it
works?

25. I wrote unit tests!

26. How do you know
that your tests work?

27. Hmm...

28. really?

29. code

30. code
test

31. code
test
test

32. code
test
test
test

33. code
test
test
test
test

35. Does testing make
any sense at all?

36. return a + b;

37. return a + b;

38. Coverage does not
prove that you have
TESTED every line.
All it proves is that you
have EXECUTED every
line. And that is a big
difference.

39. public class Calculator {
public int sum(int a, int b) {
return a + b;
}
}

40. @Test
public void
force100PercentCoverage() {
calculator.sum(0, 0);
}

41. Richard Lipton found the
good answer in 1971

46. "If you want to know if a test
suite has properly checked
some code, introduce a bug"

47. return a + b;

48. return a - b;

49. Is it the whole idea behind the
mutation testing?

50. Actually, Yes!

51. ● Artificial bug: MUTATION OPERATION

52. ● Artificial bug: MUTATION OPERATION
● Code with artificial bug: MUTANT

53. ● Artificial bug: MUTATION OPERATION
● Code with artificial bug: MUTANT
● When the test fails on mutant: KILLED the
mutant

54. ● Artificial bug: MUTATION OPERATION
● Code with artificial bug: MUTANT
● When the test fails on mutant: KILLED the
mutant
● When the test succeeds on mutant:
mutant SURVIVED

55. return a - b;
KILLED SURVIVED

56. killing is good

57. My code is much more
complex than adding two
numbers!

58. Mutation
Operators

59. Math
+ to - * to /
- to + / to *

60. Boundary
< to <= > to >=
<= to < <= to >

61. Negate
== to !=
>= to <

62. Complex
Remove IFs

63. Complex
Remove IFs Remove method calls

64. Complex
Remove IFs
Modify return
Remove method calls

65. Complex
Remove IFs
Modify constants
Modify return
Remove method calls

71. Do I need to change the
code on my own?

72. PIT Mutation Testing

73. public class Calculator {
public int sum(int a, int b) {
return a + b;
}
}
@Test
public
void force100PercentCoverage() {
calculator.sum(0, 0);
}

74. DEMO

75. > MathMutator
>> Generated 1 Killed 0 (0%)
> KILLED 0 SURVIVED 1
> TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0
> STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0

76. @Test public void
Should_ReturnSum_When_SummingTwoValues() {
// given
int a = 1;
int b = 2;
// when
int result = calculator.sum(a, b);
// then
assertThat(result).isEqualTo(3);
}

77. > MathMutator
>> Generated 1 Killed 1 (100%)
> KILLED 1 SURVIVED 0
> TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0
> STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0

78. Do I need to read the
console?

79. DEMO

80. How about bigger
projects?

81. Hazelcast Kubernetes Plugin
● 5000 LOC
● 12 classes

82. DEMO

83. Why NASA engineers
didn't use it in 1998?

84. 1971
Richard
Lipton

85. 1971
Richard
Lipton
2000

86. 1971
Richard
Lipton
2000 2012

87. 1
Equivalent
Mutants

88. if (a >= 100) {
throw new IllegalArgumentException();
}
if (a < 100) {
fooBar();
}

89. if (a >= 100) {
throw new IllegalArgumentException();
}
if (a < 100) {
fooBar();
}

90. 2
Slow
Performance

93. return a + b;

94. return a + b;

95. return a + b;
return a - b
return 0

96. return a + b;
return a - b
return 0

97. :)

98. What about my
team?

103. Main class paradox

106. // given

107. // given
// when

108. // given
// when
?

109. What to do with people
who write tests without
assertions?

113. Do people really use
mutation testing?

116. Test your tests

122. Thank You!
Rafał Leszko
@RafalLeszko
rafalleszko.com