### Mutation testing with PIT

1. Mutation Testing with PIT Rafał Leszko @RafalLeszko rafalleszko.com Hazelcast
2. About me ● Integration Team Lead at Hazelcast ● Worked at Google and CERN ● Author of the book "Continuous Delivery with Docker and Jenkins" ● Trainer and conference speaker ● Live in Kraków, Poland
3. About Hazelcast ● Distributed Company ● Open Source Software ● 140+ Employees ● Products: ○ Hazelcast IMDG ○ Hazelcast Jet ○ Hazelcast Cloud @Hazelcast
4. I wrote code for the spacecraft.
5. How do you that it works?
6. Hmm...
7. I just know
8. I feel it
9. I feel it
10. I feel it
11. I feel it
12. \$327.6 million
13. I wrote code for the spacecraft.
14. How do you that it works?
15. I wrote unit tests!
16. How do you know that your tests work?
17. Hmm...
18. really?
19. code
20. code test
21. code test test
22. code test test test
23. code test test test test
24. Does testing make any sense at all?
25. return a + b;
26. return a + b;
27. Coverage does not prove that you have TESTED every line. All it proves is that you have EXECUTED every line. And that is a big difference.
28. public class Calculator { public int sum(int a, int b) { return a + b; } }
29. @Test public void force100PercentCoverage() { calculator.sum(0, 0); }
30. Richard Lipton found the good answer in 1971
31. "If you want to know if a test suite has properly checked some code, introduce a bug"
32. return a + b;
33. return a - b;
34. Is it the whole idea behind the mutation testing?
35. Actually, Yes!
36. ● Artificial bug: MUTATION OPERATION
37. ● Artificial bug: MUTATION OPERATION ● Code with artificial bug: MUTANT
38. ● Artificial bug: MUTATION OPERATION ● Code with artificial bug: MUTANT ● When the test fails on mutant: KILLED the mutant
39. ● Artificial bug: MUTATION OPERATION ● Code with artificial bug: MUTANT ● When the test fails on mutant: KILLED the mutant ● When the test succeeds on mutant: mutant SURVIVED
40. return a - b; KILLED SURVIVED
41. killing is good
42. My code is much more complex than adding two numbers!
43. Mutation Operators
44. Math + to - * to / - to + / to *
45. Boundary < to <= > to >= <= to < <= to >
46. Negate == to != >= to <
47. Complex Remove IFs
48. Complex Remove IFs Remove method calls
49. Complex Remove IFs Modify return Remove method calls
50. Complex Remove IFs Modify constantsModify return Remove method calls
51. Do I need to change the code on my own?
52. PIT Mutation Testing
53. public class Calculator { public int sum(int a, int b) { return a + b; } } @Test public void force100PercentCoverage() { calculator.sum(0, 0); }
54. DEMO
55. > MathMutator >> Generated 1 Killed 0 (0%) > KILLED 0 SURVIVED 1 > TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 > STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0
56. @Test public void Should_ReturnSum_When_SummingTwoValues() { // given int a = 1; int b = 2; // when int result = calculator.sum(a, b); // then assertThat(result).isEqualTo(3); }
57. > MathMutator >> Generated 1 Killed 1 (100%) > KILLED 1 SURVIVED 0 > TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 > STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0
58. Do I need to read the console?
59. DEMO
61. Hazelcast Kubernetes Plugin ● 5000 LOC ● 12 classes
62. DEMO
63. Why NASA engineers didn't use it in 1998?
64. 1971 Richard Lipton
65. 1971 Richard Lipton 2000
66. 1971 Richard Lipton 2000 2012
67. 1 Equivalent Mutants
68. if (a >= 100) { throw new IllegalArgumentException(); } if (a < 100) { fooBar(); }
69. if (a >= 100) { throw new IllegalArgumentException(); } if (a < 100) { fooBar(); }
70. 2 Slow Performance
71. return a + b;
72. return a + b;
73. return a + b; return a - b return 0
74. return a + b; return a - b return 0
75. :)