Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The CI as a partner for test improvement suggestions

209 views

Published on

Talk at DevoxxFR on April 11, 2019

Published in: Software
  • Be the first to comment

  • Be the first to like this

The CI as a partner for test improvement suggestions

  1. 1. 1 The CI as a partner for test improvement suggestions Benoit Baudry Project Coordinator and Scientific Leader KTH, Sweden Caroline Landry Project Technical Manager INRIA, France 19-Apr-2019
  2. 2. This work was partially supported by the EU Project STAMP ICT-16-10 No.731529 •4 research institutions •5 companies •1 open source consortium •Automated Testing in DevOps
  3. 3. DevOps 3
  4. 4. DevOps 4 • Better quality
  5. 5. DevOps 5 • Better quality • Shorter release cycles
  6. 6. DevOps 6 • Better quality • Shorter release cycles • Reconnect Ops to Dev
  7. 7. DevOps 7 Automation• Better quality • Shorter release cycles • Reconnect Ops to Dev
  8. 8. DevOps 8 Step acceptance & Feedback Automation• Better quality • Shorter release cycles • Reconnect Ops to Dev
  9. 9. DevOps 9
  10. 10. DevOps 10 CI
  11. 11. DevOps 11 IDEs CI completion linter
  12. 12. DevOps 12 unit perf. UI IDEs CI completion linter
  13. 13. DevOps 13 unit perf. UI Config. IDEs CI completion linter
  14. 14. DevOps 14 unit perf. fuzzing UI Config. IDEs CI completion linter coverage mutation
  15. 15. DevOps 15 unit perf. fuzzing UI Config. IDEs CI completion linter coverage mutation CD
  16. 16. DevOps 16 unit perf. fuzzing UI chaos A/B testing Config. IDEs CI completion linter coverage mutation CD
  17. 17. DevOps 17 unit perf. fuzzing UI chaos Logs analysis A/B testing Config. IDEs CI completion linter coverage mutation Crash analysis CD
  18. 18. DevOps 18 unit perf. fuzzing UI chaos Logs analysis A/B testing Config. IDEs Continuous testing CI completion linter coverage mutation Crash analysis CD
  19. 19. 19
  20. 20. STAMP’s concept: amplification  Amplify (v.): to increase the size or effect of something https://dictionary.cambridge.org/dictionary/english/amplify 20
  21. 21. STAMP’s concept: amplification  Amplify (v.): to increase the size or effect of something  Test amplification: Increase the effect of test assets 21https://dictionary.cambridge.org/dictionary/english/amplify
  22. 22. STAMP’s concept: amplification  Amplify (v.): to increase the size or effect of something  Test amplification: Increase the effect of test assets  Test assets: test cases, configuration files, production logs  Effect metrics: mutation score, feature interactions  Automatic amplification 22https://dictionary.cambridge.org/dictionary/english/amplify
  23. 23. DevOps – STAMP focus 23
  24. 24. DevOps – STAMP focus 24
  25. 25. DevOps – STAMP focus 25
  26. 26. DevOps – STAMP focus 26
  27. 27. Unit test amplification 27 Descartes
  28. 28. Test Your Tests •What do you expect from test cases? 28
  29. 29. Test Your Tests •What do you expect from test cases? • Cover requirements • Code coverage • Stress the application • Reveal bugs 29
  30. 30. Test Your Tests •What do you expect from test cases? • Cover requirements • Code coverage • Stress the application • Reveal bugs 30
  31. 31. Example 31 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; }
  32. 32. Example 32 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } Coverage
  33. 33. Example 33 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } @Test factorialWith5Test() { long obs = fact(5); assertTrue(obs > 5); } Coverage
  34. 34. Example 34 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } @Test factorialWith5Test() { long obs = fact(5); assertTrue(obs > 5); } @Test factorialWith0Test() { assertEqual(fact(0), 1); } Coverage
  35. 35. Example 35 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } Coverage @Test factorialWith5Test() { long obs = fact(5); assertTrue(obs > 5); } @Test factorialWith0Test() { assertEqual(fact(0), 1); } Is these test suite good at detecting bugs?
  36. 36. Example 36 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } Coverage @Test factorialWith5Test() { long obs = fact(5); assertTrue(obs > 5); } @Test factorialWith0Test() { assertEqual(fact(0), 1); } Is these test suite good at detecting bugs? Let’s mutate our code to see.
  37. 37. Mutation Analysis 37
  38. 38. Mutation Analysis 38 inject fault Mutant 1
  39. 39. Mutation Analysis 39 inject fault Mutant 1 Run
  40. 40. Mutation Analysis 40 inject fault Mutant 1 Run ✗ Mutant 1
  41. 41. Mutation Analysis 41 inject fault Mutant 1 Run ✗ Mutant 1 Killed 
  42. 42. Mutation Analysis 42 inject fault Mutant 1 Mutant 2 Run ✗ Mutant 1 Killed 
  43. 43. Mutation Analysis 43 inject fault Mutant 1 Mutant 2 Run ✗ Mutant 1 Killed 
  44. 44. Mutation Analysis 44 inject fault Mutant 1 Mutant 2 Run ✗ ✗ Mutant 1 Mutant 2 Killed 
  45. 45. Mutation Analysis 45 inject fault Mutant 1 Mutant 2 Run ✗ ✗ Mutant 1 Mutant 2 Killed  Killed 
  46. 46. Mutation Analysis 46 inject fault Mutant 1 Mutant 2 Mutant 3 Run ✗ ✗ Mutant 1 Mutant 2 Killed  Killed 
  47. 47. Mutation Analysis 47 inject fault Mutant 1 Mutant 2 Mutant 3 Run ✗ ✗ Mutant 1 Mutant 2 Killed  Killed 
  48. 48. Mutation Analysis 48 inject fault Mutant 1 Mutant 2 Mutant 3 Run ✗ ✓ ✗ Mutant 1 Mutant 2 Mutant 3 Killed  Killed 
  49. 49. Mutation Analysis 49 inject fault Mutant 1 Mutant 2 Mutant 3 Run ✗ ✓ ✗ Mutant 1 Mutant 2 Mutant 3 Killed  Alive  Killed 
  50. 50. Mutation Analysis 50 inject fault Mutant 1 Mutant 2 Mutant 3 Run ✗ ✓ ✗  Mutation score Mutant 1 Mutant 2 Mutant 3 Killed  Alive  Killed 
  51. 51. Example 51 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } @Test factorialWith5Test() { long obs = fact(5); assertTrue(obs > 5); } @Test factorialWith0Test() { assertEqual(fact(0), 1); }
  52. 52. long fact(int n) { if (n != 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } Example 52 ✗ ✓ @Test factorialWith5Test() { long obs = fact(5); assertTrue(obs > 5); } @Test factorialWith0Test() { assertEqual(fact(0), 1); }
  53. 53. long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } Example 53 n!=0 return 1+1 i < n !(i<=n) i-- result/i result+1
  54. 54. Example 54 n!=0 return 1+1 i < n !(i<=n) i-- result/i result+1 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } @Test factorialWith5Test() { long obs = fact(5); assertTrue(obs > 5); }
  55. 55. Example 55 n!=0 return 1+1 i < n !(i<=n) i-- result/i result+1 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } @Test factorialWith0Test() { assertEqual(fact(0), 1); }
  56. 56. Example 56 n!=0 return 1+1 i < n !(i<=n) i-- result/i result+1 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } @Test factorialWith5Test() { long obs = fact(5); assertTrue(obs > 5); } @Test factorialWith0Test() { assertEqual(fact(0), 1); }
  57. 57. Example 57  Mutation score = 71% Test suite: • Weak oracle • Missing input long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } @Test factorialWith5Test() { long obs = fact(5); assertTrue(obs > 5); } @Test factorialWith0Test() { assertEqual(fact(0), 1); }
  58. 58. Example 58 n!=0 return 1+1 i < n !(i<=n) i-- result/i result+1 @Test factorialWith5Test() { assertEqual(fact(5),120); } long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } @Test factorialWith5Test() { long obs = fact(5); assertTrue(obs > 5); } @Test factorialWith0Test() { assertEqual(fact(0), 1); }
  59. 59. Mutation Analysis •Tests are good if they can detect bugs •Mutation operators • Based on common faults •PIT or PITest • Open source, in active development and production ready • Integrates with major build systems • State of the art mutation testing • Extensible via plugins • Concurrent execution • Test selection 59
  60. 60. Limitations of mutation testing •Expensive computation •Huge number of mutants •Presence of equivalent mutants
  61. 61. Limitations of mutation testing •Expensive computation •Huge number of mutants •Presence of equivalent mutants  Extreme Mutation
  62. 62. Extreme mutation •Proposed in 2016 by Niedermayr et al. •Remove the body of the method •Replace by a single return •Less mutants •Most equivalent mutants can be detected 62 R. Niedermayr, E. Juergens, and S. Wagner, “Will my tests tell me if I break this code?,” in Proceedings of the International Workshop on Continuous Software Evolution and Delivery, 2016, pp. 23–29.
  63. 63. long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } Example 63 n!=0 return 1+1 i < n !(i<=n) i-- result/i result+1
  64. 64. long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } Example 64 n!=0 return 1+1 i < n !(i<=n) i-- result/i result+1
  65. 65. long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } long fact(int n) { return 0; } long fact(int n) { return 1; } Example 65
  66. 66. Descartes I mutate therefore I am • A mutation engine for PIT • Implement extreme mutation • Compute code coverage & mutation score • Identify weaknesses in your tests • Find pseudo-tested methods
  67. 67. Pseudo-tested methods •A method executed by the test suite •No extreme mutation is detected •Found in well tested projects
  68. 68. class SingletonListIterator implements Iterator<Node> { ... void add() { throw new UnsupportedOperationException(); } ... } A pseudo-tested method 68 Apache Commons Collections
  69. 69. A pseudo-tested method 69 class SingletonListIterator implements Iterator<Node> { ... void add() { throw new UnsupportedOperationException(); } ... } @Test void testAdd() { SingletonListIterator it = ...; try { it.add(value); } catch(Exception ex) {} } Apache Commons Collections
  70. 70. class SingletonListIterator implements Iterator<Node> { ... void add() { throw new UnsupportedOperationException(); } ... } A pseudo-tested method 70 Pseudo-tested @Test void testAdd() { SingletonListIterator it = ...; try { it.add(value); } catch(Exception ex) {} } Apache Commons Collections
  71. 71. class SingletonListIterator implements Iterator<Node> { ... void add() { throw new UnsupportedOperationException(); } ... } A pseudo-tested method 71 No exception is thrown A fail is needed here Pseudo-tested @Test void testAdd() { SingletonListIterator it = ...; try { it.add(value); } catch(Exception ex) {} } Apache Commons Collections
  72. 72. public class VersionedSet { private long version = 0; private ArrayList elements = new ArrayList(); public void add(Object item) { if (! elements.contains(item)) { elements.add(item); incrementVersion(); } } private void incrementVersion() { version++; } } A pseudo-tested method (2) 72
  73. 73. public class VersionedSet { private long version = 0; private ArrayList elements = new ArrayList(); public void add(Object item) { if (! elements.contains(item)) { elements.add(item); incrementVersion(); } } private void incrementVersion() { version++; } } A pseudo-tested method (2) 73 @Test public void testAdd() { VersionedSet list = new VersionedSet(); list.add(1); assertEquals(list.size(), 1); }
  74. 74. public class VersionedSet { private long version = 0; private ArrayList elements = new ArrayList(); public void add(Object item) { if (! elements.contains(item)) { elements.add(item); incrementVersion(); } } private void incrementVersion() { version++; } } A pseudo-tested method (2) 74 @Test public void testAdd() { VersionedSet list = new VersionedSet(); list.add(1); assertEquals(list.size(), 1); } Pseudo-tested
  75. 75. public class VersionedSet { private long version = 0; private ArrayList elements = new ArrayList(); public void add(Object item) { if (! elements.contains(item)) { elements.add(item); incrementVersion(); } } private void incrementVersion() { version++; } } A pseudo-tested method (2) 75 @Test public void testAdd() { VersionedSet list = new VersionedSet(); list.add(1); assertEquals(list.size(), 1); } Pseudo-tested Testability issue
  76. 76. Descartes reports 76
  77. 77. Descartes reports 77
  78. 78. Descartes reports 78
  79. 79. Descartes reports 79
  80. 80. Descartes results 80
  81. 81. Unit test amplification 81 Descartes
  82. 82. Unit test amplification 82 Descartes & DSpot
  83. 83. DSpot •Automatically enhances existing JUnit test suites •Generate new assertions or new test cases 83
  84. 84. Dspot principle 84
  85. 85. x Dspot principle 85 Test criterion
  86. 86. x Dspot principle 86 Test criterion (Mutation score)
  87. 87. x Dspot principle 87 Test criterion (Mutation score)
  88. 88. x Dspot principle 88 Test criterion (Mutation score)
  89. 89. x Dspot principle 89 Test criterion (Mutation score)
  90. 90. Dspot – How does it work ? 90
  91. 91. Dspot – How does it work ? 91 Mutation Analysis ✗ ✓ ✗ ✗ ✓
  92. 92. Dspot – How does it work ? 92 Amplified Test 1 Mutation Analysis ✗ ✓ ✗ ✗ ✓ amplify
  93. 93. 93 @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); benjamin.eat(kouignAmann); assertFalse(benjamin.isHungry()); } Example Original test case
  94. 94. 94 @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); benjamin.eat(kouignAmann); assertFalse(benjamin.isHungry()); } Example Original test case Remove a method call
  95. 95. 95 @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); assertFalse(benjamin.isHungry()); } Example Original test case Remove existing assertions Remove a method call
  96. 96. 96 @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); benjamin.isHungry(); Log.log(benjamin, id : "benjamin"); } Example Original test case Instrument the test
  97. 97. 97 @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); benjamin.isHungry(); Log.log(benjamin, id : "benjamin"); } Example Original test case Instrument the test Run the instrumented test
  98. 98. benjamin.isHungry()true benjamin.isHappy()false 98 @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); benjamin.isHungry(); Log.log(benjamin, id : "benjamin"); } Example Original test case Instrument the test Run the instrumented test
  99. 99. benjamin.isHungry()true benjamin.isHappy()false 99 @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); benjamin.isHungry(); Log.log(benjamin, id : "benjamin"); } Example Original test case Instrument the test Run the instrumented test assertTrue(benjamin.isHungry()); assertFalse(benjamin.isHappy());Generate assertions
  100. 100. @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); assertTrue(benjamin.isHungry()); assertFalse(benjamin.isHappy()); } 100 Example Original test case Amplified test case @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); benjamin.eat(kouignAmann); assertFalse(benjamin.isHungry()); }
  101. 101. Dspot – How does it work ? 101 Amplified Test 1 Mutation Analysis ✗ ✓ ✗ ✗ ✓ amplify
  102. 102. Dspot – How does it work ? 102 Amplified Test 1 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✓ amplify
  103. 103. Dspot – How does it work ? 103 Amplified Test 1 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✓ amplify
  104. 104. Dspot – How does it work ? 104 Amplified Test 1 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✓ amplify
  105. 105. Dspot – How does it work ? 105 Amplified Test 1 Amplified Test 2 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✓ amplify
  106. 106. Dspot – How does it work ? 106 Amplified Test 1 Amplified Test 2 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ amplify
  107. 107. Dspot – How does it work ? 107 Amplified Test 1 Amplified Test 2 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ amplify
  108. 108. Dspot – How does it work ? 108 Amplified Test 1 Amplified Test 2 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ amplify
  109. 109. Dspot – How does it work ? 109 Amplified Test 1 Amplified Test 2 Amplified Test 3 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ amplify
  110. 110. Dspot – How does it work ? 110 Amplified Test 1 Amplified Test 2 Amplified Test 3 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ amplify
  111. 111. Dspot – How does it work ? 111 Amplified Test 1 Amplified Test 2 Amplified Test 3 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ amplify
  112. 112. Dspot – How does it work ? 112 Amplified Test 1 Amplified Test 2 Amplified Test 3 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ amplify
  113. 113. Dspot – How does it work ? 113 Amplified Test 1 Amplified Test 2 Amplified Test 3 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ amplify
  114. 114. 114 DSpot Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects. B. Danglot, O. Luis Vera-Pérez, B. Baudry, M. Monperrus. Submitted to EMSE.
  115. 115. The pull request loop Collaborative platform 115
  116. 116. The pull request loop Collaborative platformpull req. 116
  117. 117. The pull request loop Collaborative platformpull req. code 117
  118. 118. The pull request loop Collaborative platformpull req. code analysesfeedback 118
  119. 119. The pull request loop Collaborative platformpull req. code analysesfeedback 119
  120. 120. The pull request loop pull req. code analysesfeedback 120
  121. 121. The pull request loop Collaborative platformpull req. code analysesfeedback 121
  122. 122. The pull request loop Collaborative platformpull req. code analysesfeedback 122 Novel analyses
  123. 123. The pull request loop Collaborative platformpull req. code analysesfeedback 123 Novel analyses
  124. 124. Integration •Command line •Eclipse plugins •Maven plugins •Gradle plugins
  125. 125. The pull request loop Collaborative platformpull req. code analysesfeedback 125 Novel analyses
  126. 126. The pull request loop Collaborative platformpull req. code analysesfeedback 126 Novel analyses
  127. 127. Integration •Jenkins • Plugin to monitor score and pseudo-tested methods • Xwiki • Same strategy used with the code coverage • Threshold on the mutation score • https://massol.myxwiki.org/xwiki/bin/view/Blog/MutationTestingDescar tes
  128. 128. The pull request loop Collaborative platformpull req. code analysesfeedback 128 Novel analyses
  129. 129. The pull request loop Collaborative platformpull req. code analysesfeedback 129 Novel analyses
  130. 130. Integration •In the CI as a service • GitHub Application • Find pseudo-tested methods in pull-requests • Feedback using GitHub Check Runs API
  131. 131. More •Open Source tools • https://github.com/STAMP-project •Public website • https://www.stamp-project.eu •Medium on Descartes • https://medium.com/@almyre/short-circuiting-method- executions-to-assess-test-quality-2d3fda45bc7f •Publications • https://www.stamp-project.eu/view/main/publications 131
  132. 132. Contacts https://github.com/STAMP-project/ http://stamp-project.eu/ baudry@kth.se caroline.landry@inria.fr http://www.diverse-team.fr/ barais@irisa.fr 132
  133. 133. 133

×