Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to Increase Reliability

1. Finding Bugs, Fixing Bugs, Preventing Bugs   Exploiting Automated Tests to Increase Reliability March 2022 — © Prof. Serge Demeyer

2. in vitro in vivo

3. Ericsson, Bombardier, Saab, System Veri fi cation, Empear, Verifyter, KTH, MDH, RISE Comiq, E fi Code, Ponsse, Siili, Qentinel, Symbio, Uni.Oulu, VTT Axini, Testwerk, TNO, Open Uni. AKKA, Expleo, EKS, FFT, Fraunhofer, IFAK, OFFIS, Parasoft Alerion, Prodevelop,, Uni.Mandragon Kuveyt Bank, Saha BT The TESTOMAT project will allow software teams to increase the development speed without sacri fi cing quality. To achieve this goal, the project will advance the state-of-the-art in test automation for software teams moving towards a more agile development process.

4. Six decades into the computer revolution, four decades since the invention of the microprocessor, and two decades into the rise of the modern Internet, all of the technology required to transform industries through software fi nally works and can be widely delivered at global scale.

6. © Serge Demeyer Continuous Integration Pipeline 10 <<Breaking the Build>> version control build developer tests deploy scenario tests deploy to production measure & validate

8. c++ java

9. © Serge Demeyer Test Suite 13 TEST(FindLastTests, emptyVector) {  EXPECT_EQ(-1, findLast({}, 3));  } TEST(FindLastTests, doubleOccurrence) {  EXPECT_EQ(3, findLast({1, 2, 42, 42, 63}, 42));  } TEST(FindLastTests, noOccurrence) { EXPECT_EQ(-1, findLast({1, 2, 42, 42, 63}, 99)); int findLast(std::vector<int> x, int y) {  if (x.size() == 0)  return -1;  for (int i = x.size() - 1; i >= 0; i--)  if (x[i] == y)  return i;  return -1;  }

10. 100% line coverage 100% statement coverage 100% branch coverage all tests passed

11. © Serge Demeyer Inject Mutant (Killed) 15 01 int findLast(std::vector<int> x, int y) {  02 if (x.size() == 0)  03 return -1;  04 for (int i = x.size() - 1; i < 0; i--)  05 if (x[i] == y)  06 return i;  07 return -1;  08 } [ PASSED ] 2 tests. [ FAILED ] 1 test, listed below: [ FAILED ] FindLastTests.doubleOccurrence Relational Operator Replacement (ROR) “i >= 0” becomes “i < 0”

12. © Serge Demeyer Inject Mutant (Survived - Live) 16 01 int findLast(std::vector<int> x, int y) {  02 if (x.size() == 0)  03 return -1;  04 for (int i = x.size() - 1; i > 0; i--)  05 if (x[i] == y)  06 return i;  07 return -1;  08 } [==========] 3 tests from 1 test suite ran. (0 ms total) [ PASSED ] 3 tests. Relational Operator Replacement (ROR) “i >= 0” becomes “i > 0”

13. © Serge Demeyer TEST(FindLastTests, occurrenceOnBoundary) {  EXPECT_EQ(0, findLast({1, 2, 42, 42, 63}, 1));  } [==========] 4 tests from 1 test suite ran. (0 ms total) [ PASSED ] 3 tests. [ FAILED ] 1 test, listed below: [ FAILED ] FindLastTests.occurrenceOnBoundary Strengthening the Test Suite 17 01 int findLast(std::vector<int> x, int y) {  02 if (x.size() == 0)  03 return -1;  04 for (int i = x.size() - 1; i > 0; i--)  05 if (x[i] == y)  06 return i;  07 return -1;  08 }

14. © Serge Demeyer Industrial Case Study 18 • 83K lines of code • Complicated structure • Lots of legacy code • Lots of black-box tests Ali Parsai, Serge Demeyer; “Comparing Mutation Coverage Against Branch Coverage in an Industrial Setting”. Software Tools for Technology Transfer

15. Unit tests only ! Segmentation Percentage 0 20 40 60 80 100 Mutation Coverage Branch Coverage

16. CI D evelop Bu ild Test W ay too slow We witnessed 48 hours of mutation testing time on a test suite comprising 272 unit tests and 5,258 lines of test code for testing a project with 48,873 lines of production code. Sten Vercammen, Serge Demeyer, Markus Borg, and Sigrid Eldh; “Speeding up Mutation Testing via the Cloud: Lessons Learned for Further Optimisations”. Proceedings ESEM 2018

17. Master 1) Initial test Build 2) ∀ files to mutate:   queue file names 3a) Generate mutants 4a) Execute mutants 3b) Store mutants 4b) Store results 3c) Queue mutant   references 5) Process results

18. 0 1h 2h 3h LittleDarwin 1 worker 2 workers 4 workers 8 workers 16 workers 0 12h 1d 1d 12h 2d 2d 12h LittleDarwin 1 worker 2 workers 4 workers 8 workers 16 workers

19. https://github.com/joakim-brannstrom/dextool

20. © Serge Demeyer DO-178C = Avionics Safety Standard 24 Mutation Testing = Actionable !

22. © Serge Demeyer 4 Quadrants 26 Supporting Team Functional Tests Examples Story Tests Prototypes Simulation Exploratory Testing Scenarios Usability Testing Acceptance Testing Alpha / Beta Unit Tests Integration Tests Performance Testing Load Testing Security Testing “ility” Testing Technology Facing Business Facing Critique Product Automated & Manual Manual Autom ated Tools

23. © Serge Demeyer Flipping the V 27 70% 20% 10% 10% 20% 70% Acceptance Tests (GUI Tests) System Tests Integration Tests Component Tests Unit Tests Manual Automated

24. © Serge Demeyer Flipping the V? 28

25. © Serge Demeyer Flakey Tests 29 version control build developer tests deploy scenario tests deploy to production measure & validate Non-determistic Result

26. © Serge Demeyer Flakey Tests: Taxonomy 30 CAUSES Avoidable Async Wait Concurrency I/O Memory Arithmetic Randomness Collections Inevitable Time Test Order External Dependencies

27. © Serge Demeyer Flakey Tests: Management 31 Zero Tolerance Manage

29. Advance the state-of-the-art in autonomous (= zero-touch) testing for DevOps software teams. Arti fi cial Intelligence Inside

30. © Serge Demeyer Testing 34 Program   Under Test Input Expected output Software Testing is the process of executing a program or system with the intent of finding errors. (Myers, Glenford J., The art of software testing. Wiley, 1979

31. © Serge Demeyer Coverage 35 Program   Under Test Input Expected output (mutation) coverage

32. © Serge Demeyer Test Ampli fi cation 36 Program   Under Test Input Expected output (mutation) coverage +coverage Extra Input Extra Input +Extra Input +Extra output Arti fi cial Intelligence Inside

33. © Serge Demeyer Example - testDeposit 37 1 def testDeposit ( self ) :   2 self.b.set_owner(’Iwena Kroka’)   3 self.b.deposit(10)   4 self.assertEqual(self.b.get_balance(), 10)   5 self.b.deposit(100)   6 self.b.deposit(100)   7 self.assertEqual(self.b.get_balance() , 210) I n p u t Expected output

34. © Serge Demeyer Example - testDeposit_ampli fi ed (1/2) 38 1 def testDeposit_amplified ( self ) :   2 self.b.set_owner(’Iwena Kroka’)   3 self.b.deposit(10)   4 self.assertEqual(self.b. 5 get_transactions(), [10])   6 self.assertFalse (self .b. is_empty () ) 7 self.assertEqual(self.b.owner, ’Iwena Kroka’) 8 self.assertEqual(self.b.get_balance(), 10) …  Assertion Am plification

35. © Serge Demeyer Example - testDeposit_ampli fi ed (2/2) 39 1 def testDeposit_amplified ( self ) :   2 self.b.set_owner(’Iwena Kroka’)   3 self.b.deposit(10)   4 self.assertEqual(self.b. 5 get_transactions(), [10])   6 self.assertFalse (self .b. is_empty () ) 7 self.assertEqual(self.b.owner, ’Iwena Kroka’) 8 self.assertEqual(self.b.get_balance(), 10) 9 with self.assertRaises(Exception):   10 self .b. deposit(−56313) 11 self.b.deposit(100) 12 self.b.set_owner(’Guido van Rossum’)   13 self.assertEqual(self.b. 14 get_transactions(), [10, 100]) … Input Am plification

36. 11 pull requests 8 merged 3 pending

37. © Serge Demeyer Research Agenda 41 Program   Under Test (mutation) coverage +coverage Arti fi cial Intelligence Inside (Intention Revealing) Test Names Explainable Test Cases Visualisation (Dashboards) Work fl ow (Github Actions)

39. Description text Mining Stack Traces Link to source code Product/Component Speci fi c vocabulary Suggestions?

40. Question Cases Precision Recall Who should fi x this bug? Eclipse, Firefox, gcc eclipse: 57% fi refox: 64% gcc: 6% — How long will it take to fi x this bug? (*) JBoss depends on the component many similar reports: off by one hour few similar reports: off by 7 hours What is the severity of this bug? (**) Mozilla, Eclipse, Gnome mozilla, eclipse:67% - 73% gnome: 75%-82% mozilla, eclipse:50% - 75% gnome: 68%-84% (*) In CSMR2012 Proceedings Who should fi x this bug? Eclipse, Firefox, gcc eclipse: 57% fi refox: 64% gcc: 6% — Irrelevant for Practitioners (**) In CSMR2011; MSR 2010 Proceedings Arti fi cial Intelligence Inside Internal vs. External  Bug Reports

42. Human   MMRE: 0.48 (*) Mean Magnitude of Relative Error Learning Curve

46. In CHI2016 Proceedings Arti fi cial Intelligence Inside

47. © Serge Demeyer Hype Cycle 51 Hype Cycle © Gartner Visibility Maturity Technology Trigger Peak of Inflated   Expectations Trough of   Disillusionment Slope of   Enlightenment Plateau of   Productivity

48. © Serge Demeyer The Future ? 52 Personal Opinion Peak of Inflated   Expectations Hype Cycle © Gartner Visibility Maturity Technology Trigger Trough of   Disillusionment Slope of   Enlightenment Plateau of   Productivity IBM Microsoft Google FaceBook … … for an experiment nearby

Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to Increase Reliability

You are reading a preview.

Check these out next

Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to Increase Reliability

More Related Content

Similar to Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to Increase Reliability (20)

More from University of Antwerp (7)

Recently uploaded (20)