Keynote VST2020 (Workshop on Validation, Analysis and Evolution of Software Tests)

1. Universiteit Antwerpen "Next Level" Test Automation Keynote VST 2020 3rd Workshop on Validation, Analysis and Evolution of Software Tests February 2020 © Prof. Serge Demeyer

2. Books • 2 books • 3 proceedings (editor) Best Teacher’s Award Top Publications Spin Off & Start Up

3. Ericsson, Bombardier, Saab, System Verification, Empear, Verifyter, KTH, MDH, RISE Comiq, EfiCode, Ponsse, Siili, Qentinel, Symbio, Uni.Oulu, VTT Axini, Testwerk, TNO, Open Uni. AKKA, Expleo, EKS, FFT, Fraunhofer, IFAK, OFFIS, Parasoft Alerion, Prodevelop,, Uni.Mandragon Kuveyt Bank, Saha BT The TESTOMAT project will allow software teams to increase the development speed without sacrificing quality To achieve this goal, the project will advance the state-of-the-art in test automation for software teams moving towards a more agile development process.

4. Industry 4.0 Internet of Things

5. variability "lot size 1" safety IEC 62061 (mechatronics) RTCA/DO-178C (avionics) IEC 62304 (medical) ISO 26262 (automotive) agility rapid customer feedback faster release cycles

6. Six decades into the computer revolution, four decades since the invention of the microprocessor, and two decades into the rise of the modern Internet, all of the technology required to transform industries through software ﬁnally works and can be widely delivered at global scale.

7. Software Testing is the process of executing a program or system with the intent of ﬁnding errors. (Myers, Glenford J., The art of software testing. Wiley, 1979)

8. "Next Level" Test Automation (VST Keynote) Context 11 Continuous Integration Continuous Delivery Continuous Deployment DevOps Tesla “over-the-air” updates ± once every month Amazon deploys to production ± every 11,6 seconds September 2015 Amazon Web Services suffered major disruption. NetFlix recovers quickly! (Chaos Monkey)

9. "Next Level" Test Automation (VST Keynote) Lean Manufacturing 12 Eliminate Waste (e.g. Spaghetti Diagrams) © Christoph Roser on AllAboutLean.com By Dr Ian Mitchell - Own work© Smooth Flow (e.g. KanBan Boards)

10. "Next Level" Test Automation (VST Keynote) Integration Hell 13 Requirement  Collection Analysis Design Implementation Testing Deployment N otsm ooth Lots ofW aste

11. "Next Level" Test Automation (VST Keynote) Continuous Integration Pipeline 14 <<Breaking the Build>> version control build developer tests deploy scenario tests deploy to production measure & validate

12. [Khom2014] Khomh, F. Adams, B, Dhaliwal, T and Zou, Y Understanding the Impact of Rapid Releases on Software Quality: The Case of Firefox, Empirical Software Engineering, Springer. http://link.springer.com/article/10.1007/s10664-014-9308-x 1.0 1.5 2.0 3.0 3.5 3.6 4.0 5.0 7.0 8.0 9.0 Traditional Release Cycle Rapid Release Cycle (a) Time Line of Major Versions of FireFox (b) Time Line of Minor Versions of FireFox Figure 1. Timeline of FireFox versions. channels are respectively 100,000 for NIGHTLY, 1 million for AURORA, 10 million for BETA and 100+ millions for a major Firefox version [11]. NIGHTLY reaches Firefox developers and contributors, while other channels (i.e., AU- RORA and BETA) recruit external users for testing. The source code on AURORA is tested by web developers who are interested in the latest standards, and by Firefox add-on developers who are willing to experiment with new browser APIs. The BETA channel is tested by Firefox’s regular beta by bug triaging developers and assigned for fixing. When a developer fixes a bug, he typically submits a patch to Bugzilla. Once approved, the patch code is integrated into the source code of Firefox on the corresponding channel and migrated through the other channels for release. Bugs that take too long to get fixed and hence miss a scheduled release are picked up by the next release’s channel. III. STUDY DESIGN

13. [Khom2014] Khomh, F. Adams, B, Dhaliwal, T and Zou, Y Understanding the Impact of Rapid Releases on Software Quality: The Case of Firefox, Empirical Software Engineering, Springer. http://link.springer.com/article/10.1007/s10664-014-9308-x ✓ bugs are fixed faster (but … harder bugs propagated to later releases) ✓ amount of pre- & post-release bugs ± the same ✓ the program crashes earlier (perhaps due to recent features) 3.6 4.0 5.0 7.0 8.0 9.0 Rapid Release Cycle rs and assigned for fixing. When he typically submits a patch to the patch code is integrated into on the corresponding channel and er channels for release. Bugs that nd hence miss a scheduled release release’s channel. TUDY DESIGN earch questions: ease cycle affect the erence in the number control for the time lease dates. However, tly lower for versions les, i.e., failures seem cycle affect the fixing ter for versions devel- e cycle affect software d release model are .e., the proportion of ersions that possibly 5.0 NIGHTLY 6.0 NIGHTLY 7.0 NIGHTLY 8.0 NIGHTLY 5.0 AURORA 6.0 AURORA 7.0 AURORA 5.0 BETA 6.0 BETA 5.0 MAIN New Feature Development 6 Weeks 6 Weeks 6 Weeks 6 Weeks Figure 2. Development and Release Process of Mozilla Firefox major release was made. Figure 1(b) shows the release dates of the minor versions of Firefox. With the advent of shorter release cycles in March 2011, new features need to be tested and delivered to users faster. To achieve this goal, Firefox changed its development process. First, versions are no longer supported in parallel, i.e.,

14. "Next Level" Test Automation (VST Keynote) Plan • Context • Test Strategy: When Should I Test? - V-Model - Fit tables (Acceptance Testing) - Flipping the V - 4 Quadrants • Test Quality: How Good Are Your Tests? • Test Analytics: Exploiting the DevOps pipeline. • Test Research: Tips for PhD Students 17

15. "Next Level" Test Automation (VST Keynote) V-Model 18 Requirements Acceptance Tests Design Integration Tests Coding Unit Tests Architecture System Tests Requirements Architecture Design Coding Testing Test Design Test Execution Acceptance Test Cases System Test Cases Integration Test Cases Unit Test Cases Unit Tests Integration Tests System Tests Acceptance Tests Integration hell?

16. "Next Level" Test Automation (VST Keynote) Fit Tables 19 Browse Music Play Music Browse Music start eg.music.browser enter library check total songs 37 Browse Music enter select 1 check title Akila check artist Toure Kunda enter select 2 check title American Tango check artist Weather Report check album Mysterious Traveller check year 1974 Example: Acceptance Test Cases http://ﬁt.c2.com Play Music start eg.music.Realtime press play check status loading pause 2 check status playing

17. "Next Level" Test Automation (VST Keynote) Test Case Management 20 Smoke Test

18. "Next Level" Test Automation (VST Keynote) Scrum — Feedback Loop 21 Product Backlog Sprint Backlog Sprint Execution Working Increment of Product 24h Sprint Planning Sprint Review Sprint Retrospective

19. "Next Level" Test Automation (VST Keynote) Behaviour Driven (User Stories) 22 As a <user role> I want to <goal> so that <beneﬁt>. • … • … • … As a clerk I want to calculate stampage so that goods get shipped fast. • Verify with nearby address • Verify with overseas address • Verify with parcels <= 1kg • Verify with fragile parcel Template Example Conditions of Satisfaction

20. "Next Level" Test Automation (VST Keynote) Flipping the V 23 Acceptance Tests (GUI Tests) System Tests Integration Tests Unit Tests 70% 20% 10% 10% 20% 70% Test Automation

21. "Next Level" Test Automation (VST Keynote) Flipping the V in Practice 24 Acceptance Tests (GUI Tests) System Tests Integration Tests Unit Tests

22. SupportingTeam Functional Tests Examples Story Tests Prototypes Simulation Exploratory Testing Scenarios Usability Testing Acceptance Testing Alpha / Beta Unit Tests Integration Tests Performance Testing Load Testing Security Testing “ility” Testing Technology Facing Business Facing CritiqueProduct Automated & Manual Manual Automated Tools

23. unit tests?

24. "Next Level" Test Automation (VST Keynote) Plan • Context • Test Strategy: When Should I Test? • Test Quality: How Good Are Your Tests? - Coverage (Control flow, Statement, Branch, Path) - Tests vs. Faults * Reach, Infect, Propagate, Reveal - Mutation Analysis - Mutation Operators - Case studies * Scaling up: the cloud • Test Analytics: Exploiting the DevOps pipeline. • Test Research: Tips for PhD Students 27

25. How good are your tests? version control build developer tests deploy scenario tests deploy to production measure & validate

26. 90,000 test executions/day

27. c++ java

28. "Next Level" Test Automation (VST Keynote) Code Coverage 34 Test System Under Test (SUT) Code is instrumented Test executes the code Instrumented code gets executed The traces are stored: • When statement is executed - CFG (Control Flow Graph) • When data is changed - DFG (Data Flow Graph)

29. import org.junit.Test; import static org.junit.Assert.assertEquals; public class TestEmployeeDetails { EmpBusinessLogic empBusinessLogic = new EmpBusinessLogic(); EmployeeDetails employee = new EmployeeDetails(); //happy day scenario for calculation of appraisal and salary @Test public void testCalculateAppriasal() { employee.setName("Rajeev"); employee.setAge(25); employee.setMonthlySalary(8000); double appraisal = empBusinessLogic.calculateAppraisal(employee); double salary = empBusinessLogic.calculateYearlySalary(employee); } } assertionless test

30. "Next Level" Test Automation (VST Keynote) The RIPR Model 36 Reach Test Oracle Strategies for Model-Based Testing. IEEE Transactions on Software Engineering 43(4):1-1 · January 2016 Infect Propagate Reveal

31. /** * Find last index of element * @param x array to search * @param y element to look for * @return last index of y in x, if absent -1 * @throws NullPointerException if x is null */ public static int findLast(int [] x, int y) { for (int i=x.length-1; i>0; i--) if (x[i] == y) return i; return -1; }

33. /** * Find last index of element * @param x array to search * @param y element to look for * @return last index of y in x, if absent -1 * @throws NullPointerException if x is null */ public static int findLast(int [] x, int y) { for (int i=x.length-1; i>0; i--) if (x[i] == y) return i; return -1; } x = null; y = 5

35. /** * Find last index of element * @param x array to search * @param y element to look for * @return last index of y in x, if absent -1 * @throws NullPointerException if x is null */ public static int findLast(int [] x, int y) { for (int i=x.length-1; i>0; i--) if (x[i] == y) return i; return -1; } x = [2,3,5]; y = 3

41. "Next Level" Test Automation (VST Keynote) Mutation Testing 47 State of the Art State of the Practice Number of mutation testing publications per year

42. int method(int v1, int v2) { if (v1 <v2) return 1; return -1; } int method(int v1, int v2) { if (v1 >=v2) return 1; return -1; }

43. Operator Description Example Before After CBM Mutates the boundary conditions a > b a >= b IM Mutates increment operators a++ a−− INM Inverts negation operator −a a MM Mutates arithmetic & logical operators a & b a | b NCM Negates a conditional operator a == b a != b RVM Mutates the return value of a function return true return false VMCM Removes a void method call voidCall(x) – Competent Programmer Hypothesis (Program is close to correct) Coupling Effect (Test suites capable of detecting simple errors  will also detect complex errors)

44. "Next Level" Test Automation (VST Keynote) Industrial Case Study 50 • 83K lines of code • Complicated structure • Lots of legacy code • Lots of black-box tests Ali Parsai, Serge Demeyer; “Comparing Mutation Coverage Against Branch Coverage in an Industrial Setting”. Software Tools for Technology Transfer

45. http://littledarwin.parsai.net

46. "Next Level" Test Automation (VST Keynote) Industrial Case 52 Unit tests only ! Segmentation Percentage 020406080100 Mutation Coverage Branch Coverage

47. CI D evelop Build Test W ay too slow We witnessed 48 hours of mutation testing time on a test suite comprising 272 unit tests and 5,258 lines of test code for testing a project with 48,873 lines of production code. Sten Vercammen, Serge Demeyer, Markus Borg, and Sigrid Eldh; “Speeding up Mutation Testing via the Cloud: Lessons Learned for Further Optimisations”. Proceedings ESEM 2018

48. Master 1) Initial test Build 2) ∀ files to mutate: queue file names 3a) Generate mutants 4a) Execute mutants 3b) Store mutants 4b) Store results 3c) Queue mutant references 5) Process results

49. 0 1h 2h 3h LittleDarwin 1 worker 2 workers 4 workers 8 workers 16 workers 0 12h 1d 1d 12h 2d 2d 12h LittleDarwin 1 worker 2 workers 4 workers 8 workers 16 workers

50. https://github.com/joakim-brannstrom/dextool

51. Presence of defect + reach the defect + infect the program state + observable on output coverage mutants Mutation Testing = Actionable !

52. "Next Level" Test Automation (VST Keynote) Mutation Testing @ google 58 version control build developer tests deploy scenario tests deploy to production measure & validate DevOps TriCoder CodeCritique

53. "Next Level" Test Automation (VST Keynote) Mutation Testing @ google 59 MutationTesting Reported beneﬁts • stronger tests, • more effective debugging, • prevention of bugs, • improved code quality. An Industrial Application of Mutation Testing: Lessons, Challenges, and Research Directions, Goran Petrovic ́ Marko Ivankovic, Bob Kurtz Paul Ammann, René Just. ICST Proceedings

54. "Next Level" Test Automation (VST Keynote) Plan • Context • Test Strategy: When Should I Test? • Test Quality: How Good Are Your Tests? • Test Analytics: Exploiting the DevOps pipeline - Test Automation Maturity Model (TAIM) - (Spectrum Based) Fault Localisation - Test Amplification - Bug Reports - Stack Overflow • Test Research: Tips for PhD Students 60

55. version control build developer tests deploy scenario tests deploy to production measure & validate Artiﬁcial Intelligence Inside

57. Henri Heiskanen, Mika Maunumaa, and Mika Katara. A test process improvement model for automated test generation. In Oscar Dieste, Andreas Jedlitschka, and Natalia Juristo, editors, Product-Focused Software Process Improvement: 13th International Conference, PROFES 2012, Madrid, Spain, June 13-15, 2012 Proceedings, pages 17--31, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg. Sigrid Eldh, Kenneth Andersson, Andreas Ermedahl, and Kristian Wiklund. Towards a test automation improvement model (TAIM). In Proceedings of the 2014 IEEE International Conference on Software Testing, Verification, and Validation Workshops, ICSTW '14, pages 337--342, Washington, DC, USA, 2014. IEEE Computer Society. Ana Paula C. C. Furtado, Silvio R. L. Meira, and Marcos Wanderley Gomes. Towards a maturity model in software testing automation. In Proceedings of ICSEA 2014 : The Ninth International Conference on Software Engineering Advances, ICSEA2014, pages 282 - 285. IARIA, 2014. Sigrid Eldh. Test Automation Improvement Model — TAIM 2.0. In Proceedinsg NEXTA 2020 (Workshop on Next Level Test Automation) Test Process Improvement for Automated Test Generation Version: 1.01.51 (2010‐05‐06) [Draft] TPI is an evaluation framework to evaluate thematurity of a software testing process. We have modified the original TPI framework to evaluate test processes that areusing Automated Test Generation methodsand tools. Henri Heiskanen, Mika Maunumaa, Mika Katara 10.2.2010 Tampere University of Technology, Department of Software Systems

58. Fault Localisation: Debugger

59. Spectrum Based Fault Localisation wastedeffort 7 46 56 122 48 2 104 124 74 6 109 9 83 68 116 90 86 27 32 97 128 16 115 81 52 57 60 42 62 63 69 75 23 26 88 65 77 84 73 1 133 51 113 76 117 53 61 38 34 105 29 72 78 96 43 103 33 4 87 49 30 85 15 112 25 118 125 82 20 110 132 39 45 89 71 10 114 5 24 64 35 54 40 3 127 70 28 120 98 121 37 19 100 11 99 108 14 17 131 50 95 66 58 41 80 92 93 47 44 91 67 8 126 31 36 79 55 13 21 22 106 119 123 130 102 111 129 12 18 101 94 107 59 7 46 56 122 48 2 104 124 74 6 109 9 83 68 116 90 86 27 32 97 128 16 115 81 52 57 60 42 62 63 69 75 23 26 88 65 77 84 73 1 133 51 113 76 117 53 61 38 34 105 29 72 78 96 43 103 33 4 87 49 30 85 15 112 25 118 125 82 20 110 132 39 45 89 71 10 114 5 24 64 35 54 40 3 127 70 28 120 98 121 37 19 100 11 99 108 14 17 131 50 95 66 58 41 80 92 93 47 44 91 67 8 126 31 36 79 55 13 21 22 106 119 123 130 102 111 129 12 18 101 94 107 59 11513515517519511151140116511901 11513515517519511151140116511901 Patterned Spectrum Analysis Raw Spectrum Analysis wastedeffort 3 15 2 8 9 23 1 14 17 5 18 16 4 7 26 21 25 22 6 13 24 10 12 19 20 27 3 15 2 8 9 23 1 14 17 5 18 16 4 7 26 21 25 22 6 13 24 10 12 19 20 27 151101151201251301351401451 151101151201251301351401451 Patterned Spectrum Analysis Raw Spectrum Analysis Artiﬁcial Intelligence Inside

60. unit tests?

61. "Next Level" Test Automation (VST Keynote) Test Ampliﬁcation 68 Benjamin Danglot, Oscar Vera-Pérez, Benoit Baudry, Martin Monperrus. Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects. Empirical Software Engineering, Springer Verlag, 2019, pp.1-35. 10.1007/s10664-019-09692-y . Mehrdad Abdi, Henrique Rocha and Serge Demeyer. Test Ampliﬁcation in the Pharo Smalltalk Ecosystem. Proceedings IWST 2019 (International Workshop on Smalltalk Technologies) input generation + assertion generation testWithdraw |b| b := SmallBank new.  b deposit: 100.  self assert: b balance equals: 100. b withdraw: 30.  self assert: b balance equals: 70 Genetic Algorithms Inside testWithdraw_12 | b | b := SmallBank new. b deposit: 100. b withdraw: SmallInteger maxVal. self assert: b balance equals: 100

62. Description text Mining Stack Traces Link to source code Product/Component Speciﬁc vocabulary Suggestions? Artiﬁcial Intelligence Inside

63. Question Cases Precision Recall Who should fix this bug? Eclipse, Firefox, gcc eclipse: 57% firefox: 64% gcc: 6% — How long will it take to fix this bug? (*) JBoss depends on the component many similar reports: off by one hour few similar reports: off by 7 hours What is the severity of this bug? (**) Mozilla, Eclipse, Gnome mozilla, eclipse:67% - 73% gnome: 75%-82% mozilla, eclipse:50% - 75% gnome: 68%-84% Promising results but … • how much training is needed? (cross-project training?) • how reliable is the data? (estimates, severity, assigned-to) • does this generalise? (on industrial scale?) replication is needed (*) In CSMR2012 Proceedings Who should fix this bug? Eclipse, Firefox, gcc eclipse: 57% firefox: 64% gcc: 6% — Irrelevant for Practitioners (**) In CSMR2011; MSR 2010 Proceedings Artificial Intelligence Inside

64. "Next Level" Test Automation (VST Keynote) Q&A support 71

65. In CHI2016 Proceedings Artiﬁcial Intelligence Inside

66. "Next Level" Test Automation (VST Keynote) Plan • Context • Test Strategy: When Should I Test? • Test Quality: How Good Are Your Tests? • Test Analytics: Exploiting the DevOps pipeline. • Test Research: Tips for PhD Students - Descriptive Statistics - Unit Tests vs. Integration Tests 73

67. "Next Level" Test Automation (VST Keynote) Descriptive Statistics 74 Semi-automatic Test Case Expansion for Mutation Testing ZhongXiLu,StenVercammen,andSergeDemeyer—UniversityofAntwerp,Belgium . . . . . . . . . . . . . . . . . . . . . . 1 An Early Investigation of Unit Testing Practices of Component-Based Software Systems Georg Buchgeher, Stefan Fischer, Michael Moser, and Josef Pichler — Software Competence Center Hagenberg, Austria; University ofAppliedSciencesUpperAustria,Austria ............................................ 12

68. "Next Level" Test Automation (VST Keynote) Test Code Quality 75 At minimum include some code coverage measurement + assertion density.

69. "Next Level" Test Automation (VST Keynote) Unit Tests vs. Integration Tests (1/2) 76 Further analysis conﬁrmed that it is worthwhile to treat unit tests and integration tests differently: we discovered that unit tests cause more breaking builds, that ﬁxing the defects exposed by unit tests takes longer and implies more coordination between team members.

70. "Next Level" Test Automation (VST Keynote) Unit Tests vs. Integration Tests (2/2) 77 Hence, if we apply the IEEE deﬁnition to Java a unit test is a test that tests only units from within one package (i.e., related units). On the other hand, an integration test tests units from more than one package.

71. version control build developer tests deploy scenario tests deploy to production measure & validate Artiﬁcial Intelligence Inside If your AI system cannot motivate its decision practitioners will not accept it.

Keynote VST2020 (Workshop on Validation, Analysis and Evolution of Software Tests)

University of Antwerp

Keynote VST2020 (Workshop on Validation, Analysis and Evolution of Software Tests)