Fragility of Layout-based and Visual GUI test scripts: an assessment study on a Hybrid Mobile Application

1. Fragility of Layout-based and Visual GUI test scripts: an assessment study on a Hybrid Mobile Application Luca Ardito, Riccardo Coppola, Marco Torchiano Politecnico di Torino, Dipartimento di Automatica e Informatica

2. Different categories of mobile applications • Native • Web-Based • Hybrid 2

3. Different categories of mobile applications • Native • Web-Based • Hybrid 2

4. Mobile GUI Testing Techniques 3 • Manual • Automated

5. Mobile GUI Testing Techniques •Coordinate-Based •Layout-Based •Visual / Image Recognition-based 3 • Manual Automated

6. Mobile GUI Testing Techniques •Coordinate-Based: 1st generation •Layout-Based: 2nd generation •Visual / Image Recognition-based: 3rd generation 3 • Manual

7. Mobile GUI Testing Techniques •Coordinate-Based: 1st generation •Layout-Based: 2nd generation •Visual / Image Recognition-based: 3rd generation 3 • Manual Eye Automate

8. 48 Tests the actual appearance of the app Tests layout and hierarchy details Resilient to device diversity Resilient to graphical changes Resilient to changes in layout definition Unaffected by components with same appearance Unaffected by missing layout properties End-2-End Mobile Testing Techniques

9. GUI Testing Fragility 5 9 The very high variability of User Interfaces and its definition can lead to GUI Testing Fragility. A GUI automated test is said to be fragile when: • It fails when the application evolves; • The failure is not due to the modification of the functionalities under test, but to changes in the interface arrangement and definition.

10. GUI Testing Fragility 5 10 The very high variability of User Interfaces and its definition can lead to GUI Testing Fragility.

13. Testing issues for hybrid apps 6 13 Limited specialized literature about hybrid app testing; Limited diffusion of specialized frameworks; Software fragmentation issue; Layouts often generated automatically by development frameworks.

14. Study Design – Goal Analyze the E2E testing process to understand what technique: • Layout-based • Visual Yields • Better performance in found defects • Better resilience to fragility

15. PoliTO App, hybrid app: - developed with Apache Cordova; - Available on Play Store, App Store, Microsoft Store; - Features for both students and teachers; - 26 releases (last 1.7.2); - 40,000 users (80% on Android). Study Design – Software Object

16. Study Design – Software Object

17. RQ1 – Defects: Were test suites written with 2nd and 3rd generation tools able to spot defects in the AUT? Study Design – Research Questions and Metrics

18. RQ2 – Maintenance: What is the amount of maintenance effort needed by the Layout-based and visual test suites? Study Design – Research Questions and Metrics

19. Study Design – Research Questions and Metrics MMR (Modified Methods Ratio) = changed test methods in transition between release i-1 and i; TMV (Test Method Volatility) = Ratio of releases in which a given test method is modified; TSV (Test Suite Volatility) = Ratio of test methods modified at least one time; MRR (Modified Releases Ratio) = Ratio of releases with changes in test methods.

20. RQ3 – Fragility causes: What kind of fragilities are encountered during the evolution of a hybrid mobile app? Study Design – Research Questions and Metrics

21. Study Design: Procedure Creation of a test suite with both techniques on the first release of the application.

22. Study Design: Procedure 1.1.0 1.1.2 1.2.0 … 1.7.2 Test suite executed on all releases: On each release, metrics are computed to measure how many maintenance interventions are needed to keep tests aligned with the application.

23. Results: Spotted defects Visualization defect since release 1.6.4: empty element in drop- down menu. Graphic defect not fixed by the developers, and not spotted by the Layout-based testing tool.

24. Results: Fragilities

25. Results: Fragilities

26. Results: Fragility Causes Number of occurrences of each fragility reason

27. Examples of Pure Graphic Fragilities

28. Examples of Widget Arrangement Fragilities

29. Conclusions 50/60% methods of tests suites had to be modified at least once during the evolution of the app; Effort in maintaining test suites for hybrid apps is comparable to that for maintaining test suites for native apps; Developers of hybrid applications should always consider to provide unique identifiers for the elements of the GUI to avoid fragilities; A combined usage of both techniques should be considered a best practicers for mobile testers.

30. Next Steps Implementation and validation of a combined translation-based tool for Visual testing of (hybrid) mobile apps; Automated repair of locators of Visual or Layout-based test suites.