Paper Presentation at A-Test 2019 https://dl.acm.org/citation.cfm?id=3342824f

1. Fragility of Layout-based and Visual GUI test
scripts: an assessment study on a
Hybrid Mobile Application
Luca Ardito, Riccardo Coppola, Marco Torchiano
Politecnico di Torino, Dipartimento di Automatica e Informatica

2. Different categories of mobile applications
• Native
• Web-Based
• Hybrid
2

3. Different categories of mobile applications
• Native
• Web-Based
• Hybrid
2

4. Mobile GUI Testing Techniques
3
• Manual
• Automated

5. Mobile GUI Testing Techniques
•Coordinate-Based
•Layout-Based
•Visual / Image Recognition-based
3
• Manual
Automated

6. Mobile GUI Testing Techniques
•Coordinate-Based: 1st generation
•Layout-Based: 2nd generation
•Visual / Image Recognition-based: 3rd generation
3
• Manual

7. Mobile GUI Testing Techniques
•Coordinate-Based: 1st generation
•Layout-Based: 2nd generation
•Visual / Image Recognition-based: 3rd generation
3
• Manual
Eye Automate

8. 48
Tests the actual appearance of the app
Tests layout and hierarchy details
Resilient to device diversity
Resilient to graphical changes
Resilient to changes in layout definition
Unaffected by components with same
appearance
Unaffected by missing layout properties
End-2-End Mobile Testing Techniques

9. GUI Testing Fragility
5
9
The very high variability of User Interfaces and its definition
can lead to GUI Testing Fragility.
A GUI automated test is said to be fragile when:
• It fails when the application evolves;
• The failure is not due to the modification of the
functionalities under test, but to changes in the interface
arrangement and definition.

10. GUI Testing Fragility
5
10
The very high variability of User Interfaces and its definition
can lead to GUI Testing Fragility.

11. GUI Testing Fragility
5
11
The very high variability of User Interfaces and its definition
can lead to GUI Testing Fragility.

12. GUI Testing Fragility
5
12
The very high variability of User Interfaces and its definition
can lead to GUI Testing Fragility.

13. Testing issues for hybrid apps
6
13
Limited specialized literature about hybrid app testing;
Limited diffusion of specialized frameworks;
Software fragmentation issue;
Layouts often generated automatically by development
frameworks.

14. Study Design – Goal
Analyze the E2E testing process to understand what technique:
• Layout-based
• Visual
Yields
• Better performance in found defects
• Better resilience to fragility

15. PoliTO App, hybrid app:
- developed with Apache Cordova;
- Available on Play Store, App Store, Microsoft Store;
- Features for both students and teachers;
- 26 releases (last 1.7.2);
- 40,000 users (80% on Android).
Study Design – Software Object

16. Study Design – Software Object

17. RQ1 – Defects: Were test suites written with 2nd and 3rd
generation tools able to spot defects in the AUT?
Study Design – Research Questions and Metrics

18. RQ2 – Maintenance: What is the amount of maintenance effort
needed by the Layout-based and visual test suites?
Study Design – Research Questions and Metrics

19. Study Design – Research Questions and Metrics
MMR (Modified Methods Ratio) = changed test methods in
transition between release i-1 and i;
TMV (Test Method Volatility) = Ratio of releases in which a
given test method is modified;
TSV (Test Suite Volatility) = Ratio of test methods modified at
least one time;
MRR (Modified Releases Ratio) = Ratio of releases with
changes in test methods.

20. RQ3 – Fragility causes: What kind of fragilities are encountered
during the evolution of a hybrid mobile app?
Study Design – Research Questions and Metrics

21. Study Design: Procedure
Creation of a test suite with both techniques on the first release
of the application.

22. Study Design: Procedure
1.1.0 1.1.2 1.2.0 … 1.7.2
Test suite executed on all releases:
On each release, metrics are computed to measure how many
maintenance interventions are needed to keep tests aligned with
the application.

23. Results: Spotted defects
Visualization defect since release
1.6.4: empty element in drop-
down menu.
Graphic defect not fixed by the
developers, and not spotted by the
Layout-based testing tool.

24. Results: Fragilities

25. Results: Fragilities

26. Results: Fragility Causes
Number of occurrences of each fragility reason

27. Examples of Pure Graphic Fragilities

28. Examples of Widget Arrangement Fragilities

29. Conclusions
50/60% methods of tests suites had to be modified at least
once during the evolution of the app;
Effort in maintaining test suites for hybrid apps is comparable to
that for maintaining test suites for native apps;
Developers of hybrid applications should always consider to
provide unique identifiers for the elements of the GUI to avoid
fragilities;
A combined usage of both techniques should be considered a
best practicers for mobile testers.

30. Next Steps
Implementation and validation of a combined translation-based tool for
Visual testing of (hybrid) mobile apps;
Automated repair of locators of Visual or Layout-based test suites.