This document discusses automated mobile UI testing fragility in Android applications. It explores three ways of testing Android UIs: manual tests, automated unit tests, and automated instrumented tests. Challenges of Android UI testing include different device configurations and dynamic application swapping. GUI automated tests are considered fragile if they fail due to interface changes rather than functionality changes. The study examines an open-source Android app across 336 releases to analyze how many tests need modification for new releases and the causes of fragility. Up to 75% of code-based and 100% of image-based tests required adaptation. Seven fragility causes were characterized, with all causing failures. Future work areas include better characterizing and preventing fragility.

1. Automated Mobile UI Testing Fragility
An Exploratory Assessment Study on Android
Riccardo Coppola, Emanuele Raffero, Marco Torchiano
Politecnico di Torino, Dipartimento di Automatica e Informatica

2. Android UI Testing
Three ways of testing Android User
Interfaces:
• Manual tests;
• Automated Unit Tests;
• Automated Instrumented Tests.

3. Android UI Testing - Challenges
• Different configurations of display sizes, layouts, pixel
densities;
• Different input modalities;
• Dynamic swapping of applications;
• Fast evolution of the operative system, and therefore of
the app features.

4. Android UI Testing - Fragility
A GUI automated test is said to be fragile when:
• It fails when the application evolves;
• The failure is not due to the modification of the
functionalities under test, but to changes in the interface
arrangement and definition.
When a fragility occurs, extra effort is needed to:
• Verify that no regression has occurred;
• Modify the failing tests to adapt them to the changed UI.

5. Research questions
RQ1 How many tests have to be modified to adapt to
subsequent releases of the same app?
RQ2 Which are the intrinsic causes of the fragilities of UI
tests?

6. Selected Application
• Open-source project;
• 120k LOC;
• 336 releases;
• Stable graphic interface.

7. Testing Tools: Code-Based
Use of ids and
content to identify
UI elements.

8. Testing Tools: Image Recognition
Use of captures to
identify UI elements.

9. Test Suite
Successful Authentication
Send message
Reply to a message
Delete a message
Add user account
Delete user account
Delete account data
Restore account data
Export account settings
Import account settings
v2.102
v2.995
v3.309
v3.993
v4.804
v5.010
6 releases
10 features

10. Code-based test suites: Evolution
Test suite evolution
Tests
not failed

11. Image-recognition test suites: Evolution
Test suite evolution
Tests
not failed

12. Fragilities Classification
• Identifier Change;
• Text Change;

13. Fragilities Classification
• Deletion or Relocation;
• Physical Buttons use;

14. • Graphics Change;
• Activity Flow Change;
• Execution Time Variability.
Fragilities Classification

15. Fragilities Occurrence
Causes of fragilities in code-based failed test cases:
Cause v2.995 v3.993 v4.804
Text Change 0/2 2/2 3/6
Identifier Change 0/2 0/2 3/6
Deletion or Relocation 2/2 0/2 3/6
Physical Buttons use 0/2 0/2 3/6

16. Findings
RQ1 How many tests have to be modified to adapt to
subsequent releases of the same app?
Up to 75% of code-based test scripts, and up to
100% of image recognition test scripts, had to be
adapted.

17. Findings
RQ2 Which are the intrinsic causes of the fragilities of UI
tests?
Seven potential causes of fragility have been
characterized. All of them caused fragilities in the test
suite developed.

18. Future Work
• Test Fragility Characterization:
– Survey of existing tools;
– Frequency of tool adoption;
– How much and often tests are changed;
– Classification of fragility causes;
– Frequency of occurrence of specific causes.

19. Future Work
• Guidelines definition:
– Set of DOs and DONTs;
• Test Code Analysis;
• Test Code Refactoring:
– Set of refactoring recipes;
– Plug-in for an IDE.

20. Questions?

21. Thank You