This document presents a technique for automatically determining when to stop random testing of Android applications based on the concept of saturation. The technique runs multiple random testing sessions simultaneously and stops when their code coverage converges, indicating saturation. An experiment on 18 Android apps found that the technique was effective at reaching saturation and termination points optimized coverage versus cost. Key factors influencing effectiveness included app preconditions, testing platform configuration, and unreachable code. The technique provides a fully automated and cost-effective method for random testing of Android apps.

1. MOBILESoft 2015 - 2nd ACM International Conference on Mobile Software Engineering and Syste
Exploiting the Saturation Effect in Automatic
Random Testing of Android Applications
Domenico
Amalfitano,
Nicola Amatucci,
Anna Rita Fasolino,
Porfirio Tramontana
Emily Kowalczyk,
Atif M. Memon

2. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
Context and Motivations
 Context
◦ Automatic GUI Random Testing of Mobile
Applications.
 Motivation
◦ To formalize a stopping criterion for
random testing.
MOBILESoft 2015 – Florence, Italy – May 16 2

3. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
GUI Random Testing
Techniques
 A.K.A. Monkey Fuzz Testing
Techniques.
 Usually exploited for stress testing.
 Sequences of random events are sent
to the application under test (AUT).
◦ Generally only user events (i.e. mouse or
keyboard events).
 Fully automated.
MOBILESoft 2015 – Florence, Italy – May 16 3

4. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
MFT Tools
 Tools implementing MFT techniques.
 Easy to use  growing popularity for
testing mobile applications.
4MOBILESoft 2015 – Florence, Italy – May 16
https://github.com/reverse-
unina/AndroidRipper

5. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
The Problem …
 When the execution of an MFT tool must
be stopped?
 The stop condition is:
◦ able to affect the effectiveness and the cost
of automatic testing processes.
◦ even more important with random testing
techniques, affected by problems of
reliability and efficiency of the test adequacy.
 Covered Statements Percentage
usually exploited to measure the test
adequacy of testing techniques.
CSP =
#SetCoveredStatements
#SetStatementsAUT
´100
MOBILESoft 2015 – Florence, Italy – May 16 5

6. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
Mostly Used Stop Criteria …
 Predefined testing time or number of
fired events.
0
10
20
30
40
50
60
70
80
90
100
300
500
700
900
1100
1300
1500
1700
1900
EXP1
EXP2
EXP3
Low #Events  (1)
Low Reliability, (2)
Low Effectiveness
Number of fired Events
CoveredStatementPercentage
MOBILESoft 2015 – Florence, Italy – May 16 6

7. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
… Mostly Used Stop Criteria
 Fix the testing time or the number of
fired events.
Too many Events
 Low Efficiency
Number of fired Events
40
45
50
55
60
65
70
75
80
85
90
100
700
1300
1900
2500
3100
3700
4300
4900
5500
6100
6700
7300
7900
8500
9100
9700
S1
S2
S3
7
CoveredStatementPercentage
MOBILESoft 2015 – Florence, Italy – May 16

8. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
The proposed solution …
 A testing process that automatically
stops its execution at a point
representing an optimal trade-off
between effectiveness and cost.
◦ A point were test adequacy is maximized
and no testing effort is wasted.
 It exploits two typical phenomena of
random testing.
8MOBILESoft 2015 – Florence, Italy – May 16

9. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
First phenomenon1
 The existence of a saturation effect in MFT.
◦ A well-known phenomenon where the test
adequacy tends to a specific value as the
amount of test execution increases.
◦ There is a point at which no more progress is
made towards maximizing a test adequacy
criterion.
 It is called saturation point.
◦ We propose it as the optimal termination point.
9
1- E. Sherman, M. B. Dwyer, and S. Elbaum, “Saturation-based testing of concurrent programs,” SIGSOFT ‘09
MOBILESoft 2015 – Florence, Italy – May 16

10. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
Second phenomenon2
 The predictability property of MFT
◦ The difference between the test
adequacies reached by two random test
cases tend to be definitely zero.
◦ In other words, two test cases tend to
reach the same test adequacy (i.e., same
found faults or same set of executed
LOCs )
10
2- A. Arcuri, M. Z. Iqbal, and L. Briand, “Random testing: Theoretical results and practical implications,” IEEE Trans. Softw.
Eng.
MOBILESoft 2015 – Florence, Italy – May 16

11. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
… The proposed solution
 The process exploits the definition of
predictability to reach a potential
saturation point.
 The determination of the potential
saturation point is based on the
simultaneous execution of several
random testing sessions.
 The testing process is stopped when
the difference in code coverage is
zero.
◦ When the sessions have executed the
same code. 11MOBILESoft 2015 – Florence, Italy – May 16

12. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
Graphical interpretation of the
solution
12MOBILESoft 2015 – Florence, Italy – May 16
40
45
50
55
60
65
70
75
80
85
90
100
600
1100
1600
2100
2600
3100
3600
4100
4600
5100
5600
6100
6600
7100
7600
8100
8600
9100
9600
S1
S2
S3
Saturation Effect – Saturation Point
Predictability – differences
in code coverage tend
definitely to zero
Unreliability Zone

13. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
Differences in code coverage
 Covered Statements Percentage (CSP) as test
adequacy.
 All the sessions have actually covered the same
statements of the AUT  all the sessions reach a
test adequacy value equals to the Cumulative
Coverage Statement Percentage (CSCP).
13MOBILESoft 2015 – Florence, Italy – May 16
CSP = 40%
CSP = 40%
CSCP = 70%
Sessions
have
covered
different
code

14. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
Differences in code coverage
 Covered Statements Percentage (CSP) as test
adequacy.
 All the sessions have actually covered the same
statements of the AUT  all the sessions reach a
test adequacy value equals to the Cumulative
Coverage Statement Percentage (CSCP).
14MOBILESoft 2015 – Florence, Italy – May 16
CSP = 55%
CSP = 55%
CSCP = 55%
Sessions
have
covered the
same code

15. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
The algorithm
Mobilesoft 2015 – Florence, Italy – May 16
Launch of k testing sessions
While the k sessions do not reach the same
coverage
Each session fires a new random
event
The termination condition is evaluated at
predefined sampling steps
15

16. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
TerP and TerL
 The algorithm produces two outputs:
◦ Termination Point (TerP): the minimum number of
events at which the termination condition is verified.
◦ Termination Level (TerL): the coverage statement
percentage reached by all the k testing sessions.
Mobilesoft 2015 – Florence, Italy – May 16
40
50
60
70
80
90
100
400
700
1000
1300
1600
1900
2200
2500
2800
3100
3400
3700
S1
S2
S3
UNION
(TerP = 3700, TerL = 85%)
16
Number of fired Events
CoveredStatementPercentage

17. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
Testing infrastructure
 Implemented for executing the testing process.
◦ It targets the Android mobile platform.
Mobilesoft 2015 – Florence, Italy – May 16 17

18. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
Experimentation – RQs
 Experiment aimed at answering the
following research questions:
◦ RQ 1: Is the proposed testing process
able to reach the Saturation at the
termination point?
◦ RQ 2 : Which are the main factors able to
affect the effectiveness of the proposed
testing process at the termination point?
Mobilesoft 2015 – Florence, Italy – May 16 18
Why TerL ≠ 100% ?

19. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
Experimentation - Subjects
 18 open source Android applications were
selected.
19

20. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
Experimentation - Metrics
 A set o metrics were proposed to assess the Saturation
effect at the termination point.
◦ RePRS it is the Residual Percentage of Reachable
Statements by the process at the termination point.
◦ If RePRS = 0 (or approximately equal to zero) then all
the reachable code of the application has been actually
covered.
Executed Code
Not Executed Code
20MOBILESoft 2015 – Florence, Italy – May 16

21. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
Experimentation - Metrics
 A set o metrics were proposed to assess the Saturation
effect at the termination point.
◦ RePRS it is the Residual Percentage of Reachable
Statements by the process at the termination point.
◦ If RePRS = 0 (or approximately equal to zero) then all
the reachable code of the application has been actually
covered.
Executed Code
Unreachable Code
Reachable Code
21MOBILESoft 2015 – Florence, Italy – May 16

22. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
Experimental Procedure
 AndroidRipper was exploited as MFT
tool.
◦ It was configured for:
 sending events on the GUI widgets having at least
a registered listener, for emulating the pressure of
the back and openMenu buttons and the device
rotation;
 filling in the EditText widgets with random numeric
values;
 firing consecutive events with a delay of 500 ms;
 For each AUT:
◦ a specific initial precondition was defined;
◦ k=12 testing sessions were executed.
 All the testing sessions were executed
on Android Virtual Devices (AVD) having 22MOBILESoft 2015 – Florence, Italy – May 16

23. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
Results
 To answer the RQs the code
statements left uncovered by the
testing process were manually
analyzed.
Mobilesoft 2015 – Florence, Italy – May 16 23

24. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
Discussion of the results
 RQ 1 answer: in all the cases the
proposed testing process was able to
reach the Saturation at the termination
point.
 RQ 2 answer: the main factors affecting
the effectiveness of the proposed testing
processes were:
1. the preconditions of the AUT,
2. the configurations of the testing platform,
3. the limitations and the configuration of the
MFT tool, and
4. the existence of unreachable statements in
the source code of the AUT.
Mobilesoft 2015 – Florence, Italy – May 16 24

25. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
Lesson learned …
 We were able to derive an alternative and
more efficient stop condition.
◦ “The data suggested that the process could be
stopped, without loss of code coverage, as soon
as the coverage of one of the sessions reached
the cumulative one.”
Mobilesoft 2015 – Florence, Italy – May 16 25
Don’t wait that the slowest
session covers the same code
executed by the other ones.
The process terminates when
the fastest session covers also
all the code executed by the
other ones.
what it means in practice

26. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
… Lesson learned
 To validate this intuition, we evaluated the new
Termination points and Termination Levels.
 The reduction rate varies between 24.48% and
93.58%.
 The same TerL values were obtained.
Mobilesoft 2015 – Florence, Italy – May 16 26

27. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
Conclusions
 A fully automatic MFT process that is able to find
a cost-effective termination point was presented.
 The process is supported by a software
infrastructure we developed for testing Android
mobile applications.
 The validity of the approach was shown by an
experiment.
27Mobilesoft 2015 – Florence, Italy – May 16
Future Works
 Improve the experimentation in order to extend
the validity of the proposed approach in mobile
context.
 Extend the validity of the approach in other
contexts, i.e. desktop applications.

28. Exploiting the Saturation Effect in Automatic Random Testing of Android Applications
Thank you !!!
Mobilesoft 2015 – Florence, Italy – May 16 28