The document presents a conceptual framework for comparing various fully automated GUI testing techniques, particularly in dynamic online testing for Android applications. This framework categorizes testing methods into random and active learning techniques, evaluates their performances based on parameter choices, and proposes a generalized algorithm for systematic comparison. Additionally, it outlines experiments using the Trolly Android application, analyzing factors like test adequacy and complexity, while also emphasizing the influence of description strategies on testing outcomes.

1. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
A Conceptual Framework for the
Comparison of Fully Automated GUI
Testing Techniques
Domenico Amalfitano
Nicola Amatucci
Anna Rita Fasolino
Porfirio Tramontana

2. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Context and Motivations
 Context
◦ Dynamic online testing techniques
◦ Android GUI Testing
 Motivation
◦ Objective and systematic comparison of
fully automatic testing techniques.

3. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Dynamic online testing
techniques
 According to Rothermel these
techniques are defined as:
◦ Dynamic event extraction-based GUI test
case generation techniques
 In these techniques:
◦ Test cases are actually sequences of one or
more GUI events that must be automatically
fired on the application under test (AUT).
◦ Test cases are executed as soon as they are
automatically defined at run-time.
 There is not a clear distinction between the steps of
Test Cases Generation and Test Cases Execution.

4. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
A first classification proposal
 By a preliminary analysis of the
literature it is possible to classify these
techniques into two families:
◦ Random techniques
◦ Active Learning techniques

5. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Random testing techniques
 Automatically exercise the subject
applications by firing pseudo-random
sequences of events through their
GUI.
◦ Usually exploited to perform stress testing
of the AUT.

6. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Active Learning testing
techniques
 Combine together GUI testing and
model learning techniques.
 Try to learn a model of the GUI and
generate user event sequences on the
basis of this model.
◦ Usually exploit systematic exploration
strategies of the GUI, such as the ones
emulating well-known graph exploration
algorithms like Breadth-first search (BFS),
Depth-first search (DFS).

7. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Performances of dynamic testing
techniques
 Intuitively the performances of these
techniques may be influenced by
different choices, such as:
◦ When the process must be stopped?
◦ How the next sequence of events to be fired
is selected?
◦ In the case of active learning techniques,
when two GUIs can be considered as
equivalent?
◦ Etc.
 A way for comparing in a systematic
manner the performances of different
techniques is needed.

8. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Proposed approach
 The approach is based on a
generalized algorithm that abstracts
the key aspects of dynamic GUI
testing techniques.
 The algorithm provides a conceptual
framework that can be used to define
and compare different GUI testing
techniques.

9. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
The proposed generic GUI
testing technique

10. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
The proposed generic GUI
testing technique
It is iterative
At each iteration the
termination condition
is evaluated on the
basis of a given
termination criterion
Set of
parameters
requested by
the algorithm

11. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
The proposed generic GUI
testing technique
Description strategy
parameter mainly
determines how the
descriptions of two or
more user interfaces
must be compared in
order to understand if
they can be
considered equivalent
or not.
An abstraction of the current user
interface is performed and the
learned GUI model is updated.

12. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
The proposed generic GUI
testing technique
A sequence of user
events that can be
executed on the
current GUI is
planned according
to the scheduling
strategy parameter.

13. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
The proposed generic GUI
testing technique
The planned
sequence of events
is actually executed
on the current GUI.
How these events
are
fired is determined
by the execution
strategy parameter.

14. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Testing techniques as instances
of the proposed algorithm
 The generalized algorithm can be
used to describe testing techniques
that vary as the values of the
algorithm parameters vary.
◦ Instances of the proposed algorithm are
dynamic GUI testing techniques.
◦ For a given instance, each parameter
assumes a specific value.

15. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Algorithm as a systematic
comparison framework
 The Algorithm can be exploited as a
framework for the systematic
comparison of different GUI testing
techniques.
◦ It can be exploited to understand how the
values assumed by one of the parameters
can influence the performances of the
technique.
 By varying one dimension and by fixing the
other ones.

16. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Evaluation
 We showed how the conceptual
framework can be exploited for defining
different fully automated testing
techniques and systematically comparing
their performances.
 We intended to understand:
1. How much the choice of the
parameters influences the
performances of fully automated testing
techniques.
2. How much the choice of the
parameters influences the models
inferred by Active Learning testing
techniques.

17. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Subject and metrics
 As subject application we considered
the Trolly Android application.
 Performances evaluated as:
◦ Test adequacy in terms of:
 LOCs coverage and LOCs Coverage %
◦ The cost for the execution in terms of:
 # of Fired Events
 The complexity of the inferred GUI
trees evaluated in terms of:
 # of Nodes, # of Edges, # of Leaves and
Depth.

18. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Framework Configurations
 We defined different values of each
parameter of the algorithm to
implement different fully automated
testing techniques.
◦ A single value for precondition and
execution strategy.

19. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Framework Configurations
cont’d
 Three values of Termination
Criterion:
◦ Nmg: all the visited GUIs have been
already encountered.
◦ MaxEv: a given maximum number of
events has been executed.
◦ Sat: a given number of parallel
executions of the algorithm covers the
same set of LOCs.

20. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Framework Configurations
cont’d
A OW WFV N
Description
Strategy
the current
GUI is
characterized
just by the
name of the
Activity class
that renders it.
the current
GUI is
characterized
by the types
of the widgets
it includes.
like OW, but
widgets are
also described
in terms of all
their
attributes
-
GUI
Equivalence
two GUIs are
considered
equivalent if
they are
rendered by
the same
Activity class
two GUIs are
considered
equivalent if
they have
identical
widgets
descriptions.
two GUIs are
considered
equivalent if
they have
identical
widgets
descriptions.
This strategy
assumes that
two GUIs are
never
equivalent.
 Four values of Description Strategy:

21. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Example of OW and WFV
 The two GUIs
are considered
as equivalent
according to
OW.
 The two GUIs
are not
considered as
equivalent
according to
WFV.

22. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Framework Configurations
cont’d
 Three values of Scheduling Strategy:
◦ BF: breadth first GUI navigation
◦ DF: depth first GUI navigation
◦ R: uniform random GUI navigation

23. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Values Combinations
 By combining some of the values 9 fully automated testing
techniques were defined and implemented.
 Some combinations were not meaningful or not applicable.
◦ Nmg termination criterion can not be exploited in conjunction
with the N description strategy.
◦ Sat termination criterion can be exploited only in conjunction
with the R scheduling strategy.

24. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Performance Results

25. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Performance Results
Random technique obtained
better test adequacy results
than the Active Learning one.

26. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Performance Results
The description strategies
affect the performances of the
Active Learning testing
techniques.
Compare T1– T2 – T3 and T4
– T5 – T6

27. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Performance Results
BF and DF do no affect the
performance of the Active
Learning testing techniques.
Compare T1– T4, T2 – T5 and
T3 – T6

28. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
GUI tree Complexity Results
Complexity is mainly affected
only by the description
strategies.
Compare T1– T2 – T3 and T4
– T5 – T6

29. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
About inferred GUI trees
 Techniques T1 (based on BF) and T4 (based on
DF) learned GUI trees having the same number
of nodes, edges and leaves, but T4 inferred a
deeper GUI tree.
◦ Inferred GUI trees have different topology.
GUI Tree inferred by T1
GUI Tree inferred by T4

30. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Conclusions
 A conceptual framework that can be
exploited for describing and
implementing different fully automated
GUI testing techniques was proposed.
 The framework can be adopted for a
systematic comparison of different
testing techniques.

31. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Future Works
 To extend and validate the framework
on the basis of the different
techniques proposed in the literature,
in order to discover additional or
alternative parameters
 To exploit the framework for designing
an empirical study aiming at
systematically comparing the
performance of different techniques

32. i
TESTBEDS 2015 – Lincoln, Nebraska – November 10th
Thanks for your attention
Questions?
Further Information:
http://reverse.dieti.unina.it
@REvERSE_UNINA
domenico.amalfitano@unina.it