1. Espresso vs. EyeAutomate:
comparing two generations of
Android GUI testing
Luca Ardito, Riccardo Coppola, Maurizio Morisio, and Marco Torchiano
EASE 2019- Evaluation and Assessment in Software Engineering

2. is hard!

3. Diversity

4. Evolution

5. Minor Changes

6. Testing mobile apps is hard
• Wide variety of devices
• Versions of OS
• Screen aspect ratio
• Display Hw vs. Sw buttons
• Quick evolution of UIs
• Test script fragility
• Even minor UI changes can break tests

7. End-2-End Testing Techniques
• Manual
• Scripted
«E2E testing is used to test whether
the flow of an application right from
start to finish is behaving as expected»

8. End-2-End Testing Techniques
• Manual
• Coordinate-based
• Layout-based
• Visual / Image Recognition
Scripted

9. End-2-End Testing Techniques
• Manual
• Coordinate-based : 1st generation
• Layout-based : 2nd generation
• Visual / Image Recognition : 3rd generation

10. Layout-based testing
Test scripts leverage the UI structure to
• Identify and detect UI components
• Interact with them

11. Layout-based testing tools

12. Visual testing
• Image recognition techniques are used to identify elements of the
user interface to interact with.
• Ease of definition of test cases with no tech knowledge required
• only screen captures are needed
• Can be applied seamlessly to any kind of software provided with an
(emulated) user interface
• Very high fragility to even minor changes in the GUI
• Difficult in-depth testing of application functionalities.

13. Visual testing tools
Eye Automate

14. Experiment Goal
• Analyze the E2E testing process
• to understand what technique
• Layout-based
• Visual
• yields
• higher productivity
• high quality of tests

15. Experiment Goal
• Analyze the E2E testing process
• to understand what technique
• Layout-based
• Visual
• yields
• higher productivity
• high quality of tests
Omni-Notes
Context
Android native app:
MSc Students in
Software Engineering course

16. Research Questions
RQ1 Productivity: What is the productivity in test script production of
graduate students with Visual and Layout-based testing tools for
Android App testing?
RQ2 Quality: What is the quality of the test suites produced by
graduate students using Visual and Layout-based testing tools for
Android App testing?
RQ3 Obstacles: What are the perceived difficulties in applying Visual
and Layout-based testing tools for Android App testing?

17. Experiment Design
Session 1 Session 2
Scenarios
Eye Automate
Test
scripts
Test
scripts
Feedback
Test
results
Test
results
Feedback
RQ3
RQ1
RQ2

18. Scenarios
Open Info Screen
Open AboutActivity and verify that App icon and copyright notice
are shown;
Add a note
Insert a note with custom title and content, and verify that it is
shown in the note list;
Search a note Input a note’s text in search bar, and verify that such note is shown;
Check available
languages
Open Language option in SettingsActivity and verify that English,
Italian and French language are available;
Delete a note
Delete and then restore a note from the TrashAc- tivity, then check
that it is shown again in the MainActivity;
Restore a note
Add and then delete a note, and check that it is no longer shown in
the MainActivity;
Add note category
Add a note category with custom name and color, and check that it
is shown among available ones in the DrawerMenu.

19. Feedback Questionnaire
• Implementing the test suite with EyeAutomate / Espresso was easy
and intuitive
• The EyeStudio / Android Studio IDE was helpful in the creation of test
scripts
• It was easy to identify elements with the EyeAutomate / Espresso
technique
• What were the main issues in identifying elements of the screen
using the EyeAutomate / Espresso testing approach? (Open)
• Which tool would you choose if you had to perform E2E testing in the
future? And why? (Open)

20. ?(Open)
Espresso frame-
plementation of
using the layout-
elements of the
rform UI testing
omment)
average number
was 0.6.
21.5%) reported
Layout−based Visual
−0.25 0.00 0.25 −0.25 0.00 0.25
0
2
4
6
Productivity[testcases]
Results: productivity
⚖️
No significant difference
( p = 0.625 )

21. ?(Open)
Espresso frame-
plementation of
using the layout-
elements of the
rform UI testing
omment)
average number
was 0.6.
21.5%) reported
Layout−based Visual
−0.25 0.00 0.25 −0.25 0.00 0.25
0
2
4
6
Productivity[testcases]
Layout−based
No Yes
0
2
4
6
Productivity[testcases]
Results: productivity
Within Espresso student
using Test Recorder
(average 6.7) and those
who didn’t (average 4.13)
show a significant
difference (p<0.001)

22. Layout Visual
−0.25 0.00 0.25 −0.25 0.00 0.25
0.00
0.25
0.50
0.75
1.00
TestCaseQuality
Results: quality
👍
Significant difference
( p = 0.025
Cliff’s δ = 0.18 )

23. Layout Visual
−0.25 0.00 0.25 −0.25 0.00 0.25
0.00
0.25
0.50
0.75
1.00
TestCaseQuality
Results: quality
👍
Layout−based
No Yes
0.00
0.25
0.50
0.75
1.00
TestCaseQuality
Within Espresso, student
using Test Recorder
(average 0.57) and those
who didn’t (average 0.71)
show a significant
difference (p=0.021)

24. Results: perception
by
ed
g
e
st
ge
st
ar,
he
g
ns
s,
h
g
n
h
ad
ot
es
be
ft
Hs30
participants’ perceived easiness in
nding properties to discriminate GUI
components, using theVisual or the
Layout-based approach
0.0181 Reject
Layout−based
Visual
1 2 3 4 5
(a) 2.2 - 2.6: Implementing the test suite was easy and in-
tuitive
Layout−based
Visual
1 2 3 4 5
(b) 2.3 - 2.7: The IDE was helpful in the creation of test
scripts
Visual
g
re
st
ge
st
ar,
he
ng
ns
s,
h
ng
n
h
ad
ot
es
be
ft
as
components, using theVisual or the
Layout-based approach
Layout−based
Visual
1 2 3 4 5
(a) 2.2 - 2.6: Implementing the test suite was easy and in-
tuitive
Layout−based
Visual
1 2 3 4 5
(b) 2.3 - 2.7: The IDE was helpful in the creation of test
scripts
Layout−based
Visual
ere
est
ge
est
Bar,
he
ng
ns
ns,
th
ng
in
th
ad
pot
es
be
eft
as
ms
der
Layout−based
Visual
1 2 3 4 5
(a) 2.2 - 2.6: Implementing the test suite was easy and in-
tuitive
Layout−based
Visual
1 2 3 4 5
(b) 2.3 - 2.7: The IDE was helpful in the creation of test
scripts
Layout−based
Visual
1 2 3 4 5
Ease of use
IDE support
Element identification
👍
🏻
👍
🏻
👍
🏻

25. Results: visual obstacles
• Issues with image recognition
• Capture size
• Click position
• Other issues (e.g. small images)
• Resolution / Portability

26. Results: layout-based obstacles
• Identifying widgets
• Missing / ambiguous properties
• Layout hierarchy
• Recording tool

27. Results
• RQ1: the learnability of the two tools is similar, for non-professional
developers approaching them.
• RQ2: the test suites developed with EyeAutomate possess a higher
quality than those developed with Espresso.
• Better quality has been achieved by the participants when manually writing
test scripts, as opposed to leveraging the Capture & Replay approach
implemented by Espresso Test Recorder.

28. Results
• RQ3: developing a test suite with EyeAutomate (w/EyeStudio IDE) is
slightly easier than with Espresso (w/Android Studio IDE)
• The main obstacles are:
• imprecision of the image recognition library
(especially with small elements and very simple patterns)
• difficulty of finding unambiguous layout properties
• Preference towards the visual approach
• higher intuitiveness and ease of use in building test scripts through screen
captures.

29. Summary
• Visual testing tools enable testers to deliver similar amount of tests
but with higher quality when compared to layout-based tools
• Visual testing tools exhibit problems in image capture and recognition
• Especially with small items
• Layout-based test scripts is difficult due to the incomplete or
ambiguous definition of GUI components or layouts

30. Take away
• If you are an app developer, then
• consider adopting “testability” guidelines
• If you are a tester, then
• think about visual tools
• If you are a visual tool developer, then
• improve image recognition
• If you are a layout-based tool developer, then
• improve script capturing