Most of the mobile phones have turned into full-connected devices. This provides companies with a perfect channel to interact with their potential clients and employees. The quality of the experience with these applications can directly affect the profits of the company it represents. Focusing on the mobile field and its extremely dynamic context, the quality of the experience can highly fluctuate. Inside this field, several methods and tools have been developed by defining a context of use. However, current methods can only capture it through adding external capture tools (added cameras, human observers...) that can change the experience. The main contribution in this article is a new approach to automatically measure effectiveness through a tiny but powerful mobile tool that can capture interaction metrics and the surrounding context without biasing the measured experience.

1. Effectiveness Measurement Framework for
Field-Based Experiments Focused on Android
Devices
CLIHC
2013
Ivan Pretel, Ana B. Lago
DeustoTech - Deusto Institute of Technology, University of Deusto
http://www.morelab.deusto.es
December 6, 2013
CLIHC
2013
1/37

2. Outline
Introduction
Usability, Quality and Effectiveness
Context model focused on Mobile Devices
Context model Capturer
System Validation
Conclusion
CLIHC
2013
2/37

3. Introduction
Usability, Quality and Effectiveness
Context model focused on Mobile Devices
Context model Capturer
System Validation
Conclusion
CLIHC
2013
3/37

4. CLIHC
2013
4/37

5. Introduction (I)
►
According to the last Ericsson Mobility Report
(November 2013)
► By
Q3 2013, total mobile subscriptions were around 6.6
billion. By the end of 2019, they are expected to reach around
9.3 billion.
CLIHC
2013
Introduction
5/37

6. Introduction (II)
►
Testing tools have changed and have been focusing on
the web domain
►
But, the web domain is focused not only on its own but
also on the mobile one
►
Mobile device -> More contexts than ever
► Issue
► The
quality of mobile applications fluctuates depending on
the context in which they are used
► When we try to capture it, we modify it
CLIHC
2013
Introduction
6/37

7. Introduction (III)
►
Approach
►A
mobile-based tool to automatically evaluate the
effectiveness of interactions and capture metrics of the
surrounding context without adding disturbing elements
CLIHC
2013
Introduction
7/37

8. Introduction
Usability, Quality and Effectiveness
Context model focused on Mobile Devices
Context model Capturer
System Validation
Conclusion
CLIHC
2013
8/37

9. Usability, Quality and Effectiveness (I)
►
Usability
► ISO
9241-11
► “Usability is the extent to which a product can be used by
specified users to achieve specified goals with effectiveness,
efficiency and satisfaction in a specified context of use”
– “Effectiveness is the level of accuracy and completeness with
which users achieve specified goals”
► ISO
9126-4
► “Usability is the capability of the software product to be
understood, learned, used and attractive to the user, when is
used under specified conditions”
– “Effectiveness is the capability of the software product to enable
users to achieve specified goals with accuracy and
completeness in a specified context of use”
Usability, Quality and Effectiveness
CLIHC
2013
9/37

10. Usability, Quality and Effectiveness (II)
►
Quality according to ISO 9126
►
”Represents a property of the software product defined in terms of a set of
interdependent attributes (usability, security, reliability, performance,
complexity, readability, reusability) expressed at different levels of detail and
also taken into account the particular context of use”
► Can
be measured by 3 quality aspects
► Internal
Quality
– From an internal view (e.g. spent resources)
► External Quality
– Its behaviour (e.g. number of wrong expected reactions)
► Quality in Use
– “The quality of software that user can perceive when the
software is used in an explicit context of use”
– It is measured by 4 main capabilities
» Effectiveness, satisfaction, productivity and safety
Usability, Quality and Effectiveness
10/37
CLIHC
2013

11. Usability, Quality and Effectiveness (III)
►
Effectiveness metrics (ISO/IEC 9126)
► Task
Effectiveness
► What
proportion of the goals is achieved?
– Measures the quantity of the goals achieved by a user
–  0<=TE<=1
► Task
Completion
► What
proportion of the tasks is completed?
– Measures the level of success the user achieves
performing tasks
–  0<=TC<=1 
► Error
Frequency
► What
is the frequency of errors?
– Measures the number of times that an error is made within
a given period
–  0<=EF<=1 
Usability, Quality and Effectiveness
11/37
CLIHC
2013

12. Introduction
Usability, Quality and Effectiveness
Context model focused on Mobile Devices
Context model Capturer
System Validation
Conclusion
CLIHC
2013
12/37

13. CLIHC
2013
13/37

14. Context model focused on Mobile Devices (I)
►
It has been taken into account and has been defined
several times
► Physical
location
► Weather attributes
► Involved communities & stakeholders
► Goals
►…
►
Tries to answer
► Who
are the users?
► What are they doing?
► Where?
► Which resources are nearby?
Context model focused on Mobile Devices
CLIHC
2013
14/37

15. Context model focused on Mobile Devices (II)
►
Context of Use model based on 3 main components
►
Nuevo modelo basado en tres pilares
CLIHC
2013
Context model focused on Mobile Devices
15/37

16. Introduction
Usability, Quality and Effectiveness
Context model focused on Mobile Devices
Context model Capturer
System Validation
Conclusion
CLIHC
2013
16/37

17. CLIHC
2013
17/37

18. Context model Capturer - Lab vs Field (I)
►
Capturing tools
► Lab-based
► Log
information in a highly controlled
environment using specific devices and users
– Control: high
– Context variables: limited
– Objectivity: low
► Field-based
► Log
using added agents and invasive evaluation
methods
– Control: very low
– Context variables: unlimited
– Objectivity: depends on the added agents
Context model Capturer
CLIHC
2013
18/37

19. Context model Capturer - Lab vs Field (II)
► If
you do not want to influence the context, you have to
remove all the biasing elements which can influence the
captured data and use only the elements which is
composing it
Capture using only the mobile device
► This
tool should capture the context model via the built-in
mobile sensors and logging interaction events
CLIHC
2013
Context model Capturer
19/37

20. Context model Capturer - Purposed System
►
The purposed system is formed by a tiny Android library
and a server to store and log the performed interactions
1. The app-to-test (ATT) should be integrated with the library
► 2. This library automatically captures context and interaction
information and stores it in a local database
► 3. When the device has internet connection and its owner wants,
all the information is uploaded
►
CLIHC
2013
Context model Capturer
20/37

21. Context model Capturer – Library (I)
►
Interaction attributes
► Using
one command
► log(event_type,
interface, object)
CLIHC
2013
Context model Capturer
21/37

22. Context model Capturer – Library (II)
►
Static attributes
► Captured
►
during the registration
Automatic
►
Manual
CLIHC
2013
Context model Capturer
22/37

23. Context model Capturer – Library (III)
►
Dynamic attributes
► Captured
►
►
during the interaction (all automatically)
Android API
Device
Sensors
►
External
Services
CLIHC
2013
Context model Capturer
23/37

24. Introduction
Usability, Quality and Effectiveness
Context model focused on Mobile Devices
Context model Capturer
System Validation
Conclusion
CLIHC
2013
24/37

25. System Validation – Experiment (I)
4 subjects
► A tiny memory game
► 1 day registering interactions
► Contexts
►
► At
home [H]
► Public Transport [PT]
► Walking Down the street [C]
► At work (office) [T]
CLIHC
2013
System Validation
25/37

26. System Validation – Experiment (I)
►
Steps
►
Download the Android app
– From http://qualityinuseserver.appspot.com
►
►
►
Sign up on the testing platform through the app
Play the game
Upload info to server
CLIHC
2013
System Validation
26/37

27. System Validation – Experiment (I)
►
Focusing on the effectiveness
measurement
► Task:
To end up with all of the cards
flipped face up in less than 15 turns
► More
than 15 turns: task is not
completed
►8
pairs of cards
► It
means the best round is made by only
8 interactions and 0 errors
CLIHC
2013
System Validation
27/37

28. System Validation
►
Effectiveness metrics
►
►
►
Task Effectiveness (TE)
► TE = 1 – ΣAi * error weight
► The weight of an error is 1/7 = 0.143
– Max number of errors is 6 with 8
correct interactions
– If you make 7 errors
» (1- (7*1/7) = 0)
Task Completion (TC)
► TC = won rounds / played rounds
Error Frequency (EF)
► EF = Number of errors / Total turns
CLIHC
2013
System Validation
28/37

29. System Validation – Results
►
►
►
The outdoor contexts (PT and S) affect subjects and lead them to be
less effective than in the indoor contexts (H and W).
TE indicates that the S context is the context where subjects have
the worst results
The most efficient context is W, although EF exposes that is where
subjects make more mistakes
CLIHC
2013
System Validation
29/37

30. Introduction
Usability, Quality and Effectiveness
Context model focused on Mobile Devices
Context model Capturer
System Validation
Conclusion
CLIHC
2013
30/37

31. Conclusion
►
The effectiveness and the new context model, which can
bias the usability and the quality of mobile applications,
can be automatically captured and measured by the own
mobile device minimizing the interaction bias
CLIHC
2013
Conclusion
31/37

32. Conclusion
►
Found issues
► Security
–android.permission.INTERNET
android.permission.ACCESS_FINE_LOCATION
–android.permission.ACCESS_NETWORK_STATE android.permission.ACCESS_WIFI_STATE
–android.permission.MODIFY_AUDIO_SETTINGS android.permission.RECORD_AUDIO
–android.permission.WRITE_EXTERNAL_STORAGE
► Weather
variables capture depends on the Internet
connection
► Developers have to specify where are the errors
►
Next steps
► Solve
the found issues
► Increase the number of users
► Calculate correlations
CLIHC
2013
Conclusion
32/37

33. Thank you for your
attention
¡Pura vida!
CLIHC
2013
33/37

34. Effectiveness Measurement Framework for
Field-Based Experiments Focused on Android Devices
Iván Pretel
{ivan.pretel@deusto.es}
DeustoTech - Deusto Institute of Technology, University of Deusto
http://www.morelab.deusto.es
CLIHC
2013
34/37

35. All rights of images are reserved by the original
owners*, the rest of the content is licensed under a
Creative Commons by-sa 3.0 license.
*
•
•
•
•
•
•
http://www.fotopedia.com/items/flickr-2830319467
http://www.fotopedia.com/items/flickr-4609827016
http://en.wikipedia.org/wiki/File:Skyline_oklahoma_city.JPG
http://www.flickr.com/photos/grimsanto/6190478735/
http://commons.wikimedia.org/wiki/File%3AAstronomy_Amateur_3_V2.jpg
http://commons.wikimedia.org/wiki/File%3AAldrin_Apollo_11_cropped.jpg
CLIHC
2013
35/37