Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices
Upcoming SlideShare
Loading in...5
×
 

Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices

on

  • 268 views

Most of the mobile phones have turned into full-connected devices. This provides companies with a perfect channel to interact with their potential clients and employees. The quality of the experience ...

Most of the mobile phones have turned into full-connected devices. This provides companies with a perfect channel to interact with their potential clients and employees. The quality of the experience with these applications can directly affect the profits of the company it represents. Focusing on the mobile field and its extremely dynamic context, the quality of the experience can highly fluctuate. Inside this field, several methods and tools have been developed by defining a context of use. However, current methods can only capture it through adding external capture tools (added cameras, human observers...) that can change the experience. The main contribution in this article is a new approach to automatically measure effectiveness through a tiny but powerful mobile tool that can capture interaction metrics and the surrounding context without biasing the measured experience.

Statistics

Views

Total Views
268
Views on SlideShare
267
Embed Views
1

Actions

Likes
0
Downloads
4
Comments
0

1 Embed 1

https://twitter.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Presentation Transcript

  • Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices CLIHC 2013 Ivan Pretel, Ana B. Lago DeustoTech - Deusto Institute of Technology, University of Deusto http://www.morelab.deusto.es December 6, 2013 CLIHC 2013 1/37
  • Outline Introduction Usability, Quality and Effectiveness Context model focused on Mobile Devices Context model Capturer System Validation Conclusion CLIHC 2013 2/37
  • Introduction Usability, Quality and Effectiveness Context model focused on Mobile Devices Context model Capturer System Validation Conclusion CLIHC 2013 3/37
  • CLIHC 2013 4/37
  • Introduction (I) ► According to the last Ericsson Mobility Report (November 2013) ► By Q3 2013, total mobile subscriptions were around 6.6 billion. By the end of 2019, they are expected to reach around 9.3 billion. CLIHC 2013 Introduction 5/37
  • Introduction (II) ► Testing tools have changed and have been focusing on the web domain ► But, the web domain is focused not only on its own but also on the mobile one ► Mobile device -> More contexts than ever ► Issue ► The quality of mobile applications fluctuates depending on the context in which they are used ► When we try to capture it, we modify it CLIHC 2013 Introduction 6/37
  • Introduction (III) ► Approach ►A mobile-based tool to automatically evaluate the effectiveness of interactions and capture metrics of the surrounding context without adding disturbing elements CLIHC 2013 Introduction 7/37
  • Introduction Usability, Quality and Effectiveness Context model focused on Mobile Devices Context model Capturer System Validation Conclusion CLIHC 2013 8/37
  • Usability, Quality and Effectiveness (I) ► Usability ► ISO 9241-11 ► “Usability is the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” – “Effectiveness is the level of accuracy and completeness with which users achieve specified goals” ► ISO 9126-4 ► “Usability is the capability of the software product to be understood, learned, used and attractive to the user, when is used under specified conditions” – “Effectiveness is the capability of the software product to enable users to achieve specified goals with accuracy and completeness in a specified context of use” Usability, Quality and Effectiveness CLIHC 2013 9/37
  • Usability, Quality and Effectiveness (II) ► Quality according to ISO 9126 ► ”Represents a property of the software product defined in terms of a set of interdependent attributes (usability, security, reliability, performance, complexity, readability, reusability) expressed at different levels of detail and also taken into account the particular context of use” ► Can be measured by 3 quality aspects ► Internal Quality – From an internal view (e.g. spent resources) ► External Quality – Its behaviour (e.g. number of wrong expected reactions) ► Quality in Use – “The quality of software that user can perceive when the software is used in an explicit context of use” – It is measured by 4 main capabilities » Effectiveness, satisfaction, productivity and safety Usability, Quality and Effectiveness 10/37 CLIHC 2013
  • Usability, Quality and Effectiveness (III) ► Effectiveness metrics (ISO/IEC 9126) ► Task Effectiveness ► What proportion of the goals is achieved? – Measures the quantity of the goals achieved by a user –  0<=TE<=1 ► Task Completion ► What proportion of the tasks is completed? – Measures the level of success the user achieves performing tasks –  0<=TC<=1  ► Error Frequency ► What is the frequency of errors? – Measures the number of times that an error is made within a given period –  0<=EF<=1  Usability, Quality and Effectiveness 11/37 CLIHC 2013
  • Introduction Usability, Quality and Effectiveness Context model focused on Mobile Devices Context model Capturer System Validation Conclusion CLIHC 2013 12/37
  • CLIHC 2013 13/37
  • Context model focused on Mobile Devices (I) ► It has been taken into account and has been defined several times ► Physical location ► Weather attributes ► Involved communities & stakeholders ► Goals ►… ► Tries to answer ► Who are the users? ► What are they doing? ► Where? ► Which resources are nearby? Context model focused on Mobile Devices CLIHC 2013 14/37
  • Context model focused on Mobile Devices (II) ► Context of Use model based on 3 main components ► Nuevo modelo basado en tres pilares CLIHC 2013 Context model focused on Mobile Devices 15/37
  • Introduction Usability, Quality and Effectiveness Context model focused on Mobile Devices Context model Capturer System Validation Conclusion CLIHC 2013 16/37
  • CLIHC 2013 17/37
  • Context model Capturer - Lab vs Field (I) ► Capturing tools ► Lab-based ► Log information in a highly controlled environment using specific devices and users – Control: high – Context variables: limited – Objectivity: low ► Field-based ► Log using added agents and invasive evaluation methods – Control: very low – Context variables: unlimited – Objectivity: depends on the added agents Context model Capturer CLIHC 2013 18/37
  • Context model Capturer - Lab vs Field (II) ► If you do not want to influence the context, you have to remove all the biasing elements which can influence the captured data and use only the elements which is composing it Capture using only the mobile device ► This tool should capture the context model via the built-in mobile sensors and logging interaction events CLIHC 2013 Context model Capturer 19/37
  • Context model Capturer - Purposed System ► The purposed system is formed by a tiny Android library and a server to store and log the performed interactions 1. The app-to-test (ATT) should be integrated with the library ► 2. This library automatically captures context and interaction information and stores it in a local database ► 3. When the device has internet connection and its owner wants, all the information is uploaded ► CLIHC 2013 Context model Capturer 20/37
  • Context model Capturer – Library (I) ► Interaction attributes ► Using one command ► log(event_type, interface, object) CLIHC 2013 Context model Capturer 21/37
  • Context model Capturer – Library (II) ► Static attributes ► Captured ► during the registration Automatic ► Manual CLIHC 2013 Context model Capturer 22/37
  • Context model Capturer – Library (III) ► Dynamic attributes ► Captured ► ► during the interaction (all automatically) Android API Device Sensors ► External Services CLIHC 2013 Context model Capturer 23/37
  • Introduction Usability, Quality and Effectiveness Context model focused on Mobile Devices Context model Capturer System Validation Conclusion CLIHC 2013 24/37
  • System Validation – Experiment (I) 4 subjects ► A tiny memory game ► 1 day registering interactions ► Contexts ► ► At home [H] ► Public Transport [PT] ► Walking Down the street [C] ► At work (office) [T] CLIHC 2013 System Validation 25/37
  • System Validation – Experiment (I) ► Steps ► Download the Android app – From http://qualityinuseserver.appspot.com ► ► ► Sign up on the testing platform through the app Play the game Upload info to server CLIHC 2013 System Validation 26/37
  • System Validation – Experiment (I) ► Focusing on the effectiveness measurement ► Task: To end up with all of the cards flipped face up in less than 15 turns ► More than 15 turns: task is not completed ►8 pairs of cards ► It means the best round is made by only 8 interactions and 0 errors CLIHC 2013 System Validation 27/37
  • System Validation ► Effectiveness metrics ► ► ► Task Effectiveness (TE) ► TE = 1 – ΣAi * error weight ► The weight of an error is 1/7 = 0.143 – Max number of errors is 6 with 8 correct interactions – If you make 7 errors » (1- (7*1/7) = 0) Task Completion (TC) ► TC = won rounds / played rounds Error Frequency (EF) ► EF = Number of errors / Total turns CLIHC 2013 System Validation 28/37
  • System Validation – Results ► ► ► The outdoor contexts (PT and S) affect subjects and lead them to be less effective than in the indoor contexts (H and W). TE indicates that the S context is the context where subjects have the worst results The most efficient context is W, although EF exposes that is where subjects make more mistakes CLIHC 2013 System Validation 29/37
  • Introduction Usability, Quality and Effectiveness Context model focused on Mobile Devices Context model Capturer System Validation Conclusion CLIHC 2013 30/37
  • Conclusion ► The effectiveness and the new context model, which can bias the usability and the quality of mobile applications, can be automatically captured and measured by the own mobile device minimizing the interaction bias CLIHC 2013 Conclusion 31/37
  • Conclusion ► Found issues ► Security –android.permission.INTERNET android.permission.ACCESS_FINE_LOCATION –android.permission.ACCESS_NETWORK_STATE android.permission.ACCESS_WIFI_STATE –android.permission.MODIFY_AUDIO_SETTINGS android.permission.RECORD_AUDIO –android.permission.WRITE_EXTERNAL_STORAGE ► Weather variables capture depends on the Internet connection ► Developers have to specify where are the errors ► Next steps ► Solve the found issues ► Increase the number of users ► Calculate correlations CLIHC 2013 Conclusion 32/37
  • Thank you for your attention ¡Pura vida! CLIHC 2013 33/37
  • Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Iván Pretel {ivan.pretel@deusto.es} DeustoTech - Deusto Institute of Technology, University of Deusto http://www.morelab.deusto.es CLIHC 2013 34/37
  • All rights of images are reserved by the original owners*, the rest of the content is licensed under a Creative Commons by-sa 3.0 license. * • • • • • • http://www.fotopedia.com/items/flickr-2830319467 http://www.fotopedia.com/items/flickr-4609827016 http://en.wikipedia.org/wiki/File:Skyline_oklahoma_city.JPG http://www.flickr.com/photos/grimsanto/6190478735/ http://commons.wikimedia.org/wiki/File%3AAstronomy_Amateur_3_V2.jpg http://commons.wikimedia.org/wiki/File%3AAldrin_Apollo_11_cropped.jpg CLIHC 2013 35/37