Presents an introduction to some basic metrics for usability and some current trends in UX evaluation methods. Includes some indicative examples from UX evaluation studies conducted by the author

1. A brief intro to UX Research
Dr. Panagiotis Zaharias
13/5/2014 @ CoLab Athens

2. The UX Mantra
To measure is to know.
If you can’t measure it,
you can’t improve it…

3. Overview of the talk
Examples of UX
evaluation studies
Current trends in
UX evaluation
UX Research:
Main Concepts

4. Main Concepts in UX Research
First things first

5. The Ancestor: Human Computer Interaction
 Human – Computer Interaction, is “a discipline concerned
with the design, evaluation and implementation of interactive
computing systems for human use and the study of major
phenomena surrounding them’’.

6. User friendly? Usable? …
 Once upon a time we were talking about “user friendly”
systems…
… so general and vague
 Then usability emerged and was originally derived from the
term ‘‘user friendly’’
… but usability is still a general and vague term
 Many definitions and different approaches to how usability
should be measured

7. What is usability anyway?
 The ISO definition (9241-11)
 Usability is defined as: ‘‘the extent to which a product can be
used by specified users to achieve specified goals with
 effectiveness; the extent to which the intended goals of use are
achieved,
 efficiency; the resources that have to be expended to achieve the
intended goals and
 satisfaction; the extent to which the user finds the use of the
product acceptable,
in a specified context of use’’.

8. Examples of usability measures (1)
 Measures of effectiveness:
 Task completion (number of correct tasks, number of tasks
where users failed to finish within a set time, number of
tasks where users gave up)
 Accuracy measures (quantify the number of errors users
make during the process of completing tasks)
 Recall measures (how much information users can recall
after the use of an interface)
 Quality of outcome (attempts to measure the outcome of
the tasks, measures of understanding e.g. tests of what has
been learned from an e-learning system)

9. Examples of usability measures (2)
 Measures of efficiency
 Time (refers to measures of how long users take to
complete tasks)
 Input rate (e.g. words per minute)
 Mental effort (mental resources spend on interaction, e.g.
heart rate variability, subjective time estimation)
 Usage patterns (number of times a certain action has been
performed, how much information users access when
solving tasks, deviation from the optimal solution etc.)

10. Examples of usability measures (3)
 Measures of satisfaction
 Preference measures capture which interface users prefer
using
 Specific attitudes towards the interface (liking, fun,
annoyance etc.), toward the content of interface (quality of
information, organization of information etc.)
 Perception of outcomes (users’ assessment of their
performance, users’ perception of learning, users’ confidence
in the solution to tasks)

11. From Usability to User Experience (1)
 UX - A relatively new field - Lack of unique definition…
 But let’s take for example the ISO definition on UX:
“A person's perceptions and responses that result from the use
or anticipated use of a product, system or service”

12. From Usability to User Experience (2)
Usability
• Focused on
optimizing task
efficiency
• Removing
problems in the
user interface
• Studying
interactive
products mainly
used for work
UX
• Dynamic
context-
dependent and
subjective
• Focuses also on
hedonic
qualities of use
(identification,
stimulation,
attractiveness
etc.)

13. Current trends in UX evaluation

14. As we shift to User Experience…
Challenge#1: Need for new approaches and methods
 Existing methods are somewhat unable -it is claimed- to
capture experience
 Challenge#2:The choice of UX evaluation method
 Qualitative or quantitative methods?
 Only few studies combine qualitative and quantitative methods

15. What Dimensions of Experience Are
Studied?
Usability
• Effectiveness
• Efficiency
• Satisfaction
UX
• Emotions and affect
• Enjoyment
• Aesthetics
• And many more…

16. What Dimensions of Experience Are
Studied?
Source: Bargas- Avila and Hornbaek, 2012

17. How do we measure UX?
 What are the methods applied in UX
 Qualitative (50 %), quantitative (33%), combination (17%)
 How UX data are collected?
 Questionnaires are still the dominant UX evaluation method
(53%).
Source: Bargas- Avila and Hornbaek, 2012

18. Popular UX evaluation methods
Source: Bargas- Avila and Hornbaek, 2012

19. When to use which UX evaluation methods
Decision
Qualitative
vs.
Quantitative
Attitudinal
vs.
Behavioral
Context of
system/
product use
Source: http://www.nngroup.com/articles/which-ux-research-methods/

20. The Attitudinal vs. Behavioral Dimension
to understand,
measure, or
inform change of
people's stated
beliefs
e.g. self-reported
methods such as
surveys,
interviews, focus
groups, etc.
Attitudinal to understand
"what people do"
with minimal
interference from
the method itself
e.g.A/B testing,
observation and
video recording,
first click testing,
eyetracking, etc.
Behavioral
Source: http://www.nngroup.com/articles/which-ux-research-methods/

21. The Qualitative vs. Quantitative Dimension
- the data is usually
being gathered directly
- they are much
better suited for
answering question
about why or how
to fix a problem
Qualitative - the data is
gathered indirectly,
through an
instrument, such as a
survey or a web
server log etc.
- they do a much
better job
answering how
many and how
much type of
questions.
Quantitative
Source: http://www.nngroup.com/articles/which-ux-research-methods/

23. The Context of Use Dimension
 This dimension can be described by the following options:
 Natural or near-natural use of the system/product
 Scripted use of the system/product
 Not using the system/product during the study
 A hybrid of the above
Source: http://www.nngroup.com/articles/which-ux-research-methods/

25. What about the use of UX evaluation
methods across the development phases ?

26. UX evaluation methods across development
phases
Product Development Phase
Strategize Optimize Assess
Goal: Inspire, explore and
choose new
directions and
opportunities
Inform and optimize
designs in order to
reduce risk and
improve usability
Measure product
performance against
itself or its
competition
Approach: Qualitative and
Quantitative
Mainly Qualitative
(formative)
Mainly Quantitative
(summative)
Typical
methods:
Ethnographic field
studies, focus groups,
diary studies,
surveys, data mining
or analytics etc.
e.g. cardsorting, field
studies, observation,
paper prototype and
usability studies, etc.
e.g. Usability
benchmarking,
surveys, A/B testing,
etc.
Source: http://www.nngroup.com/articles/which-ux-research-methods/

27. Some examples from UX research
studies

28. Walls of Nicosia
 The context:
 "TheWalls of Nicosia” is an interactive application that runs on a multi-
touch table at the Leventeio Museum, Nicosia
 Students from elementary schools as museum visitors
 Goal:To conduct a comparative
summative evaluation
 Methods used:
 Multiple choice knowledge tests
 UX questionnaires (FunToolkit)
 Observation
 Indicative results:
 Mainly quantitative analysis (pre and post test analysis, t-tests)

29. UX questionnaire for children: the
FunToolkit (Read, 2007)
The Again-Again table
The Smileyometer
Source: http://hrast.pef.uni-lj.si/docs/research/Serious%20games/evaluation_paper_Janet.pdf

30. Long term UX evaluation of League of
Legends (1)
 The context:
 The well known MMOG “League of Legends”
 Players were students at TEI of Kavala
 Goal: to identify how UX of players
changed over time
 Methods used:
 An adaptation of UX Curve
 Self reported method by drawing curves that depict changes in
UX over time

31. Long term UX evaluation of League of
Legends (2)
 Indicative results:
Reason
Categories Positive Negative
General UX 43 14
Ease of use 41 26
Immersion 43 10
Social
interaction 38 12
Engagement 32 10
Total 197 72
General UX Curve for each player
Number of reasons for UX dimensions

32. Heuristic Evaluation of LMS
 The context:
 A well known cloud-based Learning Management System
platform
 Goal: to identify usability problems and possible areas for
UX improvement
 Methods used:
 Heuristic evaluation
 Heuristic evaluation protocol adapted by Mehlenbacher et al.
(2005)
 Indicative results:
 An evaluation report that contains the identified usability and
UX problems, their severity, and relevant redesign suggestions

33. Typical structure of the report of heuristic
evaluation
 A numeric identifier of the problem
 A short description of the problem (and likely difficulties
for the user)
 Specific context (defined as location of the problem in the
interface): we identified this generically in terms of
affecting the:
 Structure of the system, including layout, navigation, and basic help
functions such as print, help, etc.
 Content, including actual content for study, exploration, examples,
exercises, and so on
 Global issues, i.e., affecting both structure and learning content
 The heuristic(s) used
 The severity rate: severe, moderate, and minor

34. First Click testing with mockups
 The context:
 A project for the design of new MOOC interfaces
 Goal: to create, inform and optimize MOOC interface
design
 Methods used:
 Interviews and first click testing with users

35. First Click testing - indicative results

36. First Click testing - indicative results
0
2
4
6
8
10
12
Ραβδόγραμμα εργασιών
Σωστές εργασίες Λανθασμένες Εργασίες

37. Some UX resources:
 World’s premier magazine for both academics and industry:
 http://interactions.acm.org/
 A comprehensive list of UX methods:
 http://www.allaboutux.org/
 An indicative list of UX tools:
 http://www.smashingmagazine.com/2011/10/20/comprehensive
-review-usability-user-experience-testing-tools/
 Other sources on the web:
 http://uxmag.com/
 http://www.usability.gov/
 http://www.uxmatters.com/index.php

38. Key takeaways:
You can’t afford NOT to evaluate UX
Always use a combination of methods
Don’t forget Usability
Do it properly…

39. Thank you!
Let’s keep in touch:
 Web: about.me/panagiotis_zaharias
 Email: pz@aueb.gr
 LinkedIn: gr.linkedin.com/pub/panagiotis-zaharias/0/bb6/a91
 Twitter: twitter.com/pan_sugarman