EPFL, spring 2012 - week 12!
This document provides an overview of UX evaluation including:
1) Today's schedule involves reviewing UX evaluation plans, test runs of scripts, and individual walkthroughs.
2) An overview of UX evaluations including criteria, methods, tips, and more information.
3) The importance of UX evaluation is discussed as UX is a critical success factor and it is difficult for developers to maintain objectivity. User testing helps get outside perspective to catch usability issues.
3. overview
➝ about UX evaluations
➝ UX evaluation criteria
➝ UX evaluation methods
➝ tips and hints
➝ for more information
✱Attribution: parts of this presentation are based on "
Forum Nokia’s UX Evaluation series
4. why UX evaluation?
Success factor
➝ UX is a critical success factor for products and services
➝ We need confirmation during design and development
that we are on the right track
➝ It is challenging for the software developer to keep an
The designer is
objective view of the developed software
not the user
➝ It is human nature to become blind of your own
outcomes
➝ Thus it is important to get a second opinion of the UX,
from outside of the development team
Simulate real
➝ Preferably from real users trying it in a realistic
use situations
environment
5. usability and product quality
SW Product Quality (ISO-9126)
Functionality Reliability Efficiency Usability Portability Maintainability
Suitability Maturity Time behavior Operability Installability Testability
Accuracy Fault tolerance Resource Learnability Adaptability Stability
utilization
… … ... … … …
Implemented Code
Number of sw Function point Source lines of
functional coverage
bugs
analysis
code
requirements
➝ there are many metrics to measure software performance and quality
6. UX and product quality
In this definition, UX is not a
part of product quality.
A SW
SW Product Quality (ISO-9126)
engineering
point-of-view?
Functionality Reliability Efficiency Usability Portability Maintainability
Suitability Maturity Time behavior Operability Installability Testability
Accuracy Fault tolerance Resource Learnability Adaptability Stability
utilization
… … ... … … …
➝ product quality Expectations, motives, ➝ measuring UX is not straight-
has a strong actions, interpretations,
forward
impact on UX
…
➝ there are many intervening
issues
User Experience
7. UX evaluation criteria 1/5
➝ Nielsen (1993) on usability
➝ learnability
➝ Novices’ ability to reach a
reasonable level of performance
rapidly
UX
➝ efficiency
➝ Expert user’s level of performance
➝ errors
utility
usabi
social enjoy
lity
value
ment
➝ Number of errors users make
➝ User’s ability to recover from
errors
pleasure
stimulation
useful
relia
ease- effici
acces
connec
identifi
sibility
ting cation
➝ Existence of catastrophic errors?
ness
bility
of-use
ency
people
➝ memorability
➝ Casual user’s ability to remember drivers tailored issues
how to use a system
as such
for your detailed for
➝ satisfaction
product
your product
➝ User’s subjective assessment of
the system
basic usability heuristic
UX drivers derived from the UX elements
8. UX evaluation criteria 2/5
Melvin, 35! Jason, 25! Louise, 27! use
case
storyboard
accessing
phone
from
web
browser
• Engineer from Germany • Customer Service • Web designer, graduate
• Lives with his wife, 2 kids Assistant. student on university.
and a dog. • Keeps a blog for his friends • SMS addict - 50
• Both parents need to and family messages every day.
sometimes travel due their Uses his device for capturing • Likes arts, especially indie
jobs, and kids have lots of images to blog. movies and old films.
hobbies. Most of the images • Is currently organizing
• Would like better manage surprise parties for her
related to snowboarding.
his everyday schedules friend.
• Plays guitar in a wildly
with family.
not-yet-so-popular band.
➝ Evaluate the design against the specified UX artefacts
10. UX evaluation criteria 4/5
Bar
Slide
Full hw qwerty
Full sw qwerty
Touch screen
Touch only screen
Resistive screen
Capacitative screen
Clamshell
Swivel
➝ Run your tests with all the target devices and form factors
11. UX evaluation criteria 5/5
people
places
context
Things
time
Indirect, unspoken criteria
Direct, spoken criteria
- originating from the
context
➝ the ultimate criterion:
➝ potential users try the product in their mobile context
12. planning the UX evaluations 1/2
A step-by-step
process would
specify needs be easy to
and context of define the UX create UI evaluate the UI manage.
use
concept
designs
designs
In practice,
design and
evaluation are
specify needs
define the UX often in a parallel
create UI designs
and iterative
and context of use
concept
relationship.
evaluate concepts and designs
➝ for any UX artifacts, you can think can and should it be evaluated, somehow
13. planning the UX evaluations 2/2
My cool travel-
mate concept
➝ What artefacts do we want to evaluate
➝ Sketches, visualisations, prototypes,...
➝ What issues are we evaluating?
➝ Concept, interaction, grahical design, ...
➝ Who will do the evaluation?
➝ Experts, end users,...
Evaluate concepts and designs
14. UX evalution: expert evaluation
➝ Usability experts evaluate a system based on
common heuristics (i.e. design principles) and
personal expertise
➝ Common heuristics are based on knowledge gained
through expertise and experience
➝ Knowledge about perceptional and cognitive processes
- such as the function of memory – are utilized
➝ At least two experts take part in the evaluation to
ensure reliable results
➝ At the beginning of an expert evaluation, the value of
the system to its user and buyer/provider is
determined
➝ Based on these values, usability criteria are defined
➝ At the end, the findings documented and prioritized.
Solutions proposals are made
➝ Also user tasks can be defined and walkthroughs
conducted as part of the evaluation
15. UX evaluation: focus groups
➝ a group of people (4-7) that has
something in common will be invited
to semi-structured group discussion
session to share their views on
certain topic. Lasts usually 2 hours
➝ face-to-face sessions
➝ Sometimes on-line
➝ variations of the method
➝ Exploratory
➝ Feature prioritization
➝ Competitive analysis
➝ Trend explanation
16. UX evaluation: usability testing
➝ the test user is asked to conduct certain tasks with the
user interface under design
➝ paper prototypes can be used
➝ users are asked to think aloud
Special equipment can
be useful to catch the
Moderator
actions on mobile
Observer
devices
Tested sw
Test user
Test cases
➝ sometimes good to test the UI with pairs of users to
trigger more discussion or to test social aspects
➝ usability problems are identified and prioritized based on
the user data gathered during the usability tests
17. UX evaluation: observation
➝ the researcher does not interrupt the user
but just observes the user s actions
➝ the user is followed usually shorter time
than when shadowing in her/his own
environment, and usually only certain
activities/time period are observed
➝ good for studying users that cannot be
interrupted
➝ reveals task flows and possible detectable
If observation is not possible
(e.g. in hospital) video
problems in them and behavioral patterns
recording can be alternative. but not motivations, attitudes, values,
Video can be gone through
with the participant(s) concerns
afterwards
18. don't listen to users
➝ watch users attempting to perform tasks in the UI
➝ watch what people actually do
➝ be careful with what people say they do
➝ don't believe what people predict they may do in the
future.
19. when you listen
beware:
➝ users often tell you what they think you want to
hear
➝ often based only on recall
➝ rationalized behavior
20. UX evaluation: make a mix
➝ an expert evaluation and a usability
test can be used as
complementary methods
➝ an expert evaluation require less
time and preparation than a
usability test
➝ expert evaluations are
recommended before usability
tests in order to support designing
the test tasks
➝ a satisfaction questionnaire can be
combined with a usability test
21. UX evaluation: tips and hints
➝ the UX evaluation criteria should be in-line with
the defined UX drivers, targets and requirements
➝ make sure to share your UX evaluation criteria
with the designers before they start their work
➝ do evaluations with real users
➝ expert evaluations can provide quick useful findings
➝ try to have a end user mindset
➝ ideally your evaluation methods should scale to weekly (or
design
evaluate
even daily) evaluations
➝ design & evaluation iteration loops do not have to be
iterate
big monolithic steps
➝ UX evaluation(s) should be an integral part of
software development process, just like
software testing
