This document discusses evaluating user experience (UX). It provides an overview of UX research approaches including those that view UX as fundamental cognitive constructs and those that see it as unique to individual contexts. The document also presents a framework for attractiveness in UX and describes experiments examining how aesthetics, customization and virtual characters influence engagement. It concludes that UX is multifaceted and can be systematically evaluated through observation, questionnaires and physiological measures over time.

1. Evaluating User Experience
Alistair Sutcliffe & Jenny Hart
Manchester Business School
University of Manchester
Manchester M15 6PB,
UK
a.g.sutcliffe@manchester.ac.uk
NUX July 2011
with thanks to Ons AlShamueli & Rabia Khan

2. Presentation Outline
1. Background to UX research
2. A framework for UI ‘attractiveness’ and user engagement
3. Some experiments on users’ perceptions of design quality
4. Defining User Experience
5. Evaluating User Experience

3. So What is User Experience (UX) ?
“A person’s perceptions and responses that result from the use or
anticipated use of a product, system or service”. ISO 2010
“The effect and affect produced by aesthetic experience, the meaning
we attach to the product, and the feelings and emotions produced”
No real agreement on definitions
Essentially beyond the functional- aesthetics, attractiveness – design
that excites and holds our attention- user engagement

4. UX Research- the quants
• Affordances, aesthetics and emotion key factors in design (Norman
2004)
• What is beautiful is usable (Tractinsky 2000, 2004); expressive &
classic aesthetics, pleasure,
• Beauty, goodness, pragmatics & hedonics- Attractdiff (Hassenzahl
2004, 2006, 2010)
• Aesthetic perception and interaction (Lingaard 2006, 2009,
Hartmann, Sutcliffe & de Angeli 2008) priming effects

5. UX Research- The Quals
(or contextualists)
• Jordon 1998, Pleasure in Products
• Mc Carthy & Wright 2005 Technology As Experience, 2010 User
Experience
• Dourish 2004 Where the Action Is Embodied Interaction
• Cockton 2008, Worth maps in UI design
• Hallnäs and Redström,2005, Presence and user experience
• Designers of User Experience- Gaver, Sengers, Forlizzi and many
others

6. UX Research- Viewpoints
1. There are fundamental cognitive constructs by which we perceive
User Experience- and hence can evaluate it (Hassenzahl,
Tractinsky)
2. User Experience can only be understood in context, each
experience is unique. It can only be understood by case study
analysis and heuristics (Mc Carthy & Wright + contextualists)
3. User Experience is best understood by design- cultural probes, etc
(Gaver and other designers)
4. UX is a cognitive process which can be modelled and measured
(Sutcliffe, de Angeli)

7. The Manchester Attractiveness
Framework
Usability
Content /
Services
Attractiveness
Customisability
Aesthetics
Reputation /
Identity
Attractiveness “Pleasing or appealing to the senses, arousing interest” OED

8. UX Experiments
• User perceptions of Aesthetics and UI Design qualities
- Comparing web sites +/- attractive design features (interactive
metaphors, dynamic media)
- Sutcliffe & de Angeli INTERACT 05, DIS 06
• Framing effects and Customisation on UX
- subject background, task scenario effects, customisation
- Hartmann, Sutcliffe & de Angeli CHI 07,08
• Avatars, Immersion and UX
- attractive chatterbots & persuasion
- comparing UX in immersive v. standard environments
(Khan, Sutcliffe & de Angeli, Sutcliffe & AlRayes 2010)

9. Menu Design style
Both sites- equivalent information, different UI styles

10. Summary of Differences
Dimensions Engaging style Traditional style
Usability - +
Aesthetic classic - +
Aesthetic + -
expressive
Information - +
quality
Engagement + -
Memory Content-based Interface-based
Overall
Preference 14 (50%) 14 (50%)

11. User Judgement: Effect of
Customisation
Mobile phone
News feed
application
2 versions
Good/poor aesthetics
Customised or
General content

12. Results
aesthetics manipulation content-fit manipulation
Measures Aesth++ Standard customised generic
Aesthetic classic + - n/a n/a
Aesthetic + - n/a n/a
expressive
Engagement + - n/a n/a
Usability = = = (+) = (-)
Look and Feel + - + -
Customisability = = + -
Content = = + -
Overall + - + -
Preference

13. Summary- Components of
Attractiveness
• Attractiveness is a complex mix of factors, but content and services
probably more important
• Judgement and choice very dependent on user background and task
• Usability is important but defects will be tolerated if overall experience
is positive (halo effect)
• Aesthetics is important but only in context <user background>
• Content and customisation important components in overall
preference, but if equal then other criteria come into play

14. Experiments on Avatars/ Virtual Characters
• Avatars (human like characters) influence preferences
- make information more credible
- persuade people more effectively than text/speech alone
• Attractive avatars are more effective than less attractive
- younger, females
- similarity to target audience
• Leverages “Computer as Social Actor” effect – we treat
representations of people on line like real people
(Reeves & Nass 1996)
Khan, Sutcliffe & De Angeli 2010

15. Attractiveness in Agents
Small differences in appearance make a huge differences in
attractiveness and persuasion

16. Avatars in virtual worlds increase engagement
(the Second Life experience)
more interaction improves UX ratings

17. Components of User Experience
• UX in the wide
- Experience throughout the Product life cycle
- Initial contact (aesthetics)
- Use (functionality, content)
- Customisation
- Support
• UX in the small (interaction, engagement)
-Presence- user as interactor
-Immersion in the interactive world
-Flow and engagement in the interactive world
-Social presence

18. Engagement and Interaction
Interaction with avatars is more engaging if …
- They adapt to the user
- Use surprise and are occasionally unpredictable

19. User Experience & Engagement
increases Arousal
Content +ve exp
Functionality Emotion
induces +ve
promotes
Interaction Media Presence
Pace Flow Human Image Immersion
& Voice
Complexity Change
3D Worlds
Avatars

20. Engagement and Attractiveness-
revised framework:
Usability
Content
services
High level
impression
Reputation
Design Aesthetics
Identity
Quality
Customis- Engagement
ability
Metaphor
Cost Benefit Presence Interaction
Flow
Interactive Need / Continuum
experience Specificity
All task / context / use dependent

21. UX over time
Compared 3 web sites
– IKEA avatar virtual shop assistant
– NIKE animation and customisation
– ALDI baseline
Tasks search + interact with features
Same tasks and sites 3 visits separated by 1 week
Hypotheses - character will improve engagement
- interaction (customisation) will improve
engagement
- effects stronger after more experience

22. IKEA- virtual character

23. NIKE customisation

24. ALDI Ad pop ups

25. Preferences & ratings
• No significant difference Nike- IKEA
• Preference and rating order the same 1,2 weeks later
• But rating of criteria changed
aesthetics, usability earlier, content, brand later
• Avatar (IKEA) not engaging but animation was
• Customisation (Nike) not engaging but animation was
• ALDI pop ups disliked

26. Evaluating User Experience
(Sutcliffe & Hart 2011)
• Observation (within session)
- user activity
- body posture, gestures, facial expressions
- attention
• Questionnaire (post session)
- existing scales- expressive aesthetics + new scales Presence
(from VR), Flow-engagement, Media, and immersion
- memory recall- salient features, episodes
• Physiological Measures (in session)
- heart rate
- GSR
- pupilometry & eye tracking

27. Observation
• Activity
- system logs or sampling: pace, critical incidents & breakdowns,
task v error repair actions
• Attention
- gaze on screen/in world v elsewhere
- could supplement with eyetracking but expensive
• Non Verbal Communication
- post session analysis from video records
- rate posture for arousal/excitement
- facial expression for emotion

28. Questionnaires I
General Engagement
• Rate your general mood after using the application (positive happy
… negative depressed)
• Rate the strength of your feelings/ emotions: Pleasure, Joy,
Surprise, Sadness, Anxiety, Worry, Fear, Frustration, Disgust.
Interaction/Flow
• Rate the pace of interaction (too slow, about right, too fast)
• How challenging was operating the interface? (too easy, about right,
too difficult)
Media CASA
• Did you notice any images of people? (not at all … very much)
• How attractive were the images used in the application?

29. Questionnaires II
Presence
• How natural did your interactions with the application feel?
• How aware were you of events occurring in the real world around
you?
Immersion
• How compelling was your sense of moving inside the interactive
world?
• How natural did the interactive world appear to be?
Social Presence
• How aware were you of the person you were communicating with?
• How well did the application communicate the identity of other
people?

30. Summary & Conclusions
• User Experience is multi-faceted and will change over time
• It can be measured/systematically evaluated
• UX is context (domain) dependent
• Our judgement of UX suffers from biases and framing effects
• UX components
- in session engagement: avatars, virtual worlds and user as
actor- motivate and attract
- across session: personalisation, utility, challenge and
adaptation

31. Thanks for your attention
For more information
• Sutcliffe, A.G. (2003). Multimedia and virtual reality: Designing multisensory
user interfaces. Mahwah NJ: Lawrence Erlbaum Associates.
• Sutcliffe, A. G. (2009). Designing for user engagement: aesthetic and
attractive user interfaces. In Carroll, J.M. (Ed), Synthesis lectures on human
centered informatics. San Rafael CA: Morgan Claypool.
• Sutcliffe A.G. & de Angeli A., (2005), Assessing interaction styles in web user interfaces. In
Proceedings of Human Computer Interaction INTERACT 05, Eds Costabile M.F and Paterno F.,
Rome Sept, 2005, Springer Verlag. pp 405-417.
• De Angeli, Sutcliffe A.G. & Hartmann J. (2006) Interaction, usability and aesthetics: What
influences users’ preferences? In Proceedings of DIS 2006, Designing Interactive Systems, ACM
Press.
• Hartmann J., Sutcliffe A.G. & de Angeli A. (2007), Investigating attractiveness in web user
interfaces. in CHI07, Human Factors in Computing Systems, San Jose, CA, ACM Press.
• Hartmann, J., Sutcliffe, A. G., & De Angeli, A. (2008). Framing the user experience: Information
biases on website quality judgement. In Proceedings of CHI-08. New York: ACM Press.
• Hartmann, J., Sutcliffe, A. G., & De Angeli, A. (2008). Towards a theory of user judgment of
aesthetics and user interface quality. ACM Transactions on Computer-Human Interaction, 15(4),
15-30.
ags@manchester.ac.uk