Evaluating User Experience Alistair Sutcliffe & Jenny Hart Manchester Business School University of Manchester Manchester M15 6PB, UK firstname.lastname@example.org NUX July 2011 with thanks to Ons AlShamueli & Rabia Khan
Presentation Outline1. Background to UX research2. A framework for UI ‘attractiveness’ and user engagement3. Some experiments on users’ perceptions of design quality4. Defining User Experience5. Evaluating User Experience
So What is User Experience (UX) ?“A person’s perceptions and responses that result from the use or anticipated use of a product, system or service”. ISO 2010“The effect and affect produced by aesthetic experience, the meaning we attach to the product, and the feelings and emotions produced”No real agreement on definitionsEssentially beyond the functional- aesthetics, attractiveness – design that excites and holds our attention- user engagement
UX Research- the quants• Affordances, aesthetics and emotion key factors in design (Norman 2004)• What is beautiful is usable (Tractinsky 2000, 2004); expressive & classic aesthetics, pleasure,• Beauty, goodness, pragmatics & hedonics- Attractdiff (Hassenzahl 2004, 2006, 2010)• Aesthetic perception and interaction (Lingaard 2006, 2009, Hartmann, Sutcliffe & de Angeli 2008) priming effects
UX Research- The Quals (or contextualists)• Jordon 1998, Pleasure in Products• Mc Carthy & Wright 2005 Technology As Experience, 2010 User Experience• Dourish 2004 Where the Action Is Embodied Interaction• Cockton 2008, Worth maps in UI design• Hallnäs and Redström,2005, Presence and user experience• Designers of User Experience- Gaver, Sengers, Forlizzi and many others
UX Research- Viewpoints1. There are fundamental cognitive constructs by which we perceive User Experience- and hence can evaluate it (Hassenzahl, Tractinsky)2. User Experience can only be understood in context, each experience is unique. It can only be understood by case study analysis and heuristics (Mc Carthy & Wright + contextualists)3. User Experience is best understood by design- cultural probes, etc (Gaver and other designers)4. UX is a cognitive process which can be modelled and measured (Sutcliffe, de Angeli)
The Manchester Attractiveness Framework Usability Content / Services Attractiveness Customisability Aesthetics Reputation / IdentityAttractiveness “Pleasing or appealing to the senses, arousing interest” OED
UX Experiments• User perceptions of Aesthetics and UI Design qualities - Comparing web sites +/- attractive design features (interactive metaphors, dynamic media) - Sutcliffe & de Angeli INTERACT 05, DIS 06• Framing effects and Customisation on UX - subject background, task scenario effects, customisation - Hartmann, Sutcliffe & de Angeli CHI 07,08• Avatars, Immersion and UX - attractive chatterbots & persuasion - comparing UX in immersive v. standard environments (Khan, Sutcliffe & de Angeli, Sutcliffe & AlRayes 2010)
Menu Design styleBoth sites- equivalent information, different UI styles
Summary- Components of Attractiveness• Attractiveness is a complex mix of factors, but content and services probably more important• Judgement and choice very dependent on user background and task• Usability is important but defects will be tolerated if overall experienceis positive (halo effect)• Aesthetics is important but only in context <user background>• Content and customisation important components in overallpreference, but if equal then other criteria come into play
Experiments on Avatars/ Virtual Characters• Avatars (human like characters) influence preferences- make information more credible- persuade people more effectively than text/speech alone• Attractive avatars are more effective than less attractive - younger, females - similarity to target audience• Leverages “Computer as Social Actor” effect – we treatrepresentations of people on line like real people(Reeves & Nass 1996) Khan, Sutcliffe & De Angeli 2010
Attractiveness in AgentsSmall differences in appearance make a huge differences in attractiveness and persuasion
Avatars in virtual worlds increase engagement (the Second Life experience) more interaction improves UX ratings
Components of User Experience• UX in the wide- Experience throughout the Product life cycle- Initial contact (aesthetics)- Use (functionality, content)- Customisation- Support• UX in the small (interaction, engagement) -Presence- user as interactor -Immersion in the interactive world -Flow and engagement in the interactive world -Social presence
Engagement and InteractionInteraction with avatars is more engaging if …- They adapt to the user- Use surprise and are occasionally unpredictable
User Experience & Engagement increases Arousal Content +ve exp Functionality Emotion induces +ve promotes Interaction Media PresencePace Flow Human Image Immersion & VoiceComplexity Change 3D Worlds Avatars
Engagement and Attractiveness- revised framework: Usability Content services High levelimpression Reputation Design Aesthetics Identity Quality Customis- Engagement ability Metaphor Cost Benefit Presence Interaction FlowInteractive Need / Continuumexperience Specificity All task / context / use dependent
UX over timeCompared 3 web sites – IKEA avatar virtual shop assistant – NIKE animation and customisation – ALDI baseline Tasks search + interact with features Same tasks and sites 3 visits separated by 1 week Hypotheses - character will improve engagement - interaction (customisation) will improve engagement - effects stronger after more experience
Preferences & ratings• No significant difference Nike- IKEA• Preference and rating order the same 1,2 weeks later• But rating of criteria changed aesthetics, usability earlier, content, brand later• Avatar (IKEA) not engaging but animation was• Customisation (Nike) not engaging but animation was• ALDI pop ups disliked
Evaluating User Experience (Sutcliffe & Hart 2011)• Observation (within session) - user activity - body posture, gestures, facial expressions - attention• Questionnaire (post session) - existing scales- expressive aesthetics + new scales Presence (from VR), Flow-engagement, Media, and immersion - memory recall- salient features, episodes• Physiological Measures (in session) - heart rate - GSR - pupilometry & eye tracking
Observation• Activity - system logs or sampling: pace, critical incidents & breakdowns, task v error repair actions• Attention - gaze on screen/in world v elsewhere - could supplement with eyetracking but expensive• Non Verbal Communication - post session analysis from video records - rate posture for arousal/excitement - facial expression for emotion
Questionnaires IGeneral Engagement• Rate your general mood after using the application (positive happy … negative depressed)• Rate the strength of your feelings/ emotions: Pleasure, Joy, Surprise, Sadness, Anxiety, Worry, Fear, Frustration, Disgust.Interaction/Flow• Rate the pace of interaction (too slow, about right, too fast)• How challenging was operating the interface? (too easy, about right, too difficult)Media CASA• Did you notice any images of people? (not at all … very much)• How attractive were the images used in the application?
Questionnaires IIPresence• How natural did your interactions with the application feel?• How aware were you of events occurring in the real world around you?Immersion• How compelling was your sense of moving inside the interactive world?• How natural did the interactive world appear to be?Social Presence• How aware were you of the person you were communicating with?• How well did the application communicate the identity of other people?
Summary & Conclusions• User Experience is multi-faceted and will change over time• It can be measured/systematically evaluated• UX is context (domain) dependent• Our judgement of UX suffers from biases and framing effects• UX components - in session engagement: avatars, virtual worlds and user as actor- motivate and attract - across session: personalisation, utility, challenge and adaptation
Thanks for your attention For more information• Sutcliffe, A.G. (2003). Multimedia and virtual reality: Designing multisensory user interfaces. Mahwah NJ: Lawrence Erlbaum Associates.• Sutcliffe, A. G. (2009). Designing for user engagement: aesthetic and attractive user interfaces. In Carroll, J.M. (Ed), Synthesis lectures on human centered informatics. San Rafael CA: Morgan Claypool.• Sutcliffe A.G. & de Angeli A., (2005), Assessing interaction styles in web user interfaces. In Proceedings of Human Computer Interaction INTERACT 05, Eds Costabile M.F and Paterno F., Rome Sept, 2005, Springer Verlag. pp 405-417.• De Angeli, Sutcliffe A.G. & Hartmann J. (2006) Interaction, usability and aesthetics: What influences users’ preferences? In Proceedings of DIS 2006, Designing Interactive Systems, ACM Press.• Hartmann J., Sutcliffe A.G. & de Angeli A. (2007), Investigating attractiveness in web user interfaces. in CHI07, Human Factors in Computing Systems, San Jose, CA, ACM Press.• Hartmann, J., Sutcliffe, A. G., & De Angeli, A. (2008). Framing the user experience: Information biases on website quality judgement. In Proceedings of CHI-08. New York: ACM Press.• Hartmann, J., Sutcliffe, A. G., & De Angeli, A. (2008). Towards a theory of user judgment of aesthetics and user interface quality. ACM Transactions on Computer-Human Interaction, 15(4), 15-30. email@example.com
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.