![Most known Methods…
• Usability Methods
• System Usability scale [SUS]
• Methods that were transposed from Usability
– Cognitive Walkthrough
– Think aloud
– Diaries
2016 58@ Sónia Sousa](https://image.slidesharecdn.com/ifi7155-eusa-130603084225-phpapp02/85/Ifi7155-DT-Lesson1-58-320.jpg)
Uploaded bySónia
Ifi7155.DT-Lesson1
The document discusses various concepts and methods for evaluating user experience (UX). It begins by distinguishing between usefulness, usability, and user experience. Usability focuses on effectiveness, efficiency, and satisfaction, while user experience includes emotions, beliefs, and other subjective factors. The document then covers specific UX evaluation methods like cognitive walkthroughs, heuristic evaluation, co-discovery testing, and wizard of oz testing. It emphasizes the importance of designing an evaluation protocol that uses a mix of inspection and testing methods tailored to the product or concept being evaluated. Finally, the document lists some commonly used UX metrics like the System Usability Scale, AttrakDiff questionnaire, and emotion cards. It stresses that evaluating user experience is
More Related Content
Ifi7155.DT-Lesson1
- 1. Evaluating the UserExperience Sónia Sousa & Mati Mõttus
- 2.
- 3. Evaluation • Before thinkingon EVALUATION, – We should consider reasoning upon what we should evaluate, right? Where should evaluation practices are valuable? 2016 3@ Sónia Sousa
- 4. ISO 9241-210 iterative design earlyfocus on users and tasks empirical measurement http://www.system-concepts.com/assets/images/usability/usability%20diagram%20for%20blog.jpg John D. Gould and Clayton Lewis. 1985. Designing for usability: key principles and what designers think. Communications of the ACM 28(3), 300-11 2016 4@ Sónia Sousa
- 5. Three main designparadigms 2016 5 Purpose Evaluat i-on Artefact Benefic i-aries User centered design Engineering Applied Arts Purpose Evalu- ation Benefici- aries Purpose Evalu- ation Artefact Benefici- aries @ Sónia Sousa
- 6.
- 7. Lesson 1 • Let’sthen take a look into Evaluation related concepts such as… • Usefulness, Usability and User experience • Evaluation Methods and Tools – Objects for Evaluation – Methods & categories & metrics – Data & Scales 2016 7@ Sónia Sousa
- 8. Evaluation Concepts: usefulness, usabilityand user experience
- 9. a good userexperience depends on both pragmatic and hedonic qualities
- 10. usefulness and usabilityare pragmatic
- 11.
- 12. a user experienceoccurs when a person interacts with a product…
- 13. Usability and userexperience • Product – Behaviour – Do goal • Metrics – Efficiency – Effectiveness – User satisfaction Usability testing is… a systematic experimental evaluation of the interactions Is easy to use? Is easy to learn? Is satisfying to use? 132016 @ Sónia Sousa
- 14. Systematic measure • Nowadayswhen a person interacts with a product… they look for more than just – Efficient, effective and cheap products. • They look for products that stimulate pleasure; and – Address our needs for social and individual differentiation. 2016 14@ Sónia Sousa
- 15. Hanssenzahl and Tractinsky (2006) User Context Content Kuniavsky(2003) Identity Jetter and Gerken (2010) Design results and marketing Marketing Nielsen, Norman and Tognazzini (2011) All aspects of the end-user’s interaction Services Roto, Law, Vermeeren & Hoonhout (2011) Interface Usability Information, interaction and identity An active and passive encounter with a system Characteristics of the system Time factor Information Interaction Product company 152016 @ Sónia Sousa
- 16. Usability • According tothe ISO standard – Usability is the extent to which a product or service can be used by specific users to achieve predefined goals in a specified context of use – The standard defines three main usability dimensions: • Effectiveness, • Efficiency, and • Satisfaction 2016 @ Sónia Sousa 16
- 17. Usability • Anyway, whenlooking into usability, we normally account for: – Effectiveness – Efficiency – Satisfaction – Learnability – Memorability • We name these the pragmatic qualities of a product or service 2016 @ Sónia Sousa 17
- 18. User Experience • Accordingto its ISO standard, user experience is a person’s perception and responses that result from the use or anticipated use of a product or service 2016 18@ Sónia Sousa
- 19. User Experience • Userexperience subsumes usability and includes includes – emotions, beliefs, preferences, perceptions, physical and psychological responses, behaviors and accomplishments that occur before, during and after use • We name these the effects of the hedonic, or pleasure related, qualities of a product or service 2016 19@ Sónia Sousa
- 20. Usefulness… • The degreeto which a product enables a user to achieve his or her goals: and – Assesses user’s willingness to use the product • If a system is easy to use, easy to learn, and even satisfying • But, this vector is most often overlooked during experiments and studies in the lab • As it addresses system-oriented design takes – Jeffrey Rubin 202016 @ Sónia Sousa
- 21. Efficiency… How quicklycan users perform tasks? – How fast It can be accomplished (accurately and completely) – Quantitative measure • Associated to % of total users • Represents: • Product behavior • Users expectations • Do exactly what user wont 212016 @ Sónia Sousa
- 22. Learnability… • How easyis it for users to accomplish basic tasks – For the first time – After stop using for a while • Represents – user’s competence • ability to operate the system – after some predetermined amount and period of training • ability of infrequent users to relearn the system – after periods of inactivity 222016 @ Sónia Sousa
- 23. Satisfaction… • Does theproduct meets or satisfy users needs – Quantitative and qualitative measures • captured through both written and oral questioning • Represents – user’s perceptions – user’s feelings – User’s opinions • Typically, users are asked – to rate and rank products that they try, – Asked to reveal causes and reasons for problems that occur 232016 @ Sónia Sousa
- 24. Accessibility • Describe theability to access and benefit of something • The degree to which a product, device, service, or environment is accessible by as many people as possible, without modification – The degree it can accommodate things that people can't easily change… • Focus often – on people with disabilities and their right of access to entities 242016 @ Sónia Sousa
- 25.
- 26. Evaluation • Before addressingthe evaluation protocol, – we should consider: 1. What is the purpose of evaluation? 2. What metrics will I need? – Is a measurements: a method to measuring something 3. What methods I can use? – a particular procedure for accomplishing or approaching something 4. What type of data we will need? 2016 26@ Sónia Sousa
- 27. Evaluation • UX evaluationdepends on what we focus on… – The most significant issues preventing users from accomplishing their goals – What works and what do users find frustrating – What are the most common errors or mistakes users are making – Assessing the improvements being made from one design interaction to the next – Identifying issues that are expected to remain even when the product is launched 2016 27@ Sónia Sousa
- 28. 2016 28 What isthe usage intention? Positive outcomes Motivation aspects Granularity measure methods Value Vs importance of use Low level experience measure Overall experience measure + + AttitudesIntentions What is the Usage attitude @ Sónia Sousa
- 29. 2016 29 EmotionsValue Perceptions UserAttitudesProduct Qualities Usage IntentionsUser Beliefs Before interaction During interaction After use + First impressions Mood State Why people like and use certain products What make people use a certain product @ Sónia Sousa
- 30. What to evaluate? 201630 Product Layout, utility, functionalityFirst impressions Hedonic qualities Emotions Mood state Value Perceptions Pleasure, Affective arousal symbolic aspects of the product User perceptions @ Sónia Sousa
- 31.
- 32. Metrics and methods •Issues-based metrics – What is an issue? • Anything that prevents task completion • Anything that takes someone off-course • Anything that creates some level of confusion • Anything that produces an error • Not seeing something that should be noticed • Assuming something is correct when it is not • Performing the wrong action • Misinterpreting some piece of content • Not understanding the navigation – How are they identified? • Using an inspection method • Eventually combined with some performance data analysis 2016 @ Sónia Sousa 32
- 33. 2016 @ SóniaSousa 33 Utility Usability Pleasure Intentions Social Value Efficiency Perceptions of interactions The product Affection Satifaction Social Value Mood State Relevance of the product User intention to use usefulness Trust Learnability Accessibility UX Asthetics Social Links
- 34. Metrics and methods •Issues-based metrics – After identified, issues usually classified according to their severity • Small impact on user experience, few users experiencing issue – Low severity • Small impact on user experience, many users experiencing issue – Medium severity • Large impact on user experience, few users experiencing issue – Medium severity • Large impact on user experience, many users experiencing issue – High severity 2016 @ Sónia Sousa 34
- 35. Metrics and methods •And others exist such as – Self-reported metrics, used for assessing satisfaction among other participant perceived measures – Behavioral and physiological metrics • of which eye-tracking is one of the most used ones as far as Web usability testing is concerned – Combined and comparative metrics • based on combinations of the previously mentioned siblings – And others such as… • Server logs • Card-sorting data – Open card sorting – Closed card sorting • Accessibility indicators 2016 @ Sónia Sousa 35
- 36. Metrics and methods 201636 using Task oriented Perceived usage Usability UX Prevent Experience Reflection Concern with product attributes and prevent errors Concern in build positive experiences resulted from the interaction with the product @ Sónia Sousa
- 37. How to…choose a specificmethod?
- 38. Evaluation methods • Inspectionmethods – These are methods where an expert evaluator inspects a product or service • Testing methods – These are methods where products and services are evaluated by testing them on real users 2016 38@ Sónia Sousa
- 39. Inspection methods • Thereare also several but we will address: – Cognitive walk-through – Heuristic evaluation 2016 39@ Sónia Sousa
- 40. Cognitive walk-through • Thepurpose is to verify if the paper prototype actually allows the fulfillment of the selected user stories – This is a within team activity that ensures that your prototype complies with the identified user stories 2016 40@ Sónia Sousa
- 41. Cognitive walk-through • Designersand developers of the product or service then walk through the steps as a group, asking themselves a set of questions at each step – Data is gathered during the walk-through, and afterwards a report of potential issues is compiled – Finally the evaluated proposition is redesigned to address the issues identified 2016 41@ Sónia Sousa
- 42. Heuristic evaluation • Thisis a type of evaluation ideally carried out by an expert. – It specifically involves evaluators examining the design and judging its compliance with recognized principles, such as Jakob Nielsen’s 10 usability heuristics • These evaluation methods are now widely taught and practiced in the new media sector, where products and services are often designed in a short space of time on a budget that may restrict the amount of money available to provide for other types of interface testing 2016 42@ Sónia Sousa
- 43. Testing methods • Thereare several but we will address: – Co-discovery – Wizard of Oz 2016 43@ Sónia Sousa
- 44. Co-discovery • Two usersattempt to perform tasks together while being observed – They are to help each other in the same manner as they would if they were working together to accomplish a common goal using the product – They are encouraged to explain what they are thinking about while working on the tasks 2016 44@ Sónia Sousa
- 45. Co-discovery • The designersand developers should refrain from explaining the design decisions and rather focus on getting the most of the pair of users tacking with your prototype – Note taking is fundamental and you should run this kind of test until no significant additional information is feed back into the design process 2016 45@ Sónia Sousa
- 46. Wizard of OZ •This is a testing approach built upon a paper device instead of using a working technological artifact – This king of testing involves systematic observation under controlled conditions to determine how well people can use a product or service – Rather than showing users a rough draft and asking, Do you understand this?, this kind of testing involves watching people trying to use something for its intended purpose 2016 46@ Sónia Sousa
- 47. Wizard of OZ •Setting up such a test involves asking the test subjects to recreate a set of user stories after being introduced to the underlying scenario – For example, to test the attachment function of an e-mail program, a scenario would describe a situation where a person needs to send an e-mail attachment, and ask him or her to undertake this task – The aim is to observe how people function in a realistic manner, so that developers can see problem areas, and what people like. 2016 47@ Sónia Sousa
- 48. Testing methods • Thiskind of evaluation should be repeated until no significant added value comes from bring in an additional subject – The usual number is 5, but other claim otherwise 2016 48@ Sónia Sousa
- 49. Pre and post-tests •When applying testing methods, pre-test and post-test questionnaires are also used to gather feedback on the product being tested – A common questionnaire is the 25 years old System Usability Scale 2016 49@ Sónia Sousa
- 50. How to…design an evaluationprotocol?
- 51. Evaluation protocol • Butthese methods are useless without an adequate evaluation protocol • While designing your evaluation protocol, you should take into account… – A mixed selection of inspection and testing methods; – The user stories supported by your concepts/prototype; and – The affordances of your concepts/prototype. 2016 51@ Sónia Sousa
- 52. Evaluation protocol • Wewill work on perfection our evaluation protocol throughout the course… – In all cases, evaluation protocols should be piloted to ensure that once they are being applied, you are actually focusing in assessing the prototype or concepts and not on solving evaluation protocol issues 2016 52@ Sónia Sousa
- 53. Now it’s upto you to make it happen
- 54. ???How??? • Answer to –what you wont to evaluate? – What are your evaluation needs? • Decide for instance – Product name • Digital e-Textbook service – Product Description • Web service, mobile, desktop and tablet application – Design stage • Concept design – Product representation • Functional prototypes – Purpose of evaluation • Find the best concept idea for implementing the service – Participants • Students, teachers and publishers – Time restriction • 3 weeks – Equipment and tools • Audio recorder, screen recording, assistant for note taking – Skills of researcher • HCI students 2016 4@ Sónia Sousa
- 55. …and that iswhy it should be carefully addressed
- 56. evaluating user experiencesis tricky but possible
- 57. and we willnow address possible approaches
- 58. Most known Methods… •Usability Methods • System Usability scale [SUS] • Methods that were transposed from Usability – Cognitive Walkthrough – Think aloud – Diaries 2016 58@ Sónia Sousa
- 59. Most known Methods… •Others are adaptation of – Extended usability testing – TRUE Tracking Realtime User Experience – UX Expert evaluation – Property checklists • Others are new – AttrakDiff - – Emocards, Emotion Cards, Emofaces – Reaction Cards 2016 59@ Sónia Sousa
- 60. SUS is aquick and dirty scale popular for end-of-test subjective assessments
- 61. System Usability Scale 201661@ Sónia Sousa
- 62.
- 63. 2016 63@ SóniaSousa
- 64.
- 65. AttrackDiff assesses theuser's feelings about a system with a questionnaire addressing both hedonic and pragmatic dimensions of UX
- 67.
- 68. How user perceiveproduct features • Aesthetics and interaction features • Layout, content, functionality – When before and during • Personal expectation and needs – Record of mouse activity and mood state – AttractDiff and EMG values – Kleiss, Lavies questionnaire 2016 68@ Sónia Sousa
- 69. What is theintention to use • Judgments of the product based on – Feelings towards the product 2016 69@ Sónia Sousa
- 70. First impressions • Howuser perceive product features – Aesthetic and symbolic aspects of products • Measurements – Visual and verbal tests 2016 70@ Sónia Sousa
- 71. Values & perceptions •How relevant users see the product? – Pleasure, challenge, needs, expressions • Mood boards • Perceived quality • Efficiency • Effectiveness • sensors 2016 71@ Sónia Sousa
- 72. Emotions & Moodstate – Positive outcomes • Emotions, joy, excitement, value – Affects – BMIS – MIS – RAS – Affect grid – SAM – PreEmo 2016 72@ Sónia Sousa
Editor's Notes
- #64 participants marks their current emotional state on a 2-dimensional 9x9 grid where arousal forms the y axis and pleasantness the x axis