A brief overview of ux research


Published on

Presents an introduction to some basic metrics for usability and some current trends in UX evaluation methods. Includes some indicative examples from UX evaluation studies conducted by the author

Published in: Design, Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

A brief overview of ux research

  1. 1. A brief intro to UX Research Dr. Panagiotis Zaharias 13/5/2014 @ CoLab Athens
  2. 2. The UX Mantra To measure is to know. If you can’t measure it, you can’t improve it…
  3. 3. Overview of the talk Examples of UX evaluation studies Current trends in UX evaluation UX Research: Main Concepts
  4. 4. Main Concepts in UX Research First things first
  5. 5. The Ancestor: Human Computer Interaction  Human – Computer Interaction, is “a discipline concerned with the design, evaluation and implementation of interactive computing systems for human use and the study of major phenomena surrounding them’’.
  6. 6. User friendly? Usable? …  Once upon a time we were talking about “user friendly” systems… … so general and vague  Then usability emerged and was originally derived from the term ‘‘user friendly’’ … but usability is still a general and vague term  Many definitions and different approaches to how usability should be measured
  7. 7. What is usability anyway?  The ISO definition (9241-11)  Usability is defined as: ‘‘the extent to which a product can be used by specified users to achieve specified goals with  effectiveness; the extent to which the intended goals of use are achieved,  efficiency; the resources that have to be expended to achieve the intended goals and  satisfaction; the extent to which the user finds the use of the product acceptable, in a specified context of use’’.
  8. 8. Examples of usability measures (1)  Measures of effectiveness:  Task completion (number of correct tasks, number of tasks where users failed to finish within a set time, number of tasks where users gave up)  Accuracy measures (quantify the number of errors users make during the process of completing tasks)  Recall measures (how much information users can recall after the use of an interface)  Quality of outcome (attempts to measure the outcome of the tasks, measures of understanding e.g. tests of what has been learned from an e-learning system)
  9. 9. Examples of usability measures (2)  Measures of efficiency  Time (refers to measures of how long users take to complete tasks)  Input rate (e.g. words per minute)  Mental effort (mental resources spend on interaction, e.g. heart rate variability, subjective time estimation)  Usage patterns (number of times a certain action has been performed, how much information users access when solving tasks, deviation from the optimal solution etc.)
  10. 10. Examples of usability measures (3)  Measures of satisfaction  Preference measures capture which interface users prefer using  Specific attitudes towards the interface (liking, fun, annoyance etc.), toward the content of interface (quality of information, organization of information etc.)  Perception of outcomes (users’ assessment of their performance, users’ perception of learning, users’ confidence in the solution to tasks)
  11. 11. From Usability to User Experience (1)  UX - A relatively new field - Lack of unique definition…  But let’s take for example the ISO definition on UX: “A person's perceptions and responses that result from the use or anticipated use of a product, system or service”
  12. 12. From Usability to User Experience (2) Usability • Focused on optimizing task efficiency • Removing problems in the user interface • Studying interactive products mainly used for work UX • Dynamic context- dependent and subjective • Focuses also on hedonic qualities of use (identification, stimulation, attractiveness etc.)
  13. 13. Current trends in UX evaluation
  14. 14. As we shift to User Experience… Challenge#1: Need for new approaches and methods  Existing methods are somewhat unable -it is claimed- to capture experience  Challenge#2:The choice of UX evaluation method  Qualitative or quantitative methods?  Only few studies combine qualitative and quantitative methods
  15. 15. What Dimensions of Experience Are Studied? Usability • Effectiveness • Efficiency • Satisfaction UX • Emotions and affect • Enjoyment • Aesthetics • And many more…
  16. 16. What Dimensions of Experience Are Studied? Source: Bargas- Avila and Hornbaek, 2012
  17. 17. How do we measure UX?  What are the methods applied in UX  Qualitative (50 %), quantitative (33%), combination (17%)  How UX data are collected?  Questionnaires are still the dominant UX evaluation method (53%). Source: Bargas- Avila and Hornbaek, 2012
  18. 18. Popular UX evaluation methods Source: Bargas- Avila and Hornbaek, 2012
  19. 19. When to use which UX evaluation methods Decision Qualitative vs. Quantitative Attitudinal vs. Behavioral Context of system/ product use Source: http://www.nngroup.com/articles/which-ux-research-methods/
  20. 20. The Attitudinal vs. Behavioral Dimension to understand, measure, or inform change of people's stated beliefs e.g. self-reported methods such as surveys, interviews, focus groups, etc. Attitudinal to understand "what people do" with minimal interference from the method itself e.g.A/B testing, observation and video recording, first click testing, eyetracking, etc. Behavioral Source: http://www.nngroup.com/articles/which-ux-research-methods/
  21. 21. The Qualitative vs. Quantitative Dimension - the data is usually being gathered directly - they are much better suited for answering question about why or how to fix a problem Qualitative - the data is gathered indirectly, through an instrument, such as a survey or a web server log etc. - they do a much better job answering how many and how much type of questions. Quantitative Source: http://www.nngroup.com/articles/which-ux-research-methods/
  22. 22. The Context of Use Dimension  This dimension can be described by the following options:  Natural or near-natural use of the system/product  Scripted use of the system/product  Not using the system/product during the study  A hybrid of the above Source: http://www.nngroup.com/articles/which-ux-research-methods/
  23. 23. What about the use of UX evaluation methods across the development phases ?
  24. 24. UX evaluation methods across development phases Product Development Phase Strategize Optimize Assess Goal: Inspire, explore and choose new directions and opportunities Inform and optimize designs in order to reduce risk and improve usability Measure product performance against itself or its competition Approach: Qualitative and Quantitative Mainly Qualitative (formative) Mainly Quantitative (summative) Typical methods: Ethnographic field studies, focus groups, diary studies, surveys, data mining or analytics etc. e.g. cardsorting, field studies, observation, paper prototype and usability studies, etc. e.g. Usability benchmarking, surveys, A/B testing, etc. Source: http://www.nngroup.com/articles/which-ux-research-methods/
  25. 25. Some examples from UX research studies
  26. 26. Walls of Nicosia  The context:  "TheWalls of Nicosia” is an interactive application that runs on a multi- touch table at the Leventeio Museum, Nicosia  Students from elementary schools as museum visitors  Goal:To conduct a comparative summative evaluation  Methods used:  Multiple choice knowledge tests  UX questionnaires (FunToolkit)  Observation  Indicative results:  Mainly quantitative analysis (pre and post test analysis, t-tests)
  27. 27. UX questionnaire for children: the FunToolkit (Read, 2007) The Again-Again table The Smileyometer Source: http://hrast.pef.uni-lj.si/docs/research/Serious%20games/evaluation_paper_Janet.pdf
  28. 28. Long term UX evaluation of League of Legends (1)  The context:  The well known MMOG “League of Legends”  Players were students at TEI of Kavala  Goal: to identify how UX of players changed over time  Methods used:  An adaptation of UX Curve  Self reported method by drawing curves that depict changes in UX over time
  29. 29. Long term UX evaluation of League of Legends (2)  Indicative results: Reason Categories Positive Negative General UX 43 14 Ease of use 41 26 Immersion 43 10 Social interaction 38 12 Engagement 32 10 Total 197 72 General UX Curve for each player Number of reasons for UX dimensions
  30. 30. Heuristic Evaluation of LMS  The context:  A well known cloud-based Learning Management System platform  Goal: to identify usability problems and possible areas for UX improvement  Methods used:  Heuristic evaluation  Heuristic evaluation protocol adapted by Mehlenbacher et al. (2005)  Indicative results:  An evaluation report that contains the identified usability and UX problems, their severity, and relevant redesign suggestions
  31. 31. Typical structure of the report of heuristic evaluation  A numeric identifier of the problem  A short description of the problem (and likely difficulties for the user)  Specific context (defined as location of the problem in the interface): we identified this generically in terms of affecting the:  Structure of the system, including layout, navigation, and basic help functions such as print, help, etc.  Content, including actual content for study, exploration, examples, exercises, and so on  Global issues, i.e., affecting both structure and learning content  The heuristic(s) used  The severity rate: severe, moderate, and minor
  32. 32. First Click testing with mockups  The context:  A project for the design of new MOOC interfaces  Goal: to create, inform and optimize MOOC interface design  Methods used:  Interviews and first click testing with users
  33. 33. First Click testing - indicative results
  34. 34. First Click testing - indicative results 0 2 4 6 8 10 12 Ραβδόγραμμα εργασιών Σωστές εργασίες Λανθασμένες Εργασίες
  35. 35. Some UX resources:  World’s premier magazine for both academics and industry:  http://interactions.acm.org/  A comprehensive list of UX methods:  http://www.allaboutux.org/  An indicative list of UX tools:  http://www.smashingmagazine.com/2011/10/20/comprehensive -review-usability-user-experience-testing-tools/  Other sources on the web:  http://uxmag.com/  http://www.usability.gov/  http://www.uxmatters.com/index.php
  36. 36. Key takeaways: You can’t afford NOT to evaluate UX Always use a combination of methods Don’t forget Usability Do it properly…
  37. 37. Thank you! Let’s keep in touch:  Web: about.me/panagiotis_zaharias  Email: pz@aueb.gr  LinkedIn: gr.linkedin.com/pub/panagiotis-zaharias/0/bb6/a91  Twitter: twitter.com/pan_sugarman