Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Eye Tracking the UX of Mobile: What You Need to Know

5,506 views

Published on

workshop at MoDevUX 2015 on eye tracking and mobile. There should be a video up soon too!

Published in: Technology

Eye Tracking the UX of Mobile: What You Need to Know

  1. 1. Eye tracking the UX of mobile: What you need to know Jen Romano Bergstrom UX Researcher Facebook @romanocog 1 Jon Strohl UX Researcher Fors Marsh Group @jonstrohl March  23,  2015   ModevUX    |    Arlington  VA  
  2. 2. We will be answering these questions! 2 Ø  Why is mobile important? Ø  What makes mobile different? Ø  Do I need to test my design? •  How do I get started? •  What are the benefits? Ø  Do I need to include eye tracking? •  How do I get started? •  What are the benefits? •  Can you show me how it’s done? •  Are there any examples?
  3. 3. The rise of mobile 3 @romanocog @jonstrohl #modevux
  4. 4. Mobile internet usage continues to increase 4
  5. 5. Mobile retail usage increases 5 @romanocog @jonstrohl #modevux
  6. 6. App usage continues to increase 6
  7. 7. 7 What we design for…and reality @romanocog @jonstrohl #modevux Krug, S. (2000) Don’t Make Me Think. Pearson Education *NEW 3rd Edition (2014)* @skrug
  8. 8. 8 8 Ø  Controlled  environment   Ø  All  par>cipants  have  the   same  experience   Ø  Record  and  communicate   from  control  room   Ø  Observers  watch  from   control  room  and  provide   addi>onal  probes  (via   moderator)  in  real  >me   Ø  Incorporate  physiological   measures  (e.g.,  eye   tracking,  EDA)   Ø  No  travel  costs   LABORATORY   REMOTE   IN  THE  FIELD   Ø  Par>cipants  tend  to  be   more  comfortable  in   their  natural   environments   Ø  Recruit  hard-­‐to-­‐reach   popula>ons  (e.g.,   children,  doctors)   Ø  Moderator  travels  to   various  loca>ons   Ø  Bring  equipment  (e.g.,   eye  tracker)   Ø  Natural  observa>ons   Ø  Par>cipants  in  their   natural  environments   (e.g.,  home,  work)   Ø  Use  video  chat   (moderated  sessions)  or   online  programs   (unmoderated)   Ø  Conduct  many  sessions   quickly   Ø  Recruit  par>cipants  in   many  loca>ons  (e.g.,   states,  countries)   > FMG Capabilities Where to test
  9. 9. 9 When to test Iterative Process Throughout Development @romanocog @jonstrohl #modevux
  10. 10. 10 Research goals and plan Form hypotheses Will eye tracking add value? Design the study Planning the test @romanocog @jonstrohl #modevux
  11. 11. 11 Who is involved SATQ Items Other materials Debriefing Reports . . . . . . . “Make it a spectator sport” @skrug @romanocog @jonstrohl #modevux Project Managers UX Designers Executives Engineers Researchers
  12. 12. User Experience (UX) Defined 12 @romanocog @jonstrohl #modevux Adapted from Morville, P. Semantic Studios. http://semanticstudios.com/user_experience_design/
  13. 13. A combination of measures are used in UX research. @romanocog @jonstrohl #modevux Using multiple measures allows researchers to capture the most detailed and complete picture of the user experience.
  14. 14. 14 OBSERVATIONAL First click accuracy Task accuracy Time on page/task Selection/click behavior Conversion rate SELF-REPORT Satisfaction ratings Difficulty ratings Think-aloud protocol Debriefing interview UX Measures @romanocog @jonstrohl #modevux
  15. 15. 15 http://blog.thomsonreuters.com/wp-content/uploads/2013/01/smartphones1.jpg Lots of sizes to consider… @romanocog @jonstrohl #modevux
  16. 16. 16 http://armodilo.com/support/supported-tablets/ Don’t forget about tablets! @romanocog @jonstrohl #modevux
  17. 17. …our thumbs are only so big 17 http://gizmodo.com/how-to-design-for-thumbs-in-the-era-of-huge-screens-1636955848
  18. 18. …and people use them differently. 18 @romanocog @jonstrohl #modevux Hoober (2013). UX Matters. http://www.uxmatters.com/mt/archives/2013/02/how-do-users-really-hold-mobile-devices.php
  19. 19. Eye tracking and attention 19 @romanocog @jonstrohl #modevux @agabojko Bojko, Aga (2013). Designing the User Experience. Rosenfeld Media
  20. 20. User Experience (UX) Defined 20 Eye tracking helps to explain how usable, engaging, and desirable a product is. @romanocog @jonstrohl #modevux
  21. 21. 21 OBSERVATIONAL First click accuracy Task accuracy Time on page/task Selection/click behavior Conversion rate PHYSIOLOGICAL Eye movements Electrodermal activity (EDA) Pupil dilation UX Measures @romanocog @jonstrohl #modevux SELF-REPORT Satisfaction ratings Difficulty ratings Think-aloud protocol Debriefing interview
  22. 22. User Experience (UX) Measures •  Self-report metrics tell us why participants focus on certain site aspects.   22 •  Eye tracking tells us what, how long, and how often participants focus on design elements. •  The combination of self-report, observational, and implicit data allows us to better understand perceptions and behavior.   •  Observational metrics tell us how participants navigate and interact.   We do not use eye tracking in isolation. @romanocog @jonstrohl #modevux
  23. 23. 23 Eye Tracking Considerations @romanocog @jonstrohl #modevux @agabojko Bojko, Aga (2013). Designing the User Experience. Rosenfeld Media
  24. 24. User Experience Testing + Eye Tracking 24 UX Testing UX Testing with Eye Tracking Data while completing tasks Mouse movements, clicks, comments Scan paths, dwell time, first fixation, re-fixations. Eye-tracking helps determine what led to these behaviors and comments Findings Usability problems are detected A more thorough understanding of usability issues and the sources of these issues Recommendations Quality recommendations based on available data More insightful and precise recommendations @romanocog @jonstrohl #modevux
  25. 25. Eye-Tracking Metrics ENGAGEMENT Number of fixations Total dwell time Percentage of time on an area FINDABILITY Time to first fixation Number of fixations prior to first fixation PROCESSING Fixation durations WORKLOAD/EXCITEMENT Pupil dilation PROCESSING ORDER Gaze path COMPREHENSION Repeat fixations 25
  26. 26. 26 Data Visualization– Heat & Opacity Maps @romanocog @jonstrohl #modevux
  27. 27. 27 Data Visualization– Gaze Plots @romanocog @jonstrohl #modevux
  28. 28. 28 Modern eye tracking @romanocog @dcaapor Eye Tracking in the Past
  29. 29. 29 Modern eye tracking Modern Eye Tracking
  30. 30. Mobile Device Stand 30
  31. 31. Glasses 31
  32. 32. Shopping Research 32
  33. 33. Driving Research 33
  34. 34. Design Evaluation Research 34
  35. 35. Moderator & Participant 35 Slide from: Alayoubi, L. & Romano Bergstrom, J. (2014). Paying for Education: User-Centered Research on the Paying for College Website and Tools. Presentation at edUi, Richmond, VA, Sept 2014. Example Methodology – Formative Evaluation Eye Tracker Participants: •  N = 9 •  Mix of gender, ethnicity, income. •  Formative evaluation with a single web prototype. User Experience Testing session: •  Participants read a description of the study. •  The moderator gave instructions and calibrated the eye tracker. •  Participants completed a set of tasks provided individually by the moderator. •  End-of-session satisfaction questionnaire. •  Debriefing interview.
  36. 36. 36 Formative Studies Ø Are <10 participants appropriate? •  Very often. Typical for finding large usability problems. •  Problem occurrence is 25% or higher. •  Problem discovery goal is 75% or less. Ø Are >20 participants appropriate? •  Not as common, but sometimes used when testing more mature applications or trying to find smaller, less frequent issues. •  Problem occurrence is 10% or higher. •  Problem discovery goal is 90%. Ø If eye tracking, over recruit by ~20% 1.Calculations derived from: Sauro, J., & Lewis, J. R. (2012). Quantifying the user experience: Practical statistics for user research. Elsevier. Bojko, A. (2013). Eye Tracking the User Experience. Rosenfield Media. Determining Sample Size1 @romanocog @jonstrohl #modevux
  37. 37. UX Test with Eye Tracking Demo 37 @romanocog @jonstrohl #modevux FLEET •  Beta •  Flyfleet.org Features: •  Real-time flight status updates •  Let friends and family track your flight •  Get delay times at check-in and security •  Find stores and restaurants near you Fleet was designed and developed by Rob Youmans, Daniel Gartenburg, Melissa Smith, Peter Lee, Jordan Higgins, Octavian, and Vincent Mamo
  38. 38. 38 Example Methodology – Summative Evaluation Participants: •  N = 74 | Average Age = 37 •  Mix of gender, ethnicity, income. •  Summative Evaluation with a between groups design. •  New, Old, Prototype, and Bilingual media diaries. User Experience Testing session: •  Participants read a description of the study. •  The moderator gave instructions and calibrated the eye tracker. •  Participants completed Steps 1-5 in the diary at their own pace. •  End-of-session satisfaction questionnaire. •  Debriefing interview. Eye Tracker Moderators worked from another room. Control Room Slide from: Walton, L., Romano Bergstrom, J., Hawkins, D. & Pierce, C. (2014). User Experience and Eye-Tracking Study: Paper Diary Design Decisions. Paper presentation at the American Association for Public Opinion Research (AAPOR) Conference, Anaheim, CA, May 2014.
  39. 39. 39 1.Calculations derived from: Sauro, J., & Lewis, J. R. (2012). Quantifying the user experience: Practical statistics for user research. Elsevier. Bojko, A. (2013). Eye Tracking the User Experience. Rosenfield Media. Summative Studies Ø Are <10 participants appropriate? •  Very rarely, but sometimes when trying to detect very large differences. •  Difference between groups is 60% or more. •  Assuming our confidence goal is 90%. •  Assuming our power is 80%. Ø Are >20 participants appropriate? •  Yes, very often. Typical for detecting moderate differences with confidence. •  Difference between groups is 30% or more. •  Assuming our confidence goal is 90%. •  Assuming our power is 80%. Ø If eye tracking, over recruit by ~20% Determining Sample Size1 @romanocog @jonstrohl #modevux
  40. 40. Example Methodology – Naturalistic environment 40 Participants: •  N = 8 •  Mix of gender, income. •  Participants use their own device and Internet connectivity. User Experience Testing session: •  Sessions occurred in participant’s home. •  The moderator gave instructions. •  Sessions began with an interview about how participants use the Internet. •  Participants completed a set of tasks provided individually by the moderator. •  Debriefing interview.
  41. 41. What can we learn from eye tracking?
  42. 42. 42 Fixation count heat map of 10 seconds of interaction with the page. 20 participants included.* Fixation count heat map of 10 seconds of interaction with the page. 10 Novice participants included.* Fixation count heat map of 10 seconds of interaction with the page. 9 Experienced participants included.* All Participants Novice Experienced *Heatmaps include participants who located the link from the Refunds page only. Fixation count refers to the number of times the eye fixates, or directs itself towards, an area of the page. Novice and experienced users focus on different elements.
  43. 43. 43 Making something large does not mean people will notice it. @romanocog @jonstrohl #modevux
  44. 44. 44 New/Old DVR Grid New/Old DVR Grid Prototype Grid Grids that match users’ mental model are easier to process. Walton, L., Romano Bergstrom, J., Hawkins, D., & Pierce, C. (2014). User experience testing and eye tracking inform paper diary design. Proceedings from the Human Computer Interaction International Conference, June 2014, Crete, Greece. @romanocog @jonstrohl #modevux
  45. 45. 45 Gaze plot of 60 seconds of engagement from four participants on the Loan Rehabilitation page who scanned the text without reading it thoroughly. Gaze plot of 60 seconds of engagement from four participants on the Contact Your Servicer page who read most of the text. People do not read dense text. @romanocog @jonstrohl #modevux
  46. 46. 46 The text within the orange circle indicates the information to enter on the page. The red circled area is where the user enters the ID number. The participant skips the instructions and introductory text and moves on to the actionable areas. People do not read instructions. He, J., Siu, C., Chaparro, B. & Strohl, J. (2014). Mobile. In J. Romano Bergstrom & A. Schall (Eds.), Eye Tracking in User Experience Design. San Francisco, CA: Morgan Kaufmann. @romanocog @jonstrohl #modevux
  47. 47. 47 If the mandatory fields were not completed, participants received an error message that was not helpful. This gaze plot begins after the participant clicks “Ok.” The participant looks all over the screen, searching for the missing mandatory fields. The error message did not indicate which fields were missing. People need to know what to do next He, J., Siu, C., Chaparro, B. & Strohl, J. (2014). Mobile. In J. Romano Bergstrom & A. Schall (Eds.), Eye Tracking in User Experience Design. San Francisco, CA: Morgan Kaufmann. @romanocog @jonstrohl #modevux
  48. 48. Activity 48
  49. 49. Awesome! I want to use eye tracking. What’s next?
  50. 50. 50 Eye-Tracking Decision Progression Research goals and plan Form hypotheses Will eye tracking add value? Design the study Collect data Clean & prepare data Visualize qualitative data Analyze quantitative data
  51. 51. 51 Evidence: Younger and middle-age social media users access the sites/apps largely through smartphones, whereas older users access largely through desktop computers. It is important to test across user groups and across devices.   Older AdultsMiddle Age AdultsYounger Adults People  fixate  elements  of  displays  differently  across  devices.   Different user groups access via different interfaces @romanocog @jonstrohl #modevux
  52. 52. 52 Android App HomepageTablet Web HomepageTablet App Homepage People  fixate  elements  of  displays  differently  across  devices.  People attend to elements of displays differently across devices Bristol, K., Romano Bergstrom, J. & Link, M. (2014). Eye Tracking the User Experience of a Smartphone and Web Data Collection Tool. Paper presentation at the AAPOR Conference, Anaheim, CA, May 2014. @romanocog @jonstrohl #modevux
  53. 53. Thank you! Jen Romano Bergstrom UX Researcher Facebook @romanocog JenRB@fb.com Jon Strohl UX Researcher Fors Marsh Group @jonstrohl jstrohl@forsmarshgroup.com March  23,  2015   ModevUX    |    Arlington  VA  
  54. 54. 54 Website  www.forsmarshgroup.com       LinkedIn  hXp://www.linkedin.com/company/fors-­‐marsh-­‐group         Blog    www.forsmarshgroup.com/index.php/blog       TwiXer  @forsmarshgroup          

×