IUE14 Presentation - Studies UX Pros should know

1,022 views
952 views

Published on

Published in: Design, Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,022
On SlideShare
0
From Embeds
0
Number of Embeds
36
Actions
Shares
0
Downloads
0
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • 2
  • Source: http://www.telegraph.co.uk/finance/personalfinance/consumertips/9186513/National-Lottery-whats-the-luckiest-number.html
  • Compared to serial prototypers, parallel prototypers
    Designs out performed on all measures
    Click through rates
    Time spent on site
    Ratings by clients & professionals
    Generated more diverse designs
    Reported increased self-efficacy
    Serial prototypers reported negative responses to critique
    Experienced designers outperformed novices in performance, not diversity
  • Method
    Participants: 33 adults from Chicago
    Task: Filled out all 5 versions of the 15 field forms (Counterbalanced)
    Between subjects comparison
  • Method
    Participants: 33 adults from Chicago
    Task: Filled out all 5 versions of the 15 field forms (Counterbalanced)
    Between subjects comparison
  • 166 German university community members ranging from 15-64 yrs old
    (X = 27)

    Complete a form with 7 fields including personal information
    And password/login name.

    Instructions varied across participants.

  • 166 German university community members ranging from 15-64 yrs old
    (X = 27)

    Complete a form with 7 fields including personal information
    And password/login name.

    Instructions varied across participants.

  • 18 expert users
    Ages 18 - 47
    72% male, 18% female

    25 At-Risk Users
    Ages 28 – 77
    52% male, 48% female
    9 participants over 60 years old
    18 Low Literacy (REALM score under 60)
  • 18 expert users
    Ages 18 - 47
    72% male, 18% female

    25 At-Risk Users
    Ages 28 – 77
    52% male, 48% female
    9 participants over 60 years old
    18 Low Literacy (REALM score under 60)
  • 18 expert users
    Ages 18 - 47
    72% male, 18% female

    25 At-Risk Users
    Ages 28 – 77
    52% male, 48% female
    9 participants over 60 years old
    18 Low Literacy (REALM score under 60)
  • 18 expert users
    Ages 18 - 47
    72% male, 18% female

    25 At-Risk Users
    Ages 28 – 77
    52% male, 48% female
    9 participants over 60 years old
    18 Low Literacy (REALM score under 60)
  • 18 expert users
    Ages 18 - 47
    72% male, 18% female

    25 At-Risk Users
    Ages 28 – 77
    52% male, 48% female
    9 participants over 60 years old
    18 Low Literacy (REALM score under 60)

  • 30 students majoring in graphic design
    41 people majoring in something else

    Is the ADP party the second biggest or second smallest party?


    Attractiveness ratings 5 point scale Very unattractive to very attractive
    Information retrieval Is the ALP the second from the top or the second from the bottom
    Clarity rating 1-5 Very unclear to Very Clear
    Overall rating Extremely bad – Extremely good (1-10)
    Which three do you like best

  • Is the ADP party the second biggest or second smallest party?


    Attractiveness ratings 5 point scale Very unattractive to very attractive
    Information retrieval Is the ALP the second from the top or the second from the bottom
    Clarity rating 1-5 Very unclear to Very Clear
    Overall rating Extremely bad – Extremely good (1-10)
    Which three do you like best
  • Attractiveness ratings 5 point scale Very unattractive to very attractive
    Information retrieval Is the ALP the second from the top or the second from the bottom
    Clarity rating 1-5 Very unclear to Very Clear
    Overall rating Extremely bad – Extremely good (1-10)
    Which three do you like best
  • Attractiveness ratings 5 point scale Very unattractive to very attractive
    Information retrieval Is the ALP the second from the top or the second from the bottom
    Clarity rating 1-5 Very unclear to Very Clear
    Overall rating Extremely bad – Extremely good (1-10)
    Which three do you like best
  • Attractiveness ratings 5 point scale Very unattractive to very attractive
    Information retrieval Is the ALP the second from the top or the second from the bottom
    Clarity rating 1-5 Very unclear to Very Clear
    Overall rating Extremely bad – Extremely good (1-10)
    Which three do you like best
  • Attractiveness ratings 5 point scale Very unattractive to very attractive
    Information retrieval Is the ALP the second from the top or the second from the bottom
    Clarity rating 1-5 Very unclear to Very Clear
    Overall rating Extremely bad – Extremely good (1-10)
    Which three do you like best
  • Usability Testing Findings (What users focus on) 6 websites - 30 users

    Expert Review Findings (What UXers focus on) 14 experts* / 3 different ER strategies
    Collabora>ve heuris>c evaluaion
    Group Usability Expert Walkthrough
    Group Domain Expert Walkthrough (DEW)


    Found 907 problems
  • Usability Testing Findings (What users focus on) 6 websites - 30 users

    Expert Review Findings (What UXers focus on) 14 experts* / 3 different ER strategies
    Collabora>ve heuris>c evaluaion
    Group Usability Expert Walkthrough
    Group Domain Expert Walkthrough (DEW)


    Found 907 problems
  • Usability Testing Findings (What users focus on) 6 websites - 30 users

    Expert Review Findings (What UXers focus on) 14 experts* / 3 different ER strategies
    Collabora>ve heuris>c evaluaion
    Group Usability Expert Walkthrough
    Group Domain Expert Walkthrough (DEW)


    Found 907 problems
  • Usability Testing Findings (What users focus on) 6 websites - 30 users

    Expert Review Findings (What UXers focus on) 14 experts* / 3 different ER strategies
    Collabora>ve heuris>c evaluaion
    Group Usability Expert Walkthrough
    Group Domain Expert Walkthrough (DEW)


    Found 907 problems
  • IUE14 Presentation - Studies UX Pros should know

    1. 1. RESEARCH IN PRACTICE Findings usability professionals should know about Kath Straub, Usability.org
    2. 2. 2  
    3. 3. Knowing the research •  obviates design debates makes it unnecessary  
    4. 4. Knowing the research •  obviates design debates •  makes us more effective designers
    5. 5. Research Question: Does who we study influence what we find?
    6. 6. Findings If (Berkeley)preschoolers read about nutrition during snack time, they are more likely to eat more veggies. They will also understand more about why veggies are important (including details that were not in the reading.) Practitioner takeaways " •  When research findings don’t make sense, be dubious.   Citation:  Gripshover,  S.  J.,  &  Markman,  E.  M.  (2013).  Teaching  Young  Children  a  Theory  of  NutriHon:  Conceptual  Change  and   the  PotenHal  for  Increased  Vegetable  ConsumpHon.  Psychological  Science,  24(8),  1541–1553.  doi: 10.1177/0956797612474827  
    7. 7. Research Question: How many users do we need to test to be confident about our results?
    8. 8. Insight 1 If you don’t test anybody, you don’t learn anything
    9. 9. Insight 2 You hit diminishing returns on finding new usability problems at about the 5th user test
    10. 10. AssumpHons:     #  problems  found:  N(1-­‐(1-­‐λ)i)     N=  Total  #  of  usability  problems   λ  =  probability  of  finding  the  average  usability  problem   when  running  a  single,  average  subject  or  using  a  single,   average  Evaluator.  For  this  curve  λ  =  33%.   I  =  #  of  parHcipants  or  evaluators      
    11. 11. But,  it  depends.
    12. 12. Findings If you make assumptions about the % of problems an individual test participant or evaluator will find, you can model when you will reach diminishing returns for testing/evaluating. Practitioner Takeaways •  If the “takeaway” is that simple, you should probably re-evaluate for yourself. Citation: Faulkner, L. (2003). Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers : A Journal of the Psychonomic Society, Inc, 35(3), 379–83.
    13. 13. 5 participants 10 participants 15 participants
    14. 14. Findings With 5 participants, you find 85% the problems on average, but the range is 55%-98%. When you test 10, its 94%/82%. For 15, its 97% /90%. Practitioner Takeaways •  Target to test 10-12 participants, not 5. Citation: Faulkner, L. (2003). Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers : A Journal of the Psychonomic Society, Inc, 35(3), 379–83.
    15. 15. Research Question: Does how we test influence the feedback we get?
    16. 16. or  
    17. 17. Findings When you show more than one design to a usability testing participant, they can make contrastive comments. Testing multiple designs leads to more detailed, comparative and negative comments from participants Takeaways •  Show participants more than one design during usability tests and allow them to compare/contrast the user experiences. Citation: Tohidi, Maryam., Buxton, William., Baecker, Ronald., and Sellen, Abigail. . (2006) Getting the Right Design and the Design Right: Testing Many is Better Than One CHI 2006 Proceedings. Usability Methods. April 22-27, 2006. Montreal, Quebec, Canada.
    18. 18. Research Question: Does how we test influence the feedback we get-MOBILE?
    19. 19. Findings Treadmill tests and controlled walking tasks yield similar performance in mobile evaluations. Stopping/restarting is not ecologically valid on a treadmill. Takeaways Controlled walking tasks provide a better in-context evaluation. •  Feedback given Citation: Barnard, L., Yi, J. S., Jacko, J. A., & Sears, A. (2005). An empirical comparison of use-in-motion evaluation scenarios for mobile computing devices. International Journal of Human-Computer Studies, 62(4), 487–520. doi:10.1016/ j.ijhcs.2004.12.002
    20. 20. Research Question: Does how we get feedback influence our design creativity?
    21. 21. Findings Parallel prototyping leads to more effective designs and greater designer self efficacy Takeaways •  Feedback given on a single design tends to lock designers into a path or set. Create/Request feedback on more than one design •  Parallel prototyping promotes comparison without defensiveness: Forces designers to think of alternative designs so you don’t get attached to one early in the design process Citation: Dow, S., Glassco, A., & Kass, J. (2011). The effect of parallel prototyping on design performance, learning, and self-efficacy. ACM Conference on …, (September), 10. doi:10.1111/j.1432-2277.2009.00960.x
    22. 22. Research Question: How should we align labels?
    23. 23. 29  
    24. 24. 30   Fixations on " whitespace  
    25. 25. Findings People liked left- and right-aligned forms best and rate them easiest to fill out. People were most efficient (fewest fixations) on top-aligned forms. People looked longest at top- and in-field aligned forms. In-field alignment was rated most difficult. Flow was rated least appealing. Practitioner takeaways: This really doesn’t matter as much as we want to argue about it. But, consider what people are doing and using when you pick. " Citation: Bojko,  A.  A.,  &  Schumacher,  R.  M.  (n.d.).  Eye  Tracking  and  Usability  TesHng  in  Form  Layout  EvaluaHon,  1–13.   "
    26. 26. Research Question: How much instruction should we give in forms?
    27. 27. 35  
    28. 28. 36  
    29. 29. 37   No instructions   Examples   Description   Description + Examples  
    30. 30. 38   No instructions   Examples   Description   Description + Examples   Findings Participants made the most errors with no instructions. Descriptions resulted in fewer errors than examples. Giving descriptions and examples doesn’t help (significantly), but it doesn’t hurt either. Practitioner takeaways: Format examples are better than nothing. Descriptions work better than examples. Citation: Bargas-Avila, J., Orsini,S., Piosczyk, H., Urwyler, D., Opwis, K. (2011) Enhancing online forms: Use format specifications for fields with format restrictions to help respondents. Interacting with Computers, 23(1).
    31. 31. Research Question: How should you present instructions?
    32. 32. Visible   instrucHons   Hidden  Behind   Bueon   Hidden  Behind   Link    
    33. 33. Findings - How to present instructions •  “Expert” users and older users interact with instructions equally frequently whether they are hidden or not. •  Low literacy participants tended not to interact with instructions even when they need to (They didn’t help much, when they did.) •  Links draw more clicks that instruction icons. Practitioner Takeaways •  Provide visible instructions. •  If you must hide them, use links rather than icons to let users know they exist. " Reference: Alton,  N.,  Rinn,  C.  ,  Summers,  K.,  and    Straub,  K.,  (Forthcoming)  Using  Eye-­‐Tracking  and  Form  CompleHon  Data   to  OpHmize  Form  InstrucHons.  IEEE  –    IPPC  14.  Piesburgh,  PA  USA.  
    34. 34. Top  of   SecHon   Above   Field   Right   of  Field  
    35. 35. Findings - Where to put instructions •  Low literacy people and older participants made significantly more errors and missed more fields in the accordion presentation. •  Participants tended not to read page headers in the wizard. Practitioner Takeaways •  Place instructions above text input field. " Reference: Alton,  N.,  Rinn,  C.  ,  Summers,  K.,  and    Straub,  K.,  (Forthcoming)  Using  Eye-­‐Tracking  and  Form  CompleHon  Data   to  OpHmize  Form  InstrucHons.  IEEE  –    IPPC  14.  Piesburgh,  PA  USA.  
    36. 36. Research Question: How should you present long forms?
    37. 37. Findings - How to segment and present forms •  Low literacy users and older participants, and even expert users (to a lesser extent) miss more items in accordion designs. •  Wizards help lower literacy and older individuals. Practitioner Takeaways •  Avoid accordion presentations for presenting long forms. " Reference: Alton,  N.,  Rinn,  C.  ,  Summers,  K.,  and    Straub,  K.,  (Forthcoming)  Using  Eye-­‐Tracking  and  Form  CompleHon  Data   to  OpHmize  Form  InstrucHons.  IEEE  –    IPPC  14.  Piesburgh,  PA  USA.  
    38. 38. Research Question: Does the hamburger menu work?
    39. 39. Findings •  Mobile users click more frequently on an icon that says MENU. Practitioner Takeaways •  Use words when you can. " Reference:     Test 1: http://exisweb.net/mobile-menu-abtest Test 2: http://exisweb.net/menu-eats-hamburger
    40. 40. Research Question: Can plain language be credible and concise?
    41. 41.     The  trial  court  erred  in  giving  flawed  essenHal   elements  instrucHons  to  the  jury  and  thereby  denied   the  defendant  due  process  and  fundamental  fairness   since  it  is  error  to  give  the  jury,  within  the  essenHal   elements  instrucHons,  one  statement  containing   more  than  one  essenHal  element  of  the  crime  and   requiring  of  the  jury  simple  and  singular  assent  or   denial  of  that  compound  proposiHon,  fully  capable  of   disjuncHve  answer,  which  if  found  pursuant  to  the   evidence  adduced  would  exculpate  the  defendant.         The  trial  judge  erred  by  instrucHng  the  jury   to  affirm  or  deny  a  single  quesHon.  That   quesHon  included  all  the  major  elements  of   the  crime.  By  joining  all  the  major   elements,  the  judge  prevented  the  jury   from  acquinng  the  defendant  even  if  they   found  him  innocent  of  a  major   element  .This  error  denied  him  his  due-­‐ process  rights.    
    42. 42.     The  trial  court  erred  in  giving  flawed  essenHal   elements  instrucHons  to  the  jury  and  thereby  denied   the  defendant  due  process  and  fundamental  fairness   since  it  is  error  to  give  the  jury,  within  the  essenHal   elements  instrucHons,  one  statement  containing   more  than  one  essenHal  element  of  the  crime  and   requiring  of  the  jury  simple  and  singular  assent  or   denial  of  that  compound  proposiHon,  fully  capable  of   disjuncHve  answer,  which  if  found  pursuant  to  the   evidence  adduced  would  exculpate  the  defendant.         The  trial  judge  erred  by  instrucHng  the  jury   to  affirm  or  deny  a  single  quesHon.  That   quesHon  included  all  the  major  elements  of   the  crime.  By  joining  all  the  major   elements,  the  judge  prevented  the  jury   from  acquinng  the  defendant  even  if  they   found  him  innocent  of  a  major   element  .This  error  denied  him  his  due-­‐ process  rights.     Findings •  The plain language lawyer was rated easy to understand, trustworthy, logical and specific and concise by other lawyers. •  Lawyers and non-lawyer thought clients, juries and peers would understand him better •  Lawyers would be more satisfied with the plain language lawyer as their lawyer Practitioner Takeaways •  Plain language enhances credibility, even for professionals. " Reference:  Straub,  K.,  Mahaffey,  S.  and  Cheek,  A.  (Forthcoming).  Even  lawyers  want  to  understand  their  lawyers:  New   evidence  showing  plain  language  increases  lawyers’  credibility.  To  be  presented  at  Clarity  2014,  Antwerp,  Belgium.    
    43. 43. Research Question: Is infinite scrolling evil?
    44. 44. Homework:  hep://www.slideshare.net/danmckinley/design-­‐for-­‐conHnuous-­‐experimentaHon  
    45. 45. Homework:  hep://www.slideshare.net/danmckinley/design-­‐for-­‐conHnuous-­‐experimentaHon  
    46. 46. Research Question: Are infographics all that?
    47. 47. Aqer  this  recent  elecHon,  has  the  ALP  party  become   the  second  biggest  or  second  smallest  party?  
    48. 48. Standard accurate (abstract)   Standard inaccurate   Non-standard accurate   Non-standard inaccurate  
    49. 49. Findings •  Graphic designers prefer non-standard and pictorial visualizations. •  Laypeople prefer the abstract/accurate visualizations. •  Standard and abstract visualizations reduce interpretive response times. Practitioner Takeaways •  If understanding your visuals are important to your story, test them Reference  Petrie, H., & Power, C. (2012). Quispel, A., & Maes, A. (2014). Would you prefer pie or cupcakes? Preferences for data visualization designs of professionals and laypeople in graphic design. Journal of Visual Languages & Computing, 25(2), 107–116.
    50. 50. Research Question: Do UXers fixate (on things users don’t care about)?
    51. 51. Findings •  There is some overlap but also some non-overlap in the issues users focus on and the issues UXers focus on Practitioner Takeaways •  Be skeptical of your intuitions. " Reference:  Petrie,  H.,  &  Power,  C.  (2012).  What  do  users  really  care  about?  In  Proceedings  of  the  2012  ACM  annual   conference  on  Human  Factors  in  Compu@ng  Systems  -­‐  CHI  ’12  (p.  2107).  ACM  Press.  doi:10.1145/2207676.2208363  
    52. 52. Research Question: Are those cool new UX research methods something we should be planning for?
    53. 53. JUST  BECAUSE  ITS  SEXY  DATA,   DOESN’T  MEAN  ITS  USEFUL  
    54. 54. JUST  BECAUSE  ITS  SEXY  DATA,   DOESN’T  MEAN  ITS  USEFUL   Findings •  (Dead) Salmon respond differentially to human emotions Practitioner Takeaways •  Be skeptical, particularly of new/emerging/sexy/neuroscienc-y researchy methods. " Reference    Bennee,  Baird,  Miller,  and  Wolford  (2009)  Neural  correlates  of  interspecies  perspecHve  taking  in  the  post-­‐ mortem  AtlanHc  Salmon:  An  argument  for  mulHple  comparisons  correcHon.  

    ×