The Five-Second Rules: Ensuring a Healthy Five-Second Test

2,483 views

Published on

UXPA 2013 Annual Conference Wednesday July 10, 2013 4:30pm - 5:30pm ET by Paul Doncaster

Online five-second testing tools promise valuable response data that that can inform UX designs, but that value can be compromised by ignoring the restrictions of the method and designing the tests accordingly.

An analysis of more than 300 "crowdsourced" five-second tests showed that most tests are designed to encourage responses like "I don't recall" or "I cannot answer this".

From this analysis, rules and guidelines were developed and tested to increase the likelihood of obtaining useful data.

Published in: Technology, Education
  • Be the first to comment

The Five-Second Rules: Ensuring a Healthy Five-Second Test

  1. 1. The Five-Second Rules Ensuring a Healthy Five-Second Test Paul Doncaster pwdoncaster@hotmail.com @pwdoncaster
  2. 2. Agenda  The Method  The Analysis  The Rules  Special Topics ◦ Emotional Response ◦ Trust/Credibility ◦ Outside-the-Box uses
  3. 3. The Five-Second Test  Perfetti, C (2005). “5-Second Tests: Measuring Your Site's Content Pages.” http://www.uie.com/articles/five_second_test/Brief history 1. Present the focused task 2. Present the method instructions 3. Show the entire page 4. Participant recollection 5. Success verification
  4. 4. Now . . . a discount/rapid addition to the UX toolkit
  5. 5. Analysis  April - September 2012  300+ tests on fivesecondtest.com
  6. 6. Analysis  Instructions  Presentation of the image  Questions  Test Focus  Appropriateness Level III Offender 4+ violations Level II Offender 3 violations Level 1 Offender 2 violations OK 0-1 violations
  7. 7. EXAMPLES
  8. 8. We would like you to rate Quality, Professionalism and Credibility of page 1 lowest 10 highest.
  9. 9. We would like you to rate Quality, Professionalism and Credibility of page 1 lowest 10 highest.  Do you feel we are a credible training organisation?  Do you want "further" information i.e. would you Opt In?  Please tell us one Like and one Dislike of the page
  10. 10. Imagine you are engaged and browsing websites about weddings.
  11. 11. Imagine you are engaged and browsing websites about weddings.  What in particular is this site about?  What is your strongest memory of the site?  Is the site trustworthy?  What is the free offer for?  Can you name the brand?
  12. 12. Imagine that you are looking for promotional products.
  13. 13. Imagine that you are looking for promotional products.  What does the company do?  What do you think the purpose of this website is?  Does this website grab your attention?  How would you improve this site?  Does this design compel you to call [the Company Name]?
  14. 14. You are evaluating companies to hire for professional services.
  15. 15. You are evaluating companies to hire for professional services.  What does this company do?  What word would you use to describe the design?  Would you hire this company?
  16. 16. N = 319 (Complete sample) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% * 25% were clearly using the wrong method OK 24% Level I Offender 28% Level II Offender 30% Level III Offender 18%
  17. 17. N = 239 (Modified sample) 0% 20% 40% 60% 80% Poor Instructions Image Size Question Order Unfocused Elab. Answers 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% OK 30% Level I Offender 30% Level II Offender 27% Level III Offender 13%
  18. 18. N = 319 (Complete sample) 0% 10% 20% 30% 40% 5 Questions 4 Questions 3 Questions 2 Questions 1 Question • Home / Welcome page – 49% • Trust/credibility – 24% • Default instructions – 14% • “Imagine…” – 24% • Predict future behavior – 16% • Compare 2+ – 4%
  19. 19. THE FIVE SECOND RULES
  20. 20. The Golden Five-Second Rule Don't give your participants any excuse to pass or say “I don't know/remember”
  21. 21. Five-Second Rule #2  Don’t use a five-second test when another type of test is more appropriate ◦ Reading the content ◦ Context is required ◦ Predict future behavior ◦ For Homepages, limit to Emotional Response questions ◦ For Comparisons, limit to simple, singular elements (e.g., logos, buttons, etc.)
  22. 22. Five-Second Rule #3  Best results come from knowing what design aspect you want to test, and focusing the test on that single aspect ◦ Is the page’s purpose evident? ◦ Are target(s) easily discerned/memorable? ◦ Does the design elicit the desired emotional response? ◦ Does the design convey trust/credibility? ◦ Split elements into separate tests if needed
  23. 23. The Five-Second Rules “Where are you going to click?” “Imagine you're researching software vendors for your bank.”“You have 5 seconds to view the image. Afterwards, you'll be asked a few short questions.” “You are going to view a website.” “Imagine you found the site using a search engine.” “Imagine you are standing on the street and a bus drives past. You see the advertisement on the back.” “Imagine that you are looking at the following page…” “What does this page do?”
  24. 24. Five-Second Rule #4  Give proper instructions ◦ Adequate table-setting ◦ Tailor instructions to the questions ◦ When instructions are general, don’t expect retention of specifics  Use the online defaults with caution
  25. 25. The Five-Second Rules
  26. 26. The Five-Second Rules
  27. 27. The Five-Second Rules Correct Correct Pass / Don't know, 24% Pass / Don't know, 10% Incorrect Incorrect 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Big Scroll (n=21) No Scroll (n=21)
  28. 28. Five-Second Rule #5  Make sure images/pages fit the screen (no scrolling!!!)
  29. 29. The Five-Second Rules How Many?
  30. 30. The Five-Second Rules No answer given "Nothing" or "I don't know" General site attribute Specifi c or implied target 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Item1 Item2 Item3 Item4 Item5 No answer given "Nothing" or "I don't know" General site attribute Specifi c or implied target 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Item1 Item2 Item3 Item4 Item5 New Orleans (n=21) Chicago (n=21)
  31. 31. Five-Second Rule #6  Limit questions of specific recall to 2-3 ◦ Expect -20% memory impact for each question asked ◦ With 3 target questions, ~75% likelihood of a “full” response
  32. 32. The Five-Second Rules Order
  33. 33. The Five-Second Rules 0% 10% 20% 30% 40% 50% 60% 70% 80% T1: Target 1st of 4 T2: Target 4th of 4 T3: Target 5th (w/distractor) Correct Incorrect IDK/CR/Pass
  34. 34. Five-Second Rule #7  Order the questions with care ◦ Most important if the test focuses on or contains targets  Critical target(s) early, or focus the test accordingly  If you must do a mixed test, target identification question(s) come first
  35. 35. The Five-Second Rules “What articles do you think would be valuable additions to this knowledge base. Tell me anything you can think of.” “Would you find this product useful if it successfully aggregated top news and content? (and obviously was much better designed)” “Is the product/service offering clear?” “Describe the website in a couple of words.” “Does the site look professional?” Wording
  36. 36. Five-Second Rule #8  Craft the questions with care ◦ Remember, the recall clock is ticking . . . ◦ Specific is not always better than abstract ◦ Use devices (scales, etc.) judiciously to get more robust data
  37. 37. The Five-Second Rules
  38. 38. Five-Second Rule #9  Consider very carefully the “Most Prominent Element” question ◦ If you use it, put it toward the end
  39. 39. The Five-Second Rules “If you could change one thing about the page, what would it be?” “What changes would you make to the layout or design of this page?” “What would you change to improve the site?” “What would you change about the design (if anything)?”“Any suggestions?”
  40. 40. Five-Second Rule #10  Don’t ask “What would you change?” ◦ Improper question for the method
  41. 41. (SOMEWHAT) BETTER EXAMPLES
  42. 42. SPECIAL TOPICS
  43. 43. Emotional Response http://www.slideshare.net/pwdoncaster/preference-and-desirability-testing- measuring-emotional-response-to-guide-design
  44. 44. Emotional Response
  45. 45. Emotional Response
  46. 46. Emotional Response Positive Negative Professional Amateurish Clear Confusing Stimulating Dull Reassuring Intimidating
  47. 47. Emotional Response
  48. 48. Emotional Response
  49. 49. Emotional Response Positive Negative Professional Amateurish Easy-Going Assertive Modest Pretentious Reassuring Intimidating
  50. 50. Trust / Credibility
  51. 51. Trust / Credibility  Fogg, B.J. et. al. (2003). How do people evaluate the credibility of Web sites? Proceedings of DUX2003, Designing for User Experience Conference. http://www.consumerwebwatch.org/pdfs/stanfordPTL.pdf  Participants were given as much time as they wanted on sites to gauge credibility  In the end, 46.1% noted "Design Look" in their comments “Our result is consonant with findings of other research (Cockburn and McKenzie, 2001) that describes typical Web-navigation behavior as “rapidly interactive,” meaning that Web users typically spend small amounts of time at any given page, moving from page to page quickly. If such rapid navigation is indeed the norm for most types of Web use, then it makes sense that Web users have developed efficient strategies, such as focusing on the design look, for evaluating whether a Web site is worthwhile . . . In other words, the visual design may be the first test of a site’s credibility. If it fails on this criterion, web users are likely to abandon the site and seek other sources of information and services."
  52. 52. Trust / Credibility  Standardized Universal Percentile Rank Questionnaire ◦ “I feel comfortable purchasing from this website.” ◦ “This website keeps the promises it makes to me.” ◦ “I can count on the information I get on this website.” ◦ “I feel confident conducting business with this website.” ◦ “The information on this website is valuable.” http://www.suprq.com/
  53. 53. Trust / Credibility  The design reflects a company that is dependable and responsive.  The design reflects a company that is honest and ethical.  The design reflects a company that connects with the wants and needs of its customers. How does the look and feel of this page design represent the company?
  54. 54. Trust / Credibility Agree Agree Agree No Opinion No Opinion No Opinion Disagree Disagree Disagree 0% 20% 40% 60% 80% 100% Dependible/Responsive Honest/Ethical Wants/Needs of Customers Agree Agree Agree No Opinion No Opinion No Opinion Disagree Disagree Disagree 0% 20% 40% 60% 80% 100% Dependible/Responsive Honest/Ethical Wants/Needs of Customers
  55. 55. Outside-the-Box
  56. 56. Emotional Response: Triading Hawley, M. (2007). The Repertory Grid: Eliciting User Experience Comparisons in the Customer’s Voice. Available online at http://www.uxmatters.com/mt/archives/2007/12/the-repertory-grid-eliciting-user- experience-comparisons-in-the-customers-voice.php Hawley, M. (2010). Rapid Desirability Testing: A Case Study. Available online at http://www.uxmatters.com/mt/archives/2010/02/rapid-desirability-testing-a-case- study.php
  57. 57. Triading
  58. 58. Triading Consistent with triading Consistent with triading Pass / Don't know, 10% Pass / Don't know, 10% Not consistent Not consistent 0% 20% 40% 60% 80% 100% • “The gradient used in C makes it look modern.” • “A is easily recognizable (a scorpion -- rather than a wave).” • “Logo A felt very slick, and embodied the values visually . . .” • “A was not good - looked outdated.” • “B - the red is over-used.” • “Logo B was a little less stylized, and the very literal scorpion imagery didn't support the brand as well.” Q1: Particularly Good Q2: Particularly Bad
  59. 59. You're wandering on the floor of a trade show and happen to notice this booth/display.
  60. 60. You're wandering on the floor of a trade show and happen to notice this booth/display.  What type of product is being sold?  What type of person (or group of people) is this product targeted to?  Given your responses to #1 and #2, is the display's design appropriate for the intended customer?  Given your responses to #1 and #2, is the display's design visually appealing?  What is the brand name of the
  61. 61. The new fivesecondtest.com https://www.surveymonkey.com/s/5SecTest
  62. 62. Thanks for staying awake. Questions? Blog: “UX Five Second Rules” Coming Soon Website: Coming Soon! Email: pwdoncaster@hotmail.com Twitter: @pwdoncaster

×