Fast, Cheap, and Actionable: Creating an Affordable User Research Program


Published on

Done a usability study? Ready for the next step? Today we have an abundance of fast, affordable website user research methods, many of which can be done remotely with real users. Learn about available user research options and how IUP runs successful research projects that lead to actionable insights.

Published in: Design
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Fast, Cheap, and Actionable: Creating an Affordable User Research Program

  1. 1. Fast, Cheap, and Actionable Creating an Affordable User Research Program
  2. 2. Michael PowersDirector of Web Services
  3. 3. On the Web••••
  4. 4. What kinds of user researchhave you done?
  5. 5. It all started with a usability test
  6. 6. Today• Getting Permission for Research• A Process for User Research• Inventory of Actionable User Research You Can Do Fast and Cheap
  7. 7. User Research Study ↓User Research Program
  8. 8. 0 Getting Permission
  9. 9. Don’t Ask
  10. 10. “We don’t want“a research“study.”
  11. 11. “A research study?“Look, just make a red box on the home page, and put this text in there.“I just need to make sure students don’t miss it.”
  12. 12. Nobody wants aresearch study.
  13. 13. Nobody wants a website.
  14. 14. What We Really Want• Applicants• Successful Students• Academic Research• Donations
  15. 15. Create a ResearchCulture• Do a usability test and invite stakeholders to watch• Make research part of every project• Apply previous research to new projects• Show results early and often
  16. 16. Research Process1. Frame a Good 5. Run Study and Question Analyze Results2. Know What You 6. Take Action Already Know 7. Measure and Repeat3. Choose Study Type4. Find Participants
  17. 17. 1 Frame a Good Question
  18. 18. Good questions are related to your goals
  19. 19. “Students can’t find stuff on our website.”
  20. 20. Focusing a Question How can we • Can students find decrease information about attrition? support services? • Can students find out what’s happening on campus?
  21. 21. “The admissions website needs work.”
  22. 22. Focusing a Question How can we get 1. Can prospectives find our list of majors? more/more Our tuition qualified information? applicants? 2. Can prospectives register for a visit? 3. Can prospectives successfully apply on line?
  23. 23. Goal-related questions lead to faster, cheaper, more actionable user research
  24. 24. 2 Know What You Already Know
  25. 25. What Do You Already Know?• Best Practices • Server Logs• Published Studies • Analytics• Data about Your • Past Studies Institution and Students
  26. 26. Get OrganizedQuantitative Data Qualitative DataKPI Spreadsheet User Personas
  27. 27. KeyKPI = Performance Indicator
  28. 28. KPIs measure progress toward your goals.
  29. 29. KPI Spreadsheet• Focus on the KPIs youve • Include anything chosen numeric: survey results, 4Q survey, analytics,• Give specific reports to server logs, etc. stakeholders who request them • Create specific analyses for specific projects• Track over time without running multiple tools
  30. 30. KPI SpreadsheetAvailable on
  31. 31. User Personas
  32. 32. Include Scenarios Site Features Motivation Scenario Used Behaviors Find the Checks to see •Site Search •Looks for school if her major “Pre- with the is available •Undergrad Law” right pre- Admissions major, Law Website does not program find it. for me •Academics Link
  33. 33. User Personas• Fiction—but a useful • Helps bring new team fiction members up to speed• Puts what youve learned into a format you can share with your team and clients• If it is wrong, team or clients will let you know
  34. 34. Use these two tools to track your progress over time.
  35. 35. 3 Choose Study Type
  36. 36. If you’ve never done one, do a usability test first.
  37. 37. FocusGroups
  38. 38. Some Types of User Research• Usability Test • Navigation Test• Surveys • Layout Test• Pop-up Survey • A/B Test• KJ Session• Card Sort
  39. 39. Some Types of User Research• Usability Test • Card Sort• Surveys • Navigation Test• Pop-up Survey • Layout Test• KJ Session • A/B Test
  40. 40. Usability Test Photo source:
  41. 41. Analytics
  42. 42. Survey
  43. 43. Pop-Up Survey
  44. 44. Pop-up Survey What You Do: Ask site visitors questions about their experience on a particular page or the whole website. Good For: Capturing user intent; settling arguments about what belongs where When To Use: When you have a question to answer, or all the timeParticipants Needed: Calculate your sample size: sscalc.htm
  45. 45. Pop-up Survey Tools Available: •4Q Survey •iPerceptions •CrowdScience •Roll your own Cost: •$0–$10,000+
  46. 46. 4Q Survey Questions1. Based on todays visit, how would you rate your site experience overall?2. Which of the following best describes the primary purpose of your visit?3. Were you able to complete the purpose of your visit today?4. Comment • If yes: What do you value most about the website? • If no: Please tell us why you were not able to fully complete the purpose of your visit today
  47. 47. “I am always able to “I can never easilyfind what I am find what Imlooking for. Thewebsite is really looking for, if I caneasy to use.” find it at all.”
  48. 48. Coding Comments• Visual Design • Navigation Positive Positive • Navigation Negative• Visual Design Negative • Content Quality Positive• Usability Positive • Content Quality• Usability Negative Negative
  49. 49. 637,809:-%&-*(##%+,;+< NF,)/+O)-./" (:L8")/2"M*:/58" J+8I)4"K:8+,/"" @8)34:" 1+/2"A:.=4:"2(405- 9./5:/5"HI)4+56" (:,)-*:" <+5:"<:)F>G" A.8+-*:" ;@A"B:38+5:C"/./D9E<" <=:>+?>"=),:08+5:" 7)8"9./5:/5";"(::2" ()*+,)-./01+/2)3+4+56" !" #!" $!" %!" &!" !!" #!" $!" %!" &!" !"#$%&()*(##%+,-.%+/(+0+1230-2(405
  50. 50. 6757"3#45&"89":&*;&"/#00*12" <#&2"%#&()*=>+,*-.()*=>"?1@" SK1.40T.234# -?Q=#.47#R/?4:=# O0=N.9#P?=014## E=.89?# 6047#F?3B9?#3#45&" >34:?4:#MN.90:;# A0:?#A?.KCL# -?1.2/?# @EF#G?8=0:?H#434I>JA# F3=02/?# AB?C0DC#B.1?5=0:?# <.=#>34:?4:#@#-??7# -./01.23456047.8090:;# !"# $!"# %!"# &!"# !"# (!"# )!"# *!"# +!"# ,!"# $!!"# !"#$"%#&()*+,*-.()*"/#00*12&"
  51. 51. Action• Changed our yearly goals to spend more time on these issues
  52. 52. Card Sort Photo Source:
  53. 53. Card Sort What You Do: Give participants cards with website topics on them, then ask them to put them into piles that make sense to them Good For: Understanding the mental models of your users When To Use: When creating or improving your information architecture/navigation Participants Needed: 15 face to face 50 online
  54. 54. Card Sort Tools Available: •Index cards •OptimalSort (remote) •WebSort (remote) •xSort Cost: $40-$600
  55. 55. Open Closed• Users stack cards • Give users header then cards• Give each pile a name • They assign cards to the header cards
  56. 56. Where do we put counseling?
  57. 57. Action• Complete restructuring of the Student Life section of our website• Move counseling websites out of “Health” and into “Support”
  58. 58. One Subsite,Before and After Change between Compared to Comparison Periods Whole SitePageviews 76% 140%Unique 81% 142%PageviewsBounce Rate -21% 27%
  59. 59. KJ Session Photo Source:
  60. 60. Navigation Test
  61. 61. Navigation Test What You Do: Give participants a proposed site structure/navigation and let users drill down to find different things. Good For: Verifying an IA in a more realistic way (not everything is visible); finding problem areas When To Use: To verify a new information architecture Participants Needed: 50
  62. 62. Navigation Test Tools Available: •Treejack Cost: $150–$700
  63. 63. Action• Rearranged Admissions Site navigation to • Combine repeated material • Put “Costs and Tuition” higher in the site tree
  64. 64. Layout Test
  65. 65. Layout Test What You Do: Participants respond to a static image of your site Good For: Behavior, Remote, Automated, Quant. When To Use: Before building out a new design Participants Needed: About 50 per test
  66. 66. Layout Test Tools Available: •Paper •PowerPoint •Five-Second Usability Test •Chalkmark •Usabila Cost: $40–$600
  67. 67. Layout TestPositive Negative•Get reactions to real •No interactivity designs without •Users can be building out a confused when whole site doing this remotely
  68. 68. A/B Test
  69. 69. 4 Find Participants
  70. 70. The Right Number ofParticipants• Different methods have different requirements• Plan to recruit more than you need• But don’t test more than you need
  71. 71. The Right Kind of Participants• Steve Krug: “Recruit loosely and grade on a curve”• Good advice, but when possible try to recruit participants that are more like your users• For instance: faculty and students see your website very differently
  72. 72. Finding Participants• People you know • On your website (link on homepage or pop-• Students, faculty, and up) staff at your school• Local high schools • Recruitment tools like Ethnio (talk to a guidance counselor)
  73. 73. Photo source:
  74. 74. Offer a prize
  75. 75. 5 Run Study and Analyze Results
  76. 76. Keep track of results in KPIspreadsheet and personas
  77. 77. 6 Take Action
  78. 78. Share results and proposedsolutions with stakeholders
  79. 79. 7 Measure and Repeat
  80. 80. Share results
  81. 81. Challenge:Do some sort of userresearch each month
  82. 82. Research Process1. Frame a Good 5. Run Study and Question Analyze Results2. Know What You 6. Take Action Already Know 7. Measure and Repeat3. Choose Study Type4. Find Participants
  83. 83. More Information