The document outlines a framework for creating an affordable user research program at educational institutions, emphasizing the importance of actionable research to improve website usability. It details a step-by-step research process, including framing questions, selecting methods, and measuring progress with key performance indicators (KPIs). Various user research techniques are discussed, such as usability tests, surveys, and card sorting, along with their pros and cons to help institutions gather valuable user insights without extensive resources.

1. Fast, Cheap, and
Actionable:
Creating an Affordable User
Research Program
Michael Powers
Director of Web Services
Indiana University of Pennsylvania
eduWeb 2010 — July 27, 2010, Chicago, IL

3. In Answer to Your
First Question... Buffalo
Erie
Chicago
Cleveland Philadelphia
Pittsburgh
Morgantown
Indianapolis

4. In Answer to Your
First Question... Buffalo
Erie
Indiana, Pa.
Chicago
Cleveland Philadelphia
Pittsburgh
Morgantown
Indianapolis

5. In Answer to Your
First Question... Buffalo
Erie
Indiana, Pa.
Chicago
Cleveland Philadelphia
Pittsburgh
Morgantown
Indianapolis

6. On the Web
•museyroom.com
•twitter.com/mjpowers
•www.slideshare.net/michaelpowers

7. What kinds of
user research
have you done?

8. It all started with
a usability test

14. Today
•Getting Permission for Research
•A Process for User Research
•Inventory of Actionable User
Research You Can Do Fast and
Cheap

15. User Research Study
↓
User Research Program

16. 0

17. 0
Creating
Permission

18. Don’t Ask

19. “We don’t
want a
research
study.”

20. “A research study?
“Look, just make a red box
on the home page, and put
this text in there.
“I just need to make sure
students don’t miss it.”

21. Nobody wants a
research study.

22. Nobody wants a
website.

23. What We Really
•Applicants
•Successful Students
•Academic Research
•Donations

24. Create a Research
•Do a usability test and invite
stakeholders to watch
•Make research part of every project
•Apply previous research to new
projects
•Show results early and often

25. Research Process
1. Frame a Good 4. Find
Question Participants
2. Know What You 5. Run Study and
Already Know Analyze Results
3. Choose Study 6. Take Action
Type
7. Measure and
Repeat

26. 1 2 3 4 5 6 7
Frame a Good
Question

27. Good questions are
related to your goals

28. “Students can't
find stuff on our
website.”

29. Focusing a
How can we 1. Can students find
information about
decrease support services?
attrition?
2. Can students find out
what’s happening on
campus?

30. “The admissions
website needs
work."

31. Focusing a
How can we get 1. Can prospectives find
our list of majors?
more/more Our tuition
qualified information?
applicants? 2. Can prospectives
register for a visit?
3. Can prospectives
successfully apply on
line?

32. Goal-related questions
lead to faster, cheaper,
more actionable user
research

33. 1 2 3 4 5 6 7
Know What You
Already Know

34. What Do You
•Best Practices •Server Logs
•Published •Analytics
Studies •Past Studies
•Data about Your
Institution and
Students

35. Get Organized
Quantitative
Data
Qualitative
KPI Data
Spreadsheet
User Personas

36. Key
KPI Performance
Indicator

37. KPIs measure progress
toward your goals.

40. KPI Spreadsheet
•Focus on the KPIs •Track over time
you've chosen without running
multiple tools
•Give specific
reports to •Include anything
stakeholders who numeric: survey
request them results, 4Q survey,
analytics, server
logs, etc.

41. KPI Spreadsheet
Available on
museyroom.com

42. User Personas

45. Include Scenarios
Site
Motivation Scenario Features Behaviors
Used Search Looks for
Find the Checks to •Site •
school with see if her •Undergrad “Pre-Law”
the right major is Admissions major,
pre-Law available Website does not
program for •Academics find it.
me Link

46. User Personas
•Fiction—but a •If it is wrong, team
useful fiction or clients will let
you know
•Puts what you've
learned into a •Helps bring new
format you can team members up
share with your to speed
team and clients

47. Use these two tools to
track your progress
over time.

48. 1 2 3 4 5 6 7
Choose
Study Type

49. If you’ve never done
one, do a usability test
first.

50. Focus
Groups

51. Focus
Groups

52. Some Types of User
•Usability Test •Card Sort
•Surveys •Navigation
Test
•Pop-up
Survey •Layout Test
•KJ Session •A/B Test

54. Some Types of User
•Usability Test •Card Sort
•Surveys •Navigation
Test
•Pop-up
Survey •Layout Test
•KJ Session •A/B Test

55. Usability Test
Photo source: http://www.flickr.com/photos/l-i-n-k/
3654390818/

56. Usability Test
What You Do: Ask people to do things on your
website while they talk about
what they do and why they do it
Good For: Finding things to fix
When To Use: As early as possible and
regularly thereafter
Participants As few as 3 (3 groups of 5-7 is
Needed: better)

57. Usability Test
Tools •No tools are needed
Available: •Silverback (Live)
•Morae (Live
(Live)
•Any screen-sharing tool
(Skype, GoTo Meeting, WebEx)
(Remote)
Cost: •UserVue (Remote, no longer
$90–$1,000+
available)
Record In: •User Profiles

58. Usability Test
Positive Negative
•Can be cheap •Behavior of
•Holistic view of participants will
website issues not be entirely
natural
•Takes practice to
run good tests

59. Analytics

60. Analytics
What You Do: Analyze the behavior of users
on your website using your
analytics tool
Good For: Seeing real behavior on your
website
Not So Good Understanding user intent
For:
When To Use: Collect 24/7, analyze monthly
or as needed
Participants n/a
Needed:

61. Analytics
Tools •Google Analytics
Available: •Mint
•ClickTracks
•Omniture
•Many others
Cost: •$0–$10,000+

62. Analytics
Positive Negative
•Can be free •Can’t understand
•You see actual intent
use of your site •Can lead to just
•Can track looking at
changes over time pageviews

63. Survey

64. Survey
What You Do: Ask people questions about
their browsing habits or your
website.
Good For: Background information about
your usershave a question to
When To Use: When you
answer, possibly yearly
Participants Calculate your sample size:
Needed: http://www.surveysystem.com/
sscalc.htm

65. Survey
Tools •Survey Monkey
Available: •Google Docs
•Existing campus survey
product
•Many others
Cost: •$40–$10,000+

66. Survey
Positive Negative
•Tools are cheap •Shows you
or free attitudes, not
•Offer a prize and behavior, so not
you can get a great for
respectable understanding
number of how users use
summissions your website
•Lots of numbers

67. Pop-Up Survey

68. Pop-up Survey
What You Do: Ask site visitors questions
about their experience on a
particular page or the whole
Good For: website. user intent; settling
Capturing
arguments about what belongs
When To Use: whereyou have a question to
When
answer, or all the time
Participants Calculate your sample size:
Needed: http://www.surveysystem.com/
sscalc.htm

71. Pop-up Survey
Tools •4Q Survey
Available: •iPerceptions
•CrowdScience
•Roll your own
Cost: •$0–$10,000+

73. 4Q Survey
1. Based on today's visit, how would you rate your site
experience overall?
2. Which of the following best describes the primary
purpose of your visit?
3. Were you able to complete the purpose of your visit
today?
4. Comment
• If yes: What do you value most about the website?
• If no: Please tell us why you were not able to fully
complete the purpose of your visit today

74. “I am “I can never
always able easily find
to find what what I'm
I am looking looking for,
for. The if I can find
website is it at all.”

75. Coding Comments
• Visual Design • Navigation Positive
Positive
• Navigation Negative
• Visual Design
Negative
• Content Quality
Positive
• Usability Positive
• Content Quality
• Usability Negative Negative

76. 637,'809':-%&-'*(##%+,';+<'
NF,)/+O)-./"
(:L8")/2"M*:/58"
J+8I)4"K:8+,/""
@8)34:"
1+/2"A:.=4:"
2(405-'
9./5:/5"HI)4+56"
(:,)-*:"
<+5:"<:)F>G"
A.8+-*:"
;@A"B:38+5:C"/./D9E<"
<=:>+?>"=),:08+5:"
7)8"9./5:/5";"(::2"
()*+,)-./01+/2)3+4+56"
!" #!" $!" %!" &!" '!!" '#!" '$!" '%!" '&!"
!"#$%&'()'*(##%+,-'.%+/(+0+1'230-'2(405'

77. 67'57"3#4'5&"8'9":&*;&"/#00*12"
<#&2"%#&'()*=>+,*-.()*=>"?1@"
SK1.40T.234#
-?Q=#.47#R/?4:=#
O0=N.9#P?=014##
E=.89?#
6047#F?3B9?#
3#4'5&"
>34:?4:#MN.90:;#
A0:?#A?.KCL# -?1.2/?#
@EF#G?8=0:?H#434I>JA# F3=02/?#
AB?C0DC#B.1?5=0:?#
<.=#>34:?4:#@#-??7#
-./01.23456047.8090:;#
!"# $!"# %!"# &!"# '!"# (!"# )!"# *!"# +!"# ,!"# $!!"#
!"#$"%#&'()*+,*-.()*"/#00*12&"

78. Action
•Changed our yearly goals to spend
more time on these issues

79. Pop-up Survey
Positive Negative
•You get info from •Some users
real users respond
•Easy to set up negatively to
being interrupted
•Gives you a
window into user •Coding can take
intent time

80. Card Sort
Photo Source: http://www.flickr.com/photos/
rosenfeldmedia/3343501403/

81. Card Sort
What You Do: Give participants cards with
website topics on them, then
ask them to put them into piles
Good For: that make sense to mental
Understanding the them
models of your users
When To Use: When creating or improving
your information architecture/
Participants navigation face
15 face to
Needed: 50 online

82. Card Sort
Tools •Index cards
Available: •OptimalSort (remote)
•WebSort (remote)
•xSort
Cost: $40-$600

83. Card Sort
Positive Negative
•Get a picture of users’ •Hard for participants to
mental model of your understand activity
content remotely
•Understand what labels •Some participants give
your users use for your up because the sort can
content take a while
•Settle arguments about •Data analysis can be
what belongs where complex

85. Open Closed
•Users stack •Give users
cards then header cards
•Give each pile a •They assign
name cards to the
header cards

88. Where do we put
counseling?

91. Action
•Complete restructuring of the
Student Life section of our website
•Move counseling websites out of
“Health” and into “Support”

92. One Subsite,
Before and After
Change between
Compared to
Comparison
Whole Site
Periods
Pageviews 76% 140%
Unique
81% 142%
Pageviews
Bounce Rate -21% 27%

93. KJ Session
Photo Source: http://www.flickr.com/photos/drewm/
370231088/

94. KJ Session
What You Do: Gather 4-6 participants
Ask: what do people come to
our website to find or do?
Put answers on sticky notes on
the wall
Organize and name the groups
Good For: Getting the opinion of site
owners; learning about users
from site owners
When To Use: When creating a new
information architecture
Participants 4–6, or more if you divide into
Needed: smaller groups

95. KJ Session
Tools •Walls and sticky notes
Available:
Cost: $0–$40
Learn More: http://insidenewcity.com/blog/
view/unlock-the-wisdom-of-
your-group-host-a-kj-session-
for-site-architecture/

96. KJ Session
Positive Negative
•Get feedback •Not real users
from people who
know your users
•Get buy-in from
stakeholders

97. Navigation Test

98. Navigation Test
What You Do: Give participants a proposed
site structure/navigation and let
users drill down to find
Good For: different things. a more
Verifying an IA in
realistic way (not everything is
When To Use: visible); finding problem areas
To verify a new information
architecture
Participants 50
Needed:

99. Navigation Test
Tools •Treejack
Available:
Cost: $150–$700

103. Navigation Test
Positive Negative
•Tests your •Doesn’t look like
sitemap/ your site, so on-
navigation site results might
without having to be different
build it
•Can locate
problems with
your IA

104. Action
•Rearranged Admissions Site
navigation to
•Combine repeated material
•Put “Costs and Tuition” higher in
the site tree

105. Layout Test

106. Layout Test
What You Do: Participants respond to a static
image of your site
Good For: Getting reactions to visual
designs and wireframes without
building the whole thing.
When To Use: Before building out a new
design
Participants About 50 per test
Needed:

107. Layout Test
Tools •Paper
Available: •PowerPoint
•Five-Second Usability Test
•Chalkmark
•Usabila
Cost: $40–$600

108. Layout Test
Positive Negative
•Get reactions to •No interactivity
real designs •Users can be
without building confused when
out a whole site doing this
remotely

112. A/B Test

113. A/B Test
What You Do: Test two (or more) versions of
your live website
Good For: Testing page variations
When To Use: Testing calls to action or other
pages that have a clear
destination you
Participants Tool will tell
Needed:

114. A/B Test
Tools •Google Website Optimizer
Available: •Reedge.com
Cost: $0–???

115. A/B Test
Positive Negative
•Test real •Harder to set up
variations on your in a CMS
live site with real •Harder to use
users when a page has
many destinations

116. Easy to do with
Google Website
Optimizer
A
Go
al
B

117. Harder to do with
Google Website Go
Optimizer
A
Go
Go
B Go
Go

118. 1 2 3 4 5 6 7
Find Participants

119. The Right Number
•Different methods have different
requirements
•Plan to recruit more than you need
•But don’t test more than you need

120. The Right Kind of
•Steve Krug: “Recruit loosely and
grade on a curve”
•Good advice, but when possible try
to recruit participants that are more
like your users
•For instance: faculty and students
see your website very differently

121. Finding
•People you •Local high
know schools (talk to
a guidance
•Students, counselor)
faculty, and staff
at your school •On your website
(link on
homepage or
pop-up)

122. Photo source: http://en.wikipedia.org/wiki/
File:Ethnio_recruit_ad.png

123. Offer a prize

124. 1 2 3 4 5 6 7
Run Study
and Analyze
Results

125. Keep track of results
in KPI spreadsheet and
personas

126. 1 2 3 4 5 6 7
Take Action

127. Share results and
proposed solutions
with stakeholders

128. 1 2 3 4 5 6 7
Measure and
Repeat

129. Share results

130. Challenge:
Do some sort of user
research each month

131. Research Process
1. Frame a Good 4. Find
Question Participants
2. Know What You 5. Run Study and
Already Know Analyze Results
3. Choose Study 6. Take Action
Type
7. Measure and
Repeat

132. More Information
•museyroom.com
•twitter.com/mjpowers
•www.slideshare.net/michaelpowers