Fast, Cheap, and
     Actionable:
Creating an Affordable User
    Research Program
            Michael Powers
        Dire...
In Answer to Your
First Question...                              Buffalo




                                     Erie


C...
In Answer to Your
First Question...                              Buffalo




                                     Erie

  ...
In Answer to Your
First Question...                              Buffalo




                                     Erie

  ...
On the Web
•museyroom.com
•twitter.com/mjpowers
•www.slideshare.net/michaelpowers
What kinds of
 user research
have you done?
It all started with
  a usability test
Today
•Getting Permission for Research
•A Process for User Research
•Inventory of Actionable User
 Research You Can Do Fas...
User Research Study
          ↓
User Research Program
0
0

     Creating
    Permission
Don’t Ask
“We don’t
want a
research
study.”
“A research study?
“Look, just make a red box
on the home page, and put
this text in there.
“I just need to make sure
stud...
Nobody wants a
research study.
Nobody wants a
   website.
What We Really
•Applicants
•Successful Students
•Academic Research
•Donations
Create a Research
•Do a usability test and invite
 stakeholders to watch
•Make research part of every project
•Apply previ...
Research Process
1. Frame a Good    4. Find
   Question           Participants
2. Know What You   5. Run Study and
   Alre...
1 2 3 4 5 6 7

Frame a Good
  Question
Good questions are
related to your goals
“Students can't
find stuff on our
website.”
Focusing a
How can we   1. Can students find
                information about
decrease        support services?
attrition...
“The admissions
website needs
work."
Focusing a
How can we get   1. Can prospectives find
                    our list of majors?
more/more           Our tuiti...
Goal-related questions
lead to faster, cheaper,
 more actionable user
        research
1 2 3 4 5 6 7

Know What You
 Already Know
What Do You
•Best Practices    •Server Logs
•Published         •Analytics
 Studies           •Past Studies
•Data about You...
Get Organized
Quantitative
Data
               Qualitative
KPI            Data
Spreadsheet
               User Personas
Key
KPI   Performance
      Indicator
KPIs measure progress
  toward your goals.
KPI Spreadsheet
•Focus on the KPIs   •Track over time
 you've chosen        without running
                      multiple...
KPI Spreadsheet
Available on
museyroom.com
User Personas
Include Scenarios
                           Site
Motivation    Scenario     Features   Behaviors
                        ...
User Personas
•Fiction—but a      •If it is wrong, team
 useful fiction       or clients will let
                      yo...
Use these two tools to
 track your progress
      over time.
1 2 3 4 5 6 7

   Choose
 Study Type
If you’ve never done
one, do a usability test
         first.
Focus
Groups
Focus
Groups
Some Types of User
•Usability Test •Card Sort
•Surveys        •Navigation
                 Test
•Pop-up
 Survey         •L...
Some Types of User
•Usability Test •Card Sort
•Surveys        •Navigation
                 Test
•Pop-up
 Survey         •L...
Usability Test




      Photo source: http://www.flickr.com/photos/l-i-n-k/
                          3654390818/
Usability Test
What You Do: Ask people to do things on your
             website while they talk about
             what t...
Usability Test
      Tools    •No tools are needed
  Available:   •Silverback (Live)
               •Morae (Live
         ...
Usability Test
Positive            Negative
•Can be cheap       •Behavior of
•Holistic view of    participants will
 websi...
Analytics
Analytics
What You Do: Analyze the behavior of users
               on your website using your
               analytics to...
Analytics
      Tools    •Google Analytics
  Available:   •Mint
               •ClickTracks
               •Omniture
     ...
Analytics
Positive           Negative
•Can be free       •Can’t understand
•You see actual     intent
use of your site   •...
Survey
Survey
What You Do: Ask people questions about
             their browsing habits or your
             website.
   Good Fo...
Survey
      Tools    •Survey Monkey
  Available:   •Google Docs
               •Existing campus survey
                pr...
Survey
Positive             Negative
•Tools are cheap •Shows you
 or free             attitudes, not
•Offer a prize and   ...
Pop-Up Survey
Pop-up Survey
What You Do: Ask site visitors questions
             about their experience on a
             particular pa...
Pop-up Survey
      Tools    •4Q Survey
  Available:   •iPerceptions
               •CrowdScience
               •Roll you...
4Q Survey
1. Based on today's visit, how would you rate your site
   experience overall?
2. Which of the following best de...
“I am          “I can never
always able    easily find
to find what   what I'm
I am looking   looking for,
for. The       ...
Coding Comments
• Visual Design        • Navigation Positive
  Positive
                       • Navigation Negative
• Vis...
637,'809':-%&-'*(##%+,';+<'
                   NF,)/+O)-./"
               (:L8")/2"M*:/58"
                  J+8I)4"K:8+,...
67'57"3#4'5&"8'9":&*;&"/#00*12"
                               <#&2"%#&'()*=>+,*-.()*=>"?1@"
                   SK1.40T.23...
Action
•Changed our yearly goals to spend
 more time on these issues
Pop-up Survey
Positive            Negative
•You get info from •Some users
 real users         respond
•Easy to set up     ...
Card Sort




      Photo Source: http://www.flickr.com/photos/
             rosenfeldmedia/3343501403/
Card Sort
What You Do: Give participants cards with
             website topics on them, then
             ask them to put...
Card Sort
      Tools    •Index cards
  Available:   •OptimalSort (remote)
               •WebSort (remote)
              ...
Card Sort
Positive                   Negative
•Get a picture of users’   •Hard for participants to
 mental model of your  ...
Open             Closed
•Users stack     •Give users
 cards then        header cards
•Give each pile a •They assign
 name ...
Where do we put
  counseling?
Action
•Complete restructuring of the
 Student Life section of our website
•Move counseling websites out of
 “Health” and ...
One Subsite,
Before and After
              Change between
                               Compared to
                Comp...
KJ Session




      Photo Source: http://www.flickr.com/photos/drewm/
                          370231088/
KJ Session
What You Do: Gather 4-6 participants
             Ask: what do people come to
             our website to find o...
KJ Session
      Tools    •Walls and sticky notes
  Available:


       Cost: $0–$40



 Learn More: http://insidenewcity....
KJ Session
Positive           Negative
•Get feedback      •Not real users
from people who
know your users
•Get buy-in from...
Navigation Test
Navigation Test
What You Do: Give participants a proposed
             site structure/navigation and let
             user...
Navigation Test
      Tools    •Treejack
  Available:




      Cost: $150–$700
Navigation Test
Positive            Negative
•Tests your         •Doesn’t look like
sitemap/             your site, so on-...
Action
•Rearranged Admissions Site
 navigation to
 •Combine repeated material
 •Put “Costs and Tuition” higher in
   the s...
Layout Test
Layout Test
What You Do: Participants respond to a static
             image of your site

   Good For: Getting reactions ...
Layout Test
      Tools    •Paper
  Available:   •PowerPoint
               •Five-Second Usability Test
               •Ch...
Layout Test
Positive           Negative
•Get reactions to •No interactivity
 real designs     •Users can be
without buildi...
A/B Test
A/B Test
What You Do: Test two (or more) versions of
             your live website

   Good For: Testing page variations
...
A/B Test
      Tools    •Google Website Optimizer
  Available:   •Reedge.com




      Cost: $0–???
A/B Test
Positive           Negative
•Test real         •Harder to set up
 variations on your in a CMS
 live site with rea...
Easy to do with
Google Website
Optimizer
                  A
                      Go
                      al
           ...
Harder to do with
Google Website          Go
Optimizer
                    A
                        Go

                 ...
1 2 3 4 5 6 7


Find Participants
The Right Number
•Different methods have different
 requirements
•Plan to recruit more than you need
•But don’t test more ...
The Right Kind of
•Steve Krug: “Recruit loosely and
 grade on a curve”
•Good advice, but when possible try
 to recruit par...
Finding
•People you           •Local high
 know                  schools (talk to
                       a guidance
•Stude...
Photo source: http://en.wikipedia.org/wiki/
        File:Ethnio_recruit_ad.png
Offer a prize
1 2 3 4 5 6 7
  Run Study
 and Analyze
   Results
Keep track of results
in KPI spreadsheet and
       personas
1 2 3 4 5 6 7


 Take Action
Share results and
proposed solutions
 with stakeholders
1 2 3 4 5 6 7

 Measure and
   Repeat
Share results
Challenge:
Do some sort of user
research each month
Research Process
1. Frame a Good    4. Find
   Question           Participants
2. Know What You   5. Run Study and
   Alre...
More Information
•museyroom.com
•twitter.com/mjpowers
•www.slideshare.net/michaelpowers
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)
Upcoming SlideShare
Loading in …5
×

Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)

2,105 views

Published on

Done a usability study? Ready for the next step? Today we have an abundance of fast, affordable user research methods, many of which can be done remotely with real users. Learn about available user research options and how one university runs successful research projects that lead to actionable insights.

Published in: Design
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,105
On SlideShare
0
From Embeds
0
Number of Embeds
43
Actions
Shares
0
Downloads
64
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
  • Hi, my name&amp;#x2019;s Mike Powers.

  • You&amp;#x2019;ll find us in the town of Indiana, in the state of Pennsylvania. We are a 14,000-student doctoral university in Western Pennsylvania. One hour from Pittsburgh. Because of the confusion with our name, we like to be known as IUP.
  • You&amp;#x2019;ll find us in the town of Indiana, in the state of Pennsylvania. We are a 14,000-student doctoral university in Western Pennsylvania. One hour from Pittsburgh. Because of the confusion with our name, we like to be known as IUP.
  • You&amp;#x2019;ll find us in the town of Indiana, in the state of Pennsylvania. We are a 14,000-student doctoral university in Western Pennsylvania. One hour from Pittsburgh. Because of the confusion with our name, we like to be known as IUP.
  • You&amp;#x2019;ll find us in the town of Indiana, in the state of Pennsylvania. We are a 14,000-student doctoral university in Western Pennsylvania. One hour from Pittsburgh. Because of the confusion with our name, we like to be known as IUP.

  • Usability Test
    Surveys
    4q Survey
    KJ Session
    Card Sort
    Navigation Test
    Layout Test
    A/B Test
  • I got started with this user testing stuff back in 2002 with usability testing--straight out of Steve Krug&amp;#x2019;s Don&amp;#x2019;t Make Me Think.

    It was a revelation. Everyone who runs a website should do it.



  • But the more user-centered I got, the more I wanted to know about about our users. That led to surveys, analytics, card sorts.

    And then this happened...

  • I found myself in charge of a newly centralized university website. 40,000+ pages.
    It was hard to find the time to do the research we needed.

    Then this happened...


  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • An explosion of online tools for learning more about your users. They promise fast, cheap, actionable results.

    But if you&amp;#x2019;re like me, you still don&amp;#x2019;t have a lot of time or money. How do you decide what to do?

    This is the story of how we&amp;#x2019;ve using these tools at IUP.

  • Here&amp;#x2019;s what&amp;#x2019;s on the agenda today.
  • The idea is to move on from the isolated user research study to an ongoing user research program.

    Let&amp;#x2019;s get started!
  • The research process starts with step 1, but before we get there, let&amp;#x2019;s start with step 0: creating permission to do user research.


  • Of course the first rule is: Don&amp;#x2019;t ask for permission.

    You don&amp;#x2019;t need permission. User research is what web professionals need to be doing. If you&amp;#x2019;re a web designer or web developer or content strategist, this is your job.

    But others might not see it that way. You&amp;#x2019;re here because you are interested in user research. Our clients and bosses aren&amp;#x2019;t always as excited about it. So at some point you&amp;#x2019;re going to hear this...

  • This might come from your boss or you&amp;#x2019;re boss&amp;#x2019;s boss&amp;#x2019;s boss. Or it might come from an internal client if you work that way.

    And if you ask them want they want, they might say...
  • This is kind of typical of initial reactions. But&amp;#x2014;we know this is wrong, don&amp;#x2019;t we? By this theory, the perfect website would be one big red box.
  • It can help to reframe this this way: &amp;#x201C;You&amp;#x2019;re right. Nobody wants a research study.&amp;#x201D;

    Furthermore...
  • None of your (internal) clients really want a website. They always have other goals that they have restated as &amp;#x201C;we need this on the website.&amp;#x201D;
  • So back them up to what they really want. These are institutional goals. They are more than the website can provide. But we need to ask these if we are to keep our web tactics in line with institutional goals.

    If you can back them up to what they really want, you&amp;#x2019;ve got a better shot at creating permission to do research.


  • Second tactic: You need to get your clients/bosses/whoever they are used to the idea that research is a necessary part of every project.

    Luckily, since we tend to work on the same website(s) over and over, you can often apply previous research to new websites. So you don&amp;#x2019;t have to do EVERYTHING for every website.

    To get there, you need to show results early and often.
  • 1. Frame a good question: you won&amp;#x2019;t end up with anything actionable unless you&amp;#x2019;re asking the right questions

    2. Know What You Already Know: You already know a lot, you just need to keep track of it

    3. Will include the inventory of research techniques as well as some advice about the design of each type.

    4. How to find the people you need to participate

    And by then we&amp;#x2019;ll be set up to blast through the last three steps
  • This is where fast, cheap, and actionable starts
  • These are your institutional goals. And&amp;#x2014;if you want to be fast, cheap, and actionable&amp;#x2014;you need to spend your limited research time investigating issues that relate to institutional goals.
  • Here&amp;#x2019;s what happened when we took this statement and got it focused.

  • When we looked into this, it turned out to be a question about student success and support.

    Esp. finding info about support services. Question 1 gives us a good place from which to start our research. It also suggests (correctly) that there are non-web aspects to this problem.
  • A second example.
  • The purpose of the admissions site is much more clear. And we already had a pretty good understanding of what prospectives are looking for (thanks in part to eduWeb!).
  • These are your institutional goals. And&amp;#x2014;if you want to be fast, cheap, and actionable&amp;#x2014;you need to spend your limited research time investigating issues that relate to institutional goals.
  • Once you got your questions, you&amp;#x2019;ll need to pose some preliminary answers.
  • You probably know a lot already. The trick is getting it all organized so you&amp;#x2014;and others&amp;#x2014;can make use of it.

    Noel-Levitz study
    Usability studies (like Jakob Nielsen&amp;#x2019;s)
    Existing data about your students

    Once you start looking, you&amp;#x2019;ll find a lot. So it&amp;#x2019;s time to...
  • Get organized.

    You need to start synthesizing what you know into a format you can use and share: with your team, with stakeholders, with colleagues.

    I&amp;#x2019;ll demonstrate an approach for capturing a lot (not of all) of what you already know.

  • A conversion rate is a classic KPI, but you can use anything so long as it shows your progress toward your larger goals.



  • See Brian Clifton&amp;#x2019;s Advanced Web Metrics with Google Analytics for a good discussion of setting your KPIs and the germ of the spreadsheet I&amp;#x2019;m about to show you.
  • Allows you to collect raw data and format it easily into reports.



  • Check my blog site for the whole spreadsheet to download.
  • Qualitative data is harder to collect and synthesize. User personas can be really helpful.

    Does anyone use personas?

  • Some of the things you&amp;#x2019;ll put in here come from basic demographic data, etc., but the point is to make them personal.

    This gives you the chance to embody a &amp;#x201C;real&amp;#x201D; user--and you can talk about what &amp;#x201C;Harold&amp;#x201D; might do on this page.

    Really useful is this section at the bottom with tasks.
  • The scenarios section is really important. If you&amp;#x2019;re a developer, and you&amp;#x2019;ve done test-driven development, it works a bit like that. You record scenarios--successful or not--and then develop the site to make them happen. As a check, go through and try these scenarios yourself. It&amp;#x2019;s like having a series of unit tests. You can make sure that when you fix one thing, you don&amp;#x2019;t break another


  • So: what are you going to do?
  • First thing to keep in mind: if you haven&amp;#x2019;t done a usability test, start there. It&amp;#x2019;s the foundation for everything else.

  • If you&amp;#x2019;ve read Steve Krug, you know focus groups are a bad idea. People will ask if you&amp;#x2019;re going to do a focus group. They think you need to. You don&amp;#x2019;t. You&amp;#x2019;ll only get some ideas about what looks nice, or what people think they should do. But your boss might think that focus groups or surveys are the only options.

    I&amp;#x2019;ll admit: I have let my boss call some of this stuff focus groups. It was just easier. He told his boss and didn&amp;#x2019;t have to explain anything. But once people arrived, we didn&amp;#x2019;t have a focus group: we did user research.
  • These are (mostly) the kinds of research we&amp;#x2019;ve been doing. There are many other kinds, such as...
  • eyetracking. But the equipment for that is more expensive (and, you know, painful) so I haven&amp;#x2019;t tried it yet.
  • Given the time we have, I&amp;#x2019;ll be focusing on these three, showing you the tool, how we used it, and what action we took. I&amp;#x2019;ll skim over the others. If you go my website or slideshare to get the presentation, you&amp;#x2019;ll find more info about the ones we skipped, or feel free to ask questions later.
  • Steve Krug is the place to start. Also, Chas Grundy did a great presentation on this at last year&amp;#x2019;s eduWeb, and he&amp;#x2019;s done some online seminars for Higher Ed Experts.



  • A fantastic source of live info about your users.

    Place, browser, and connection speed. 20% on ie6. Progressive enhancement.

    What you don&amp;#x2019;t get is intent. That is, if your time on page goes up, is it because they were engaged or frustrated? For intent, you might try...



  • A survey. Regular surveys have their place, and can give you good background information on your users. For instance, we were able to see in 2007 that 80% of our students were logging into Facebook at least once a week&amp;#x2014;and that faculty weren&amp;#x2019;t.

    Doesn&amp;#x2019;t work out so well for questions about your website. You&amp;#x2019;re asking about what they remember about your website, not how it is.

    For that, try a...
  • Surveys have their place, esp. in finding out background info. For instance, we were able to see in 2007 that 80% of our students were logging into Facebook at least once a week&amp;#x2014;and that faculty weren&amp;#x2019;t.

    For specific questions about your website, it doesn&amp;#x2019;t work as well.




  • Work out your sample rate from here. I&amp;#x2019;ve found that 1% to 2% is sufficient.
  • Work out your sample rate from here. I&amp;#x2019;ve found that 1% to 2% is sufficient.

  • Avinash Kaushik is one of the people behind 4Q, and one of the great things about it, besides being free, is that it is short.

  • A lot of the value lies in the comments. But, just reading them isn&amp;#x2019;t enough. These are two real comments.
  • You need to read them twice. Once to come up with categories and once to code.




  • This is a really powerful technique for developing information architecture.

    But, it&amp;#x2019;s pretty time consuming unless you do it online.
  • Why so many more online? You get less data per participant online.

    Also: lots of abandonments online. People have a little trouble understanding the activity when you aren&amp;#x2019;t there to explain it for them.
  • Why $40? you&amp;#x2019;ll need to offer a prize, because this can take a while.

  • Donna Spencer&amp;#x2019;s book on card sorting is the best reference for how to do this well.
  • I&amp;#x2019;ve been using Treejack for the closed sorts.

  • How have we used it? Ran an online card sort to reorganize student life section of our website. 50 cards 384 participants.

    This was one of the questions. Counseling center is part of our health center, so that&amp;#x2019;s where is was on the website. B/c mental health is part of health. But did students see it that way?



  • Results of the reorganization for one subsite.

  • A great technique for getting at a lot of the same things you&amp;#x2019;d use a card sort for, except with a group.

    Learned this from David Poteet of New City Media last year at eduWeb.






  • Tasks come from your user personas.
  • Not good task wording here: the answer (&amp;#x201C;visit campus&amp;#x201D;) is in the question.


















  • You will get people saying &amp;#x201C;Why didn&amp;#x2019;t you get smarter participants?&amp;#x201D;

    I&amp;#x2019;ve done tests with all Ph.D.s, and they didn&amp;#x2019;t really do better in usability tests.

    So it helps to be able to say that the users were at least sort of representative

  • Ethnio provides an on-site recruiting service. You set up screener questions and then you can either call or e-mail your recruits.

    You get people who are just like your users because they are your users.

    You could just use participants who have never been to the site before.

    Cost was too much for my budget this year. And speaking of cost...
  • You can&amp;#x2019;t forget the prize. We find iPod shuffles to work well, but gift cards are an option. Having an incentive, even if it is just a drawing for a prize, not a guarantee, makes a huge difference for recruiting.






  • Not everything you try will work. But do try to fail faster.


  • Fast, Cheap, and Actionable: Creating an Affordable User Research Program (The Director's Cut)

    1. 1. Fast, Cheap, and Actionable: Creating an Affordable User Research Program Michael Powers Director of Web Services Indiana University of Pennsylvania eduWeb 2010 — July 27, 2010, Chicago, IL
    2. 2. In Answer to Your First Question... Buffalo Erie Chicago Cleveland Philadelphia Pittsburgh Morgantown Indianapolis
    3. 3. In Answer to Your First Question... Buffalo Erie Indiana, Pa. Chicago Cleveland Philadelphia Pittsburgh Morgantown Indianapolis
    4. 4. In Answer to Your First Question... Buffalo Erie Indiana, Pa. Chicago Cleveland Philadelphia Pittsburgh Morgantown Indianapolis
    5. 5. On the Web •museyroom.com •twitter.com/mjpowers •www.slideshare.net/michaelpowers
    6. 6. What kinds of user research have you done?
    7. 7. It all started with a usability test
    8. 8. Today •Getting Permission for Research •A Process for User Research •Inventory of Actionable User Research You Can Do Fast and Cheap
    9. 9. User Research Study ↓ User Research Program
    10. 10. 0
    11. 11. 0 Creating Permission
    12. 12. Don’t Ask
    13. 13. “We don’t want a research study.”
    14. 14. “A research study? “Look, just make a red box on the home page, and put this text in there. “I just need to make sure students don’t miss it.”
    15. 15. Nobody wants a research study.
    16. 16. Nobody wants a website.
    17. 17. What We Really •Applicants •Successful Students •Academic Research •Donations
    18. 18. Create a Research •Do a usability test and invite stakeholders to watch •Make research part of every project •Apply previous research to new projects •Show results early and often
    19. 19. Research Process 1. Frame a Good 4. Find Question Participants 2. Know What You 5. Run Study and Already Know Analyze Results 3. Choose Study 6. Take Action Type 7. Measure and Repeat
    20. 20. 1 2 3 4 5 6 7 Frame a Good Question
    21. 21. Good questions are related to your goals
    22. 22. “Students can't find stuff on our website.”
    23. 23. Focusing a How can we 1. Can students find information about decrease support services? attrition? 2. Can students find out what’s happening on campus?
    24. 24. “The admissions website needs work."
    25. 25. Focusing a How can we get 1. Can prospectives find our list of majors? more/more Our tuition qualified information? applicants? 2. Can prospectives register for a visit? 3. Can prospectives successfully apply on line?
    26. 26. Goal-related questions lead to faster, cheaper, more actionable user research
    27. 27. 1 2 3 4 5 6 7 Know What You Already Know
    28. 28. What Do You •Best Practices •Server Logs •Published •Analytics Studies •Past Studies •Data about Your Institution and Students
    29. 29. Get Organized Quantitative Data Qualitative KPI Data Spreadsheet User Personas
    30. 30. Key KPI Performance Indicator
    31. 31. KPIs measure progress toward your goals.
    32. 32. KPI Spreadsheet •Focus on the KPIs •Track over time you've chosen without running multiple tools •Give specific reports to •Include anything stakeholders who numeric: survey request them results, 4Q survey, analytics, server logs, etc.
    33. 33. KPI Spreadsheet Available on museyroom.com
    34. 34. User Personas
    35. 35. Include Scenarios Site Motivation Scenario Features Behaviors Used Search Looks for Find the Checks to •Site • school with see if her •Undergrad “Pre-Law” the right major is Admissions major, pre-Law available Website does not program for •Academics find it. me Link
    36. 36. User Personas •Fiction—but a •If it is wrong, team useful fiction or clients will let you know •Puts what you've learned into a •Helps bring new format you can team members up share with your to speed team and clients
    37. 37. Use these two tools to track your progress over time.
    38. 38. 1 2 3 4 5 6 7 Choose Study Type
    39. 39. If you’ve never done one, do a usability test first.
    40. 40. Focus Groups
    41. 41. Focus Groups
    42. 42. Some Types of User •Usability Test •Card Sort •Surveys •Navigation Test •Pop-up Survey •Layout Test •KJ Session •A/B Test
    43. 43. Some Types of User •Usability Test •Card Sort •Surveys •Navigation Test •Pop-up Survey •Layout Test •KJ Session •A/B Test
    44. 44. Usability Test Photo source: http://www.flickr.com/photos/l-i-n-k/ 3654390818/
    45. 45. Usability Test What You Do: Ask people to do things on your website while they talk about what they do and why they do it Good For: Finding things to fix When To Use: As early as possible and regularly thereafter Participants As few as 3 (3 groups of 5-7 is Needed: better)
    46. 46. Usability Test Tools •No tools are needed Available: •Silverback (Live) •Morae (Live (Live) •Any screen-sharing tool (Skype, GoTo Meeting, WebEx) (Remote) Cost: •UserVue (Remote, no longer $90–$1,000+ available) Record In: •User Profiles
    47. 47. Usability Test Positive Negative •Can be cheap •Behavior of •Holistic view of participants will website issues not be entirely natural •Takes practice to run good tests
    48. 48. Analytics
    49. 49. Analytics What You Do: Analyze the behavior of users on your website using your analytics tool Good For: Seeing real behavior on your website Not So Good Understanding user intent For: When To Use: Collect 24/7, analyze monthly or as needed Participants n/a Needed:
    50. 50. Analytics Tools •Google Analytics Available: •Mint •ClickTracks •Omniture •Many others Cost: •$0–$10,000+
    51. 51. Analytics Positive Negative •Can be free •Can’t understand •You see actual intent use of your site •Can lead to just •Can track looking at changes over time pageviews
    52. 52. Survey
    53. 53. Survey What You Do: Ask people questions about their browsing habits or your website. Good For: Background information about your usershave a question to When To Use: When you answer, possibly yearly Participants Calculate your sample size: Needed: http://www.surveysystem.com/ sscalc.htm
    54. 54. Survey Tools •Survey Monkey Available: •Google Docs •Existing campus survey product •Many others Cost: •$40–$10,000+
    55. 55. Survey Positive Negative •Tools are cheap •Shows you or free attitudes, not •Offer a prize and behavior, so not you can get a great for respectable understanding number of how users use summissions your website •Lots of numbers
    56. 56. Pop-Up Survey
    57. 57. Pop-up Survey What You Do: Ask site visitors questions about their experience on a particular page or the whole Good For: website. user intent; settling Capturing arguments about what belongs When To Use: whereyou have a question to When answer, or all the time Participants Calculate your sample size: Needed: http://www.surveysystem.com/ sscalc.htm
    58. 58. Pop-up Survey Tools •4Q Survey Available: •iPerceptions •CrowdScience •Roll your own Cost: •$0–$10,000+
    59. 59. 4Q Survey 1. Based on today's visit, how would you rate your site experience overall? 2. Which of the following best describes the primary purpose of your visit? 3. Were you able to complete the purpose of your visit today? 4. Comment • If yes: What do you value most about the website? • If no: Please tell us why you were not able to fully complete the purpose of your visit today
    60. 60. “I am “I can never always able easily find to find what what I'm I am looking looking for, for. The if I can find website is it at all.”
    61. 61. Coding Comments • Visual Design • Navigation Positive Positive • Navigation Negative • Visual Design Negative • Content Quality Positive • Usability Positive • Content Quality • Usability Negative Negative
    62. 62. 637,'809':-%&-'*(##%+,';+<' NF,)/+O)-./" (:L8")/2"M*:/58" J+8I)4"K:8+,/"" @8)34:" 1+/2"A:.=4:" 2(405-' 9./5:/5"HI)4+56" (:,)-*:" <+5:"<:)F>G" A.8+-*:" ;@A"B:38+5:C"/./D9E<" <=:>+?>"=),:08+5:" 7)8"9./5:/5";"(::2" ()*+,)-./01+/2)3+4+56" !" #!" $!" %!" &!" '!!" '#!" '$!" '%!" '&!" !"#$%&'()'*(##%+,-'.%+/(+0+1'230-'2(405'
    63. 63. 67'57"3#4'5&"8'9":&*;&"/#00*12" <#&2"%#&'()*=>+,*-.()*=>"?1@" SK1.40T.234# -?Q=#.47#R/?4:=# O0=N.9#P?=014## E=.89?# 6047#F?3B9?# 3#4'5&" >34:?4:#MN.90:;# A0:?#A?.KCL# -?1.2/?# @EF#G?8=0:?H#434I>JA# F3=02/?# AB?C0DC#B.1?5=0:?# <.=#>34:?4:#@#-??7# -./01.23456047.8090:;# !"# $!"# %!"# &!"# '!"# (!"# )!"# *!"# +!"# ,!"# $!!"# !"#$"%#&'()*+,*-.()*"/#00*12&"
    64. 64. Action •Changed our yearly goals to spend more time on these issues
    65. 65. Pop-up Survey Positive Negative •You get info from •Some users real users respond •Easy to set up negatively to being interrupted •Gives you a window into user •Coding can take intent time
    66. 66. Card Sort Photo Source: http://www.flickr.com/photos/ rosenfeldmedia/3343501403/
    67. 67. Card Sort What You Do: Give participants cards with website topics on them, then ask them to put them into piles Good For: that make sense to mental Understanding the them models of your users When To Use: When creating or improving your information architecture/ Participants navigation face 15 face to Needed: 50 online
    68. 68. Card Sort Tools •Index cards Available: •OptimalSort (remote) •WebSort (remote) •xSort Cost: $40-$600
    69. 69. Card Sort Positive Negative •Get a picture of users’ •Hard for participants to mental model of your understand activity content remotely •Understand what labels •Some participants give your users use for your up because the sort can content take a while •Settle arguments about •Data analysis can be what belongs where complex
    70. 70. Open Closed •Users stack •Give users cards then header cards •Give each pile a •They assign name cards to the header cards
    71. 71. Where do we put counseling?
    72. 72. Action •Complete restructuring of the Student Life section of our website •Move counseling websites out of “Health” and into “Support”
    73. 73. One Subsite, Before and After Change between Compared to Comparison Whole Site Periods Pageviews 76% 140% Unique 81% 142% Pageviews Bounce Rate -21% 27%
    74. 74. KJ Session Photo Source: http://www.flickr.com/photos/drewm/ 370231088/
    75. 75. KJ Session What You Do: Gather 4-6 participants Ask: what do people come to our website to find or do? Put answers on sticky notes on the wall Organize and name the groups Good For: Getting the opinion of site owners; learning about users from site owners When To Use: When creating a new information architecture Participants 4–6, or more if you divide into Needed: smaller groups
    76. 76. KJ Session Tools •Walls and sticky notes Available: Cost: $0–$40 Learn More: http://insidenewcity.com/blog/ view/unlock-the-wisdom-of- your-group-host-a-kj-session- for-site-architecture/
    77. 77. KJ Session Positive Negative •Get feedback •Not real users from people who know your users •Get buy-in from stakeholders
    78. 78. Navigation Test
    79. 79. Navigation Test What You Do: Give participants a proposed site structure/navigation and let users drill down to find Good For: different things. a more Verifying an IA in realistic way (not everything is When To Use: visible); finding problem areas To verify a new information architecture Participants 50 Needed:
    80. 80. Navigation Test Tools •Treejack Available: Cost: $150–$700
    81. 81. Navigation Test Positive Negative •Tests your •Doesn’t look like sitemap/ your site, so on- navigation site results might without having to be different build it •Can locate problems with your IA
    82. 82. Action •Rearranged Admissions Site navigation to •Combine repeated material •Put “Costs and Tuition” higher in the site tree
    83. 83. Layout Test
    84. 84. Layout Test What You Do: Participants respond to a static image of your site Good For: Getting reactions to visual designs and wireframes without building the whole thing. When To Use: Before building out a new design Participants About 50 per test Needed:
    85. 85. Layout Test Tools •Paper Available: •PowerPoint •Five-Second Usability Test •Chalkmark •Usabila Cost: $40–$600
    86. 86. Layout Test Positive Negative •Get reactions to •No interactivity real designs •Users can be without building confused when out a whole site doing this remotely
    87. 87. A/B Test
    88. 88. A/B Test What You Do: Test two (or more) versions of your live website Good For: Testing page variations When To Use: Testing calls to action or other pages that have a clear destination you Participants Tool will tell Needed:
    89. 89. A/B Test Tools •Google Website Optimizer Available: •Reedge.com Cost: $0–???
    90. 90. A/B Test Positive Negative •Test real •Harder to set up variations on your in a CMS live site with real •Harder to use users when a page has many destinations
    91. 91. Easy to do with Google Website Optimizer A Go al B
    92. 92. Harder to do with Google Website Go Optimizer A Go Go B Go Go
    93. 93. 1 2 3 4 5 6 7 Find Participants
    94. 94. The Right Number •Different methods have different requirements •Plan to recruit more than you need •But don’t test more than you need
    95. 95. The Right Kind of •Steve Krug: “Recruit loosely and grade on a curve” •Good advice, but when possible try to recruit participants that are more like your users •For instance: faculty and students see your website very differently
    96. 96. Finding •People you •Local high know schools (talk to a guidance •Students, counselor) faculty, and staff at your school •On your website (link on homepage or pop-up)
    97. 97. Photo source: http://en.wikipedia.org/wiki/ File:Ethnio_recruit_ad.png
    98. 98. Offer a prize
    99. 99. 1 2 3 4 5 6 7 Run Study and Analyze Results
    100. 100. Keep track of results in KPI spreadsheet and personas
    101. 101. 1 2 3 4 5 6 7 Take Action
    102. 102. Share results and proposed solutions with stakeholders
    103. 103. 1 2 3 4 5 6 7 Measure and Repeat
    104. 104. Share results
    105. 105. Challenge: Do some sort of user research each month
    106. 106. Research Process 1. Frame a Good 4. Find Question Participants 2. Know What You 5. Run Study and Already Know Analyze Results 3. Choose Study 6. Take Action Type 7. Measure and Repeat
    107. 107. More Information •museyroom.com •twitter.com/mjpowers •www.slideshare.net/michaelpowers

    ×