DIY:  Research On a Shoestring Budget   J. Todd Bennett Managing Partner, decimal152
Why do research? <ul><li>Justify </li></ul><ul><li>Improve </li></ul><ul><li>Monitor </li></ul><ul><li>Inform </li></ul>
This is not  academic  research
Quantitative vs. Qualitative <ul><li>Qualitative </li></ul><ul><ul><ul><li>Best in earlier phases of project. Not sure wha...
Some qualitative techniques <ul><li>Focus Groups </li></ul><ul><li>Interviews </li></ul><ul><li>Observations  </li></ul><u...
Some quantitative techniques <ul><li>Surveys and questionnaires </li></ul><ul><li>Data mining/ modeling </li></ul>
Mixed? <ul><li>Web analytics </li></ul><ul><li>Usability testing </li></ul>
The qualitative helps give  meaning and context  to the quantitative
Steps in the process <ul><li>Define the problem </li></ul><ul><li>Design the study  </li></ul><ul><li>Collect data  </li><...
Lay the foundation <ul><li>The problem </li></ul><ul><ul><li>“ My boss wants me to do a survey” </li></ul></ul><ul><li>The...
 
How to…
Focus Groups <ul><li>group interview  with 10-12 members of a target audience (students, faculty, alumni, etc.), conducted...
Recruiting participants <ul><li>Need 8-12 participants, so recruit 20-25. </li></ul><ul><li>They should:  </li></ul><ul><u...
Focus group location <ul><li>Don’t need a fancy room </li></ul><ul><li>Free from distractions– private </li></ul><ul><ul><...
Choosing a moderator <ul><li>Professionally  trained , ideally </li></ul><ul><li>Excellent speaking &  listening  skills <...
Role of the moderator <ul><li>promote  open discussion  and to draw out people’s differences </li></ul><ul><li>probe for  ...
Record the session <ul><li>Digital audio recording best </li></ul><ul><li>Transcribe:  http://www.verbalink.com/ </li></ul...
Lay the ground rules <ul><li>Introduction </li></ul><ul><li>Why are they here? </li></ul><ul><li>Encourage participation- ...
<ul><li>Sample Introduction Script </li></ul><ul><li>Welcome to our session on [PROJECT]. Thank you for taking the time to...
The questions <ul><li>8-12 broad questions prepared </li></ul><ul><ul><li>Several probing questions ( prompts ) for each <...
The results <ul><li>Notes/ Transcripts </li></ul><ul><li>Content analysis </li></ul><ul><li>Identify hot-button issues or ...
A few applications <ul><li>Image studies </li></ul><ul><li>Design evaluations </li></ul><ul><li>Website audience needs ass...
Surveys
Survey Options <ul><li>Mail </li></ul><ul><li>Telephone </li></ul><ul><li>Online </li></ul><ul><ul><li>Solicited </li></ul...
Steps in conducting surveys <ul><li>Decide the purpose </li></ul><ul><li>Develop the research questions </li></ul><ul><li>...
Purpose  <ul><li>What are you trying to do/learn? </li></ul><ul><ul><li>May be more than one objective </li></ul></ul><ul>...
Research questions <ul><li>The  guide  to what you will ask in the survey </li></ul><ul><li>Survey questions should  alway...
Purpose:  to compare the content preferences of our various website audiences. <ul><li>Research questions: </li></ul><ul><...
Target Population <ul><li>Who do you send the survey to? </li></ul><ul><ul><li>Determine your target population </li></ul>...
Sampling <ul><li>How many people do you need to distribute it to? </li></ul><ul><ul><li>Experiment with online sample size...
Sampling
Writing the survey <ul><li>NEVER ask a question unless you intend to do something with the info </li></ul><ul><ul><li>Nice...
Writing the survey <ul><li>Objectionable questions </li></ul><ul><ul><li>Does it need to be asked? </li></ul></ul><ul><ul>...
Attitudes, opinions, values <ul><li>Use Likert or Semantic Differential scales </li></ul><ul><li>Likert </li></ul><ul><ul>...
Tips for using scales <ul><li>Scales usually indicate  intensity  of feeling or  amount  of something </li></ul><ul><li>Do...
Building the survey <ul><li>Surveymonkey </li></ul><ul><ul><li>Free for 100 responses. $19.95/ 1000 responses, $200 annual...
The invite <ul><ul><li>The purpose of the survey </li></ul></ul><ul><ul><li>How will their responses help? </li></ul></ul>...
Pilot test <ul><ul><li>Feedback on the invite  </li></ul></ul><ul><ul><li>Test the flow (does the order make sense, is it ...
Distribution of surveys <ul><ul><li>First follow-up in 5-7 days </li></ul></ul><ul><ul><li>Final reminder 1-2 days before ...
Analysis and reporting <ul><li>Basic  frequencies  usually sufficient </li></ul><ul><li>Cross-tabs by audience/group/varia...
Competitive analysis <ul><li>Websites </li></ul><ul><li>Publications  </li></ul><ul><li>News </li></ul><ul><li>Programs </...
Competitive analysis <ul><li>Heuristics (rules of thumb) </li></ul><ul><ul><li>Navigation </li></ul></ul><ul><ul><li>Aesth...
“Why aren’t we doing that?” <ul><li>It’s easy to judge without context.   </li></ul><ul><li>Institutions’ goals and strate...
Usability testing
Talk aloud method <ul><li>Develop scenarios  </li></ul><ul><ul><li>Task and information based </li></ul></ul><ul><li>Compl...
Data collection <ul><li>Don’t need fancy software, but it helps </li></ul><ul><ul><li>Morae- fairly expensive </li></ul></...
Prototyping <ul><li>Paper or HTML </li></ul><ul><li>Useful before development </li></ul><ul><li>Quick, iterative </li></ul...
Prototyping flickr.com/photos/bensauer flickr.com/photos/cesarastudillo
Card sorting <ul><li>Useful for site  architecture  (navigation, organization, labels) </li></ul><ul><li>Looking for  patt...
Card Sorting .flickr.com/photos/yandle
5-second tests www.fivesecondtest.com <ul><li>Classic </li></ul><ul><ul><li>list the things they can recall after viewing ...
Web analytics <ul><li>Useful for measurement, but also for research </li></ul><ul><li>Do people do what they say they do? ...
Questions? <ul><li>Contact me: </li></ul><ul><li>[email_address] </li></ul><ul><li>www.twitter.com/jtoddb </li></ul><ul><l...
Upcoming SlideShare
Loading in …5
×

DIY: Research on a shoestring budget

1,758 views

Published on

Presentation given at the CASE Communications, Marketing & Technology Conference in Boston on April 15, 2009.

Learn the tools of the trade for do-it-yourself research for little or no money. This session will teach you how to conduct focus groups, surveys, usability tests and more.

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,758
On SlideShare
0
From Embeds
0
Number of Embeds
21
Actions
Shares
0
Downloads
44
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • DIY: Research on a shoestring budget

    1. 1. DIY: Research On a Shoestring Budget J. Todd Bennett Managing Partner, decimal152
    2. 2. Why do research? <ul><li>Justify </li></ul><ul><li>Improve </li></ul><ul><li>Monitor </li></ul><ul><li>Inform </li></ul>
    3. 3. This is not academic research
    4. 4. Quantitative vs. Qualitative <ul><li>Qualitative </li></ul><ul><ul><ul><li>Best in earlier phases of project. Not sure what you’re looking for. </li></ul></ul></ul><ul><li>Quantitative </li></ul><ul><ul><ul><li>Best in later phases. Test hypotheses. </li></ul></ul></ul>
    5. 5. Some qualitative techniques <ul><li>Focus Groups </li></ul><ul><li>Interviews </li></ul><ul><li>Observations </li></ul><ul><li>Subjective analysis </li></ul><ul><li>Environmental Scans </li></ul>
    6. 6. Some quantitative techniques <ul><li>Surveys and questionnaires </li></ul><ul><li>Data mining/ modeling </li></ul>
    7. 7. Mixed? <ul><li>Web analytics </li></ul><ul><li>Usability testing </li></ul>
    8. 8. The qualitative helps give meaning and context to the quantitative
    9. 9. Steps in the process <ul><li>Define the problem </li></ul><ul><li>Design the study </li></ul><ul><li>Collect data </li></ul><ul><li>Analyze data </li></ul><ul><li>Write report/ presentation </li></ul>
    10. 10. Lay the foundation <ul><li>The problem </li></ul><ul><ul><li>“ My boss wants me to do a survey” </li></ul></ul><ul><li>The research questions </li></ul><ul><ul><li>What do you want to know? </li></ul></ul><ul><li>The method </li></ul><ul><ul><li>Which method will help you get there? </li></ul></ul><ul><li>Be careful </li></ul>
    11. 12. How to…
    12. 13. Focus Groups <ul><li>group interview with 10-12 members of a target audience (students, faculty, alumni, etc.), conducted by a moderator </li></ul><ul><ul><li>relatively low cost </li></ul></ul><ul><ul><li>probe deeper as issues arise </li></ul></ul><ul><ul><li>results are almost immediate </li></ul></ul><ul><ul><li>should not be generalized to the greater population </li></ul></ul>
    13. 14. Recruiting participants <ul><li>Need 8-12 participants, so recruit 20-25. </li></ul><ul><li>They should: </li></ul><ul><ul><li>represent your target audience and be as homogenous as possible </li></ul></ul><ul><ul><li>not have expertise in market research or marketing </li></ul></ul><ul><li>Provide an incentive. Use existing groups only if necessary. </li></ul>
    14. 15. Focus group location <ul><li>Don’t need a fancy room </li></ul><ul><li>Free from distractions– private </li></ul><ul><ul><li>No public spaces </li></ul></ul><ul><li>Not too large or noisy </li></ul><ul><li>Seated in a circle or around a table </li></ul><ul><li>Pay attention to lighting and temperature </li></ul>
    15. 16. Choosing a moderator <ul><li>Professionally trained , ideally </li></ul><ul><li>Excellent speaking & listening skills </li></ul><ul><li>Not too close to the project at hand </li></ul><ul><li>Objective </li></ul>
    16. 17. Role of the moderator <ul><li>promote open discussion and to draw out people’s differences </li></ul><ul><li>probe for details or move things forward- keep focused </li></ul><ul><li>ensure everyone participates and gets a chance to speak </li></ul><ul><li>LISTEN - do not control conversation or answer questions </li></ul>
    17. 18. Record the session <ul><li>Digital audio recording best </li></ul><ul><li>Transcribe: http://www.verbalink.com/ </li></ul><ul><li>Have a notetaker in the room </li></ul><ul><ul><li>Note the non-verbal </li></ul></ul><ul><ul><li>Back-up for audio </li></ul></ul>
    18. 19. Lay the ground rules <ul><li>Introduction </li></ul><ul><li>Why are they here? </li></ul><ul><li>Encourage participation- no right or wrong answers </li></ul><ul><li>Respectful of disagreement </li></ul><ul><li>Ensure confidentiality </li></ul><ul><li>Permission to record </li></ul>
    19. 20. <ul><li>Sample Introduction Script </li></ul><ul><li>Welcome to our session on [PROJECT]. Thank you for taking the time to meet with us and to discuss [PROJECT]. My name is [NAME] . Assisting me is [NAME]. </li></ul><ul><li>You were selected because you [AUDIENCE]. We are particularly interested in your views because [REASON]. </li></ul><ul><li>I’ll start by asking some questions and would like you to respond to them. There are no right or wrong answers . Please feel free to share your point of view even if it differs from what others have said. We do ask that you are all respectful of others’ opinions and give everyone a chance to share their views. We also ask that only one person speak at a time . Because we have a number of issues to discuss, I may have to cut you short and ask to move on. Please don’t be offended. We just need to keep things moving so we can finish on-time. </li></ul><ul><li>We want you to be as open and honest as possible, so we guarantee that everything you say will be kept confidential in the sense that your name will never be linked to your comments shared with [CLIENT]. However, we would like to record our conversation because it is difficult to write and follow the discussion at the same time and it’s important we don’t miss any of your comments. Do I have everyone’s consent to record the session ? If so, let’s begin with the first question. </li></ul>
    20. 21. The questions <ul><li>8-12 broad questions prepared </li></ul><ul><ul><li>Several probing questions ( prompts ) for each </li></ul></ul><ul><li>Start with the broadest questions first </li></ul><ul><li>Several seconds of silence is ok </li></ul><ul><ul><li>Avoid the temptation to give examples </li></ul></ul><ul><li>Unexpected paths are ok </li></ul>
    21. 22. The results <ul><li>Notes/ Transcripts </li></ul><ul><li>Content analysis </li></ul><ul><li>Identify hot-button issues or areas for additional research </li></ul>
    22. 23. A few applications <ul><li>Image studies </li></ul><ul><li>Design evaluations </li></ul><ul><li>Website audience needs assessment </li></ul>
    23. 24. Surveys
    24. 25. Survey Options <ul><li>Mail </li></ul><ul><li>Telephone </li></ul><ul><li>Online </li></ul><ul><ul><li>Solicited </li></ul></ul><ul><ul><li>Site intercept </li></ul></ul><ul><li>In-person </li></ul><ul><ul><li>“mall intercept” </li></ul></ul><ul><ul><li>Group setting </li></ul></ul>
    25. 26. Steps in conducting surveys <ul><li>Decide the purpose </li></ul><ul><li>Develop the research questions </li></ul><ul><li>Define the target population </li></ul><ul><li>Create the sample </li></ul><ul><li>Write & design the survey instrument </li></ul><ul><li>Pilot test the questionnaire </li></ul><ul><li>Prepare letter/email/intro to distribute </li></ul><ul><li>Follow-up </li></ul><ul><li>Analyze and report </li></ul>
    26. 27. Purpose <ul><li>What are you trying to do/learn? </li></ul><ul><ul><li>May be more than one objective </li></ul></ul><ul><ul><li>Example: </li></ul></ul><ul><ul><li>The purpose is to compare the content preferences of our various website audiences . </li></ul></ul>
    27. 28. Research questions <ul><li>The guide to what you will ask in the survey </li></ul><ul><li>Survey questions should always be tied back to your research questions . </li></ul>
    28. 29. Purpose: to compare the content preferences of our various website audiences. <ul><li>Research questions: </li></ul><ul><li>What content is most important to prospective students, faculty and alumni? </li></ul><ul><li>What tasks are most important to prospective students, faculty and alumni? </li></ul><ul><li>Do differences exist in preferences of prospective students, faculty and alumni? </li></ul>
    29. 30. Target Population <ul><li>Who do you send the survey to? </li></ul><ul><ul><li>Determine your target population </li></ul></ul><ul><ul><li>Is the entire population known or available? </li></ul></ul><ul><ul><li>Where does that list come from? </li></ul></ul><ul><ul><ul><li>HR, Admissions, Alumni database, SIS, purchased lists (chamber of commerce, etc) </li></ul></ul></ul>
    30. 31. Sampling <ul><li>How many people do you need to distribute it to? </li></ul><ul><ul><li>Experiment with online sample size calculator, confidence interval .05 </li></ul></ul><ul><ul><li>Expect 10-20% response from prospective students (or others who don’t know you). 60%+ from internal audiences </li></ul></ul><ul><ul><li>Use your gut – how many responses would make you feel comfortable with results? </li></ul></ul>
    31. 32. Sampling
    32. 33. Writing the survey <ul><li>NEVER ask a question unless you intend to do something with the info </li></ul><ul><ul><li>Nice to know isn’t reason enough </li></ul></ul><ul><li>Tie each question to your research questions </li></ul><ul><li>Careful with language - avoid jargon, use simple words, watch for bias and unequal comparisons </li></ul><ul><li>Avoid “why” questions </li></ul><ul><li>Don’t ask “when” questions </li></ul><ul><li>No hypotheticals </li></ul>
    33. 34. Writing the survey <ul><li>Objectionable questions </li></ul><ul><ul><li>Does it need to be asked? </li></ul></ul><ul><ul><li>For income or demographics, give ranges </li></ul></ul><ul><li>Behavior </li></ul><ul><ul><li>Within 6 months </li></ul></ul><ul><ul><li>Ask “typically” or “usually” </li></ul></ul><ul><li>Attitude questions </li></ul><ul><ul><li>Avoid middle alternative </li></ul></ul><ul><ul><li>Or follow with an intensity question: “How strongly do you feel about…” </li></ul></ul>
    34. 35. Attitudes, opinions, values <ul><li>Use Likert or Semantic Differential scales </li></ul><ul><li>Likert </li></ul><ul><ul><ul><li>Strongly agree, agree, disagree, strongly disagree </li></ul></ul></ul><ul><ul><ul><li>Very happy, happy, unhappy, very unhappy </li></ul></ul></ul><ul><li>Semantic Differential (bipolar adjectives) </li></ul><ul><ul><ul><li>Efficient 7 6 5 4 3 2 1 Inefficient </li></ul></ul></ul><ul><ul><ul><li>Fair +3, 2, 1, 0, -1, -2, -3 Unfair </li></ul></ul></ul>
    35. 36. Tips for using scales <ul><li>Scales usually indicate intensity of feeling or amount of something </li></ul><ul><li>Do not include “don’t know” or “undecided” on Likert </li></ul><ul><li>Use an even number of points, equally distant </li></ul><ul><li>Avoid absolutes like “always” or “never” as end points </li></ul><ul><li>Ask one question to determine if they are qualified to answer the following one </li></ul><ul><ul><li>“ Have you ever… if yes, …” </li></ul></ul>
    36. 37. Building the survey <ul><li>Surveymonkey </li></ul><ul><ul><li>Free for 100 responses. $19.95/ 1000 responses, $200 annually unlimited </li></ul></ul><ul><li>Does your institution have a tool? </li></ul>
    37. 38. The invite <ul><ul><li>The purpose of the survey </li></ul></ul><ul><ul><li>How will their responses help? </li></ul></ul><ul><ul><li>What will they get in return? </li></ul></ul><ul><ul><li>Ensure confidentiality </li></ul></ul><ul><ul><li>Provide a deadline </li></ul></ul><ul><ul><li>How to get results </li></ul></ul><ul><ul><li>Thanks </li></ul></ul>
    38. 39. Pilot test <ul><ul><li>Feedback on the invite </li></ul></ul><ul><ul><li>Test the flow (does the order make sense, is it monotonous?) </li></ul></ul><ul><ul><li>Time to complete </li></ul></ul><ul><ul><li>Do the response patterns make sense? Are they consistent? </li></ul></ul><ul><ul><li>Do any questions cause confusion? </li></ul></ul>
    39. 40. Distribution of surveys <ul><ul><li>First follow-up in 5-7 days </li></ul></ul><ul><ul><li>Final reminder 1-2 days before close of survey </li></ul></ul><ul><ul><li>You will get exponentially fewer responses to follow-ups after the first. Don’t harass your sample! </li></ul></ul>
    40. 41. Analysis and reporting <ul><li>Basic frequencies usually sufficient </li></ul><ul><li>Cross-tabs by audience/group/variable optional </li></ul><ul><li>Surveymonkey produces a report of frequencies in nice charts </li></ul><ul><li>Excel also works fine. No real need for SPSS. </li></ul>
    41. 42. Competitive analysis <ul><li>Websites </li></ul><ul><li>Publications </li></ul><ul><li>News </li></ul><ul><li>Programs </li></ul>
    42. 43. Competitive analysis <ul><li>Heuristics (rules of thumb) </li></ul><ul><ul><li>Navigation </li></ul></ul><ul><ul><li>Aesthetic, readability & design </li></ul></ul><ul><ul><li>Content </li></ul></ul><ul><ul><li>Multimedia and features </li></ul></ul><ul><ul><li>External Web presence </li></ul></ul>
    43. 44. “Why aren’t we doing that?” <ul><li>It’s easy to judge without context. </li></ul><ul><li>Institutions’ goals and strategies may differ. </li></ul><ul><li>While one might perceive a site/publication/strategy to be “better” than another, each may be equally successful at fulfilling its intended purpose. </li></ul>
    44. 45. Usability testing
    45. 46. Talk aloud method <ul><li>Develop scenarios </li></ul><ul><ul><li>Task and information based </li></ul></ul><ul><li>Complete the tasks, talking aloud </li></ul><ul><li>Observe paths, clicks, verbal impressions, body language </li></ul><ul><ul><li>Note areas of difficulty </li></ul></ul>
    46. 47. Data collection <ul><li>Don’t need fancy software, but it helps </li></ul><ul><ul><li>Morae- fairly expensive </li></ul></ul><ul><ul><li>Silverback- $49, for Mac only </li></ul></ul><ul><ul><li>Cool service- www.userfly.com </li></ul></ul>
    47. 48. Prototyping <ul><li>Paper or HTML </li></ul><ul><li>Useful before development </li></ul><ul><li>Quick, iterative </li></ul><ul><ul><li>Mock-up, test, revise, test again </li></ul></ul>
    48. 49. Prototyping flickr.com/photos/bensauer flickr.com/photos/cesarastudillo
    49. 50. Card sorting <ul><li>Useful for site architecture (navigation, organization, labels) </li></ul><ul><li>Looking for patterns (mental models) of how users group and organize content </li></ul><ul><li>Open vs. closed card sorting </li></ul><ul><li>Focuses on content, but not tasks </li></ul>
    50. 51. Card Sorting .flickr.com/photos/yandle
    51. 52. 5-second tests www.fivesecondtest.com <ul><li>Classic </li></ul><ul><ul><li>list the things they can recall after viewing your site for 5 sec. </li></ul></ul><ul><ul><li>good for discovering the most prominent calls to action on pages </li></ul></ul><ul><li>Compare </li></ul><ul><ul><li>choose a preferred interface after viewing each for 2.5 sec. </li></ul></ul><ul><ul><li>good for assessing design variations </li></ul></ul><ul><li>Sentiment </li></ul><ul><ul><li>choose most and least favorite elements after viewing your site for 5 sec. </li></ul></ul><ul><ul><li>good for identifying strengths and weaknesses in the design </li></ul></ul>
    52. 53. Web analytics <ul><li>Useful for measurement, but also for research </li></ul><ul><li>Do people do what they say they do? </li></ul><ul><li>Look for ways the behavioral data supports/refutes what you learned in other research </li></ul>
    53. 54. Questions? <ul><li>Contact me: </li></ul><ul><li>[email_address] </li></ul><ul><li>www.twitter.com/jtoddb </li></ul><ul><li>404-551-3915 </li></ul>

    ×