This document discusses low-cost user research methods that non-profits can use to test their websites and improve outcomes. It promotes brief user tests at Starbucks to observe people completing tasks on a website. Card sorting is presented to understand how users organize information. Surveys are suggested to answer specific questions about current users, with tips on keeping them short and analyzing responses. The presentation aims to demonstrate user research can be done easily and cheaply.

1. Game Changing User Research
Research as cheap & easy as a cup of coffee
PRESENTED BY ALISSA RUEHL & NATHAN RENIHAN

2. TWEET!
#BBCON
#UserResearch
@ALISSARU

3. WHY?

4. People in the world
Visitors to your site
Visitors to
your donation
form
Donors

5. People in the world
Visitors to your site
Visitors to
your donation
form
Donors

6. People in the world
Visitors to your site
Visitors to your
donation form
Donors

7. People in the world
10% more
Visitors to your site
10% more
Visitors to your
donation form
10% more
Donors

8. People in the world
Visitors to your site
10% more
Visitors to your
donation form
10% more
Donors

9. People in the world
Visitors to your site
Visitors to
your donation
form
10% more
Donors

10. As an insider
your perspective is very different.

11. Bite sized
research

12. PROOF
• We’ve taken our own medicine
• Onsite research:
- Total time: 20 minutes
- Total cost: $0

13. vs
OUR USER RESEARCH
4 11

14. WHAT?

15. What’s the mystery you’re trying to solve?

16. • Why are people abandoning the donation form?
• Why don’t more people give recurring gifts?
• What are the main things people are looking for on
my site?
• Do people notice the promotion we’re doing?
• How should we organize our new website (or
newsletter, etc)
POTENTIAL CASES

17. WHO?

18. People involved with your organization

19. The general public

20. PARTICIPANTS
• Good targets
- People somewhat involved with your organization
• Event participants
• Donors
- The general public
• Avoid
- Employees
- Board
• Generally avoid*
- Major volunteers
- Friends & family

21. HOW?

22. •Starbucks testing
•Card sorts
•Surveys
3 QUICK & EASY METHODS

23. STARBUCKS TESTING
When you want to know if average people run into
obstacles when trying to complete a task

24. CARD SORTS
When you want to know how people think information
should be organized

25. SURVEYS
When you want to answer specific questions about
people currently involved with your organization

26. STARBUCKS TESTING
When you want to know if average people run into
obstacles when trying to complete a task

29. RULES FOR TESTING

30. TASK BASED

31. VERBAL FOLLOW UP

32. IT WORKS ABOUT LIKE THIS

34. ALWAYS BE PREPARED

35. 1. Make sure you pick something that regular people
use, you will get regular people at Starbucks.
2. Test internally for practice
3. Test internally to make sure it is truly a 5 minute
test
4. Take 1 person with you. 1 person stays with laptop,
1 person solicits.
BEFORE YOU GO

36. 1. Make sure Starbucks is ok with it. Talk to the
manager on duty first, have a pitch ready for that
conversation so you can easily explain what you
want to do.
2. Bring your charger, mouse, test internet connection
(if needed), etc.
3. Grab a table - go early or avoid rush times.
BEFORE YOU START

37. 1. Talk to the people that are sitting around, you are buying
their NEXT cup of coffee, avoid the people in line.
2. Put your best foot forward, pick someone that will get a
good response.
3. Work on your pitch, it has to be faster than an elevator
pitch. Think of your pitch like a tweet: short, sweet and
to the point.
4. Make it obvious that you are a non-profit, lead with that,
wear the gear, beat people over the head with it; people
are more likely to help.
5. It helps to stress that this will take 5 minutes or less.
WHEN YOU ARE READY

38. 1. Don’t record names
2. Do get a wide cross section of users (age,
gender, race, etc)
3. Test the savvy level of the user, make sure they
aren’t extremely savvy or extremely novice.
4. Give them their task
5. WATCH
6. Try not to help
7. Ask follow up
WHEN YOU HAVE A USER

39. • If you find a massive gotcha, STOP. No need to keep
testing. Go back and fix it and try again another day.
• Once you have 3-5 users you probably have some really
good feedback, no reason to keep going. Maybe switch
to a new thing to test.
• This whole process might take 1.5-4 hrs
• You will most likely learn things that will blow your mind.
Things you never would have thought of.
WHEN THE TEST IS OVER

41. QUESTIONS ON STARBUCKS TESTING

42. CARD SORTS
When you want to know how people think information
should be organized

43. People in the world
Visitors to your site
Visitors to
your donation
form
Donors

44. 07/10/2013 44

45. 07/10/2013 45

46. 07/10/2013 46

47. 07/10/2013 47

48. vs
OUR USER RESEARCH
12 3

49. • Write down each 2nd level menu item on
its own note card
- In this example
• Write down
– News Archive
– Wayne’s blog
– Press releases
– Media contacts
• Don’t write
– NEWS
CARD SORTS

50. 1. Get a participant and some table space
2. Give them
- your note cards
- some blank cards or post-it notes
• (No more than you have space in your top level
navigation)
3. Let them sort the pre-written note cards into
groups
4. Ask them to label each group (using the
blank note cards)
5. Record the results
6. Repeat at least 10 times
CARD SORTS

51. 07/10/2013 51

52. • Tips
- Be sure to stay quiet while the participant is working
- Ask them to think aloud and listen to their commentary
- Take a photo when each person is done to capture their
work
• Variations
- Pre-write the headers (closed card sort)
- Conduct the card sort online
• http://www.optimalworkshop.com/optimalsort.htm
CARD SORTS

54. QUESTIONS ON CARD SORTS

55. SURVEYS
When you want to answer specific questions about
people currently involved with your organization

56. POLL: WHO HAS CREATED OR SENT A SURVEY?

57. POLL: WHO HAS ABANDONED A SURVEY THEY’VE
RECEIVED?

58. • 11 out of 15 people surveyed had never done any
user testing
- Surveys were the way most people do user
research
• 15 of 15 people surveyed had abandoned a survey
half way through because it was too long or too
confusing
OUR USER RESEARCH

64. • Keep it short
- Usually have 5 – 15 questions
- Should take less than 5 minutes to complete
• Always have a general comment field
- Let them speak to you, not just answer questions
• Have someone proofread your survey
- Make sure none of the questions are confusing
• Be neutral
- You want information, not vanity metrics
• You don’t need to survey everyone
- How many responses do you need to answer your questions?
• You can use statistics, or
• General questions, homogeneous audience ~100 responses
- Assume a response rate of 10 – 20% for people somewhat involved with
your organization
SURVEY ETIQUETTE

66. • Think of each question as testing a hypothesis. It
can help to word them that way
- Less useful: We should ask how many times per year they visit our
museum
- Better: I suspect that most people visit either only once per year or
more than 4 times. This will affect how we price and market our
membership programs. If we find a lot of people who visit 3-4 times
a year, we might decide to adjust our programs
SURVEY ANALYSIS TIPS

67. • Avoid “neutral” options
- Very likely
- Somewhat likely
- Somewhat unlikely
- Very unlikely
• (But you can use “not applicable”, if applicable)
• Experiment with forced rankings
• If you’re doing a large survey, consider how many open-ended
questions you want to analyze
SURVEY ANALYSIS TIPS

68. QUESTIONS ON SURVEYS

69. OTHER THINGS

70. WEB ANALYTICS
When you want to know how people are getting to your
site & what they are doing on your site

71. 07/10/2013 71

72. 07/10/2013 72
Focus Groups

73. 07/10/2013 73
Formal Usability Studies

74. 07/10/2013 74
Blackbaud Discovery

75. THIS PRESENTATION BROUGHT TO YOU BY
BLACKBAUD USER EXPERIENCE
• We want to learn from you so we can design
experiences that delight
• Participate in Blackbaud Discovery!
- Help us with our research
- Have your voice heard for the products you use
• Sign up to participate at
www.blackbaud.com/discovery
• Stop by the Discovery booth in the expo hall

76. FOR MORE INFORMATION

77. LINKS
• Alissa’s blog posts on npengage
- http://www.npengage.com/author/alissa/
- http://www.npengage.com/online-fundraising/testing-your-donation-
form-part-1-why/
- http://www.npengage.com/online-fundraising/testing-your-donation-
form-part-2-designing/
• Other posts
- http://www.npengage.com/web-design/nonprofit-web-design-process-part-
surveys-and-interviews/

78. LINKS
• Analytics
- http://www.google.com/analytics/
- http://www.npengage.com/content-marketing/google-analytics-1/
• Starbucks testing
- http://www.mattbaline.net/3-lessons-from-guerrilla-mobile-usability-at-
starbucks/
- http://joshledgard.com/i-wish-id-thought-of-that-operation-starbucks/
- http://blog.kooaba.ch/2012/02/developing-shortcut-starbucks-usability-
testing/
• Card sorting
- http://rosenfeldmedia.com/books/card-sorting/
– http://www.optimalworkshop.com/optimalsort.htm
• General
- http://uxdesign.smashingmagazine.com/2009/05/15/optimizing-conversion-
rates-its-all-about-usability/
- http://www.nngroup.com/articles/non-profit-websites-donations/
- http://www.sensible.com/rsme.html