This document summarizes three user research hacks presented by Gene Smith at UX Lisbon in 2012. The hacks provide kludgy but effective ways to conduct user research when clients have small budgets, limited time, or lack executive support. The first hack uses a content prioritization card sort to understand users' information needs. The second hack uses A/B testing in SurveyMonkey to evaluate website mockups. The third hack boosts interview responses by having participants select emotion cards to describe their experiences. The goal is to provide low-cost, high-impact ideas for user research projects.

1. User Research Hacks
UX Lisbon | May 16, 2012
Gene Smith | @gsmith
#uxhacks

2. The Problem
Client wants major Client has a small
insights into their budget, limited time,
customers and or no executive buy-in
design problems to conduct research

3. The Problem
hack
Client wants major of solvingClient has
kludgy but effective way a problema small
insights into their budget, limited time,
customers and or no executive buy-in
design problems to conduct research

4. The Problem
hack
Client wants major of solvingClient has
kludgy but effective way a problem a small
insights into their budget, limited time,
customers and
user research hack or no executive buy-in
design problems of getting some data that will give you
kludgy but effective way to conduct research
insight into users’ needs so you can design something halfway
decent

5. Kludgy but effective
• Simple tools
• Creative experimental design or just
some creative thinking about how to
improve our results
• Spreadsheets, pivot tables & other
post-study analysis tools

6. This Presentation
• Three user research hacks
1. Determining users’ content priorities
2. A/B testing mock-ups
3. Getting better interview responses
• Goal: give you some ideas for your next
low-budget, high-impact user research
project

7. #1. Content Priorities
The client: regional financial services
company with 2,500 employees and
100+ branches
The project: a ground-up Intranet
redesign
The problem: no mandate to conduct
research with front-line staff.

8. Our Solution
• Content prioritization card sort
• Include questions about region and role
to segment the data.
• Distribute the surveys directly to
branch managers and use the tell-two-
friends method.

12. Why Content Prioritization?
• We needed to understand actual behaviour
• This survey design let us extract a lot
insights from one data set.
• What content do people need daily?
• How does that differ by job function?
• Are there significant differences between
different locations or job functions?

23. Ranking Content Use
• Hunch: front-line staff relied on Intranet
heavily for business-critical information. How
could we show that with the data we had?
• We segmented respondents into two
groups: all front-line staff and corporate staff
• We looked at the % of content each group
used daily and ranked their responses

28. 1 2
Front-line staff: > 95% using two Intranet resources daily
1 Corporate staff: 55% using two Intranet resources daily
Front-line staff: > 80% using 12 Intranet resources daily
2 Corporate staff: < 30% using 12 Intranet resources daily

29. Results
• Much better understanding of information
needs for major job functions
• Design meaningful role-based
customization features
• Confirmed that front-line staff relied on
the Intranet more heavily
• Secured additional participation from
front-line staff

30. #2. A/B Testing Mock-ups
The client: large utility company
The project: assist them with usability
evaluation for their website
The problem: how do we help them
choose between mock-ups?

31. Experimental Design:
A Digression
• In general, good experiments will meet
these two criteria
• random sampling of a population
• random assignment to one or more
experimental groups

32. Our Solution
• We created two ChalkMark surveys with identical
questions but different designs.
• We used SurveyMonkey’s random assignment feature
to randomly direct participants to one of the surveys.
• We compared responses to each question to see
what was different.
ChalkMark
Design #1
SurveyMonkey
Random Assignment
ChalkMark
Design #2

33. ChalkMark
Design #1
SurveyMonkey ChalkMark
Random Assignment Design #2
ChalkMark
Design #3

34. UserTesting.com
Current Website
SurveyMonkey
Random Assignment
UserTesting.com
Prototype

35. Lessons Learned
• Test significantly different designs
• Limits to chaining tools together
• Integration with panel management/
recruiting software
• Tracking participants for incentives
• Have a clear hypothesis you’re trying to
prove/disprove

36. #3. Boosting Interview
Responses
The client: regional government
The project: understand how citizens
access and experience government services
The problem: how do we get people to talk
about something abstract like services?

37. Our Solution
• Emotional response cards
• We used a set of 50 cards with
emotional adjectives on them to help
elicit in-depth responses from
participants.
• Used physical cards in 20 in-home
interviews, used PDF file for 20
telephone interviews.

39. How They Worked
• We started with Microsoft’s Product Reaction
Cards, which includes a list of 118 product
characteristics
• We reduced the number of cards to 50 and tried
to include opposing characteristics (similar to
BERT)
• At the end of the interview we handed
participants the cards and asked them to pick the
cards that described the experiences they had
just talked about
http://www.uxmag.com/articles/organized-approach-to-emotional-response-testing
http://www.uxforthemasses.com/bert/

41. The Results
• People remember emotions
• Few experiences are all +ve or -ve
• Props help people express themselves
• Emotions keep people honest
• Emotions lead to better stories

42. Conclusion
• These are some ways we’re pushing
our user research practice
• We’re able to get a lot of value from
simple tools and creative thinking
• Please share your own ideas at the
break, on Twitter (#uxhacks) or on your
blog

43. Thank you!
Gene Smith nForm
gene.smith@nform.com http://nform.com
@gsmith @nform