This document provides an overview of qualitative user research methods beyond traditional usability testing that can provide a holistic view of the user experience. These include field visits to observe users in their natural environment, informal interviews via phone or email with screen sharing, live recruiting tests to observe more natural user behavior, surveys to collect quantitative data, and diary studies where users record activities and issues over time. The document discusses when each method is appropriate to use and provides tips for implementation. It emphasizes starting research quickly with minimal resources and getting stakeholders involved.
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Getting a Holistic View of the User Experience
1. +
Beyond usability testing
Getting a holistic view of the user
experience
Amanda Nance, User experience researcher
Atlanta Web Design Group
Aug. 29, 2013
2. +
About me
In UX field since 2002
Studied human-centered communication & psychology, Mercer
University
M.S. in Human-computer interaction, Georgia Tech
Women’s barbershop quartet
@amandaux #beyondutest
3. +
Context
Large, international corporation (14,000 employees)
Accounting products for small businesses
Desktop and web-based
No analytics
My product
Product Management team - 5
Developers - 16
Dev Operations - 5
QA - 6
Interaction & visual designers - 3
Writers - 2 part-time
Researcher - 1
8. +
An ideal usability test would tell you
everything, but…
Ideal
In practice
Real users
yes!
Identify unknown needs
Natural environment
User's own data
Real motivation
debatable
Non-core tasks
Major usability problems apparent
yes!
Minor usability problems apparent
yes!
Quantitative information
sometimes
12. +
Use field visits before designing
to learn what people need
Before designing can mean:
Before anything
Before finalizing the design
After launch to improve product
13. +
Asking users about problems limits the
problems you’ll uncover
Used under CC license
Leah Gregg
14. + The problem we solved was one users hadn’t
thought about
“I hadn’t even thought that far out of the box.”
-User reaction to our product concept
17. +
Tips: Before the visit
Ask about pets
Talk to the participant yourself before the visit
Ask to save up some work
Even if using an external recruiter
Tell them not to clean up!
Ask for driving directions / verify Google Maps accuracy
23. +
Use screen sharing interviews to learn
how the product is (really) used
We didn’t account for users who repeatedly
subcontract for the same company.
24. +
Screen sharing tips: recruiting
Find customers who are interested in the topic
Analytics
Survey responses
Email
Explain logistics: ear buds or speakerphone required
Get participant’s phone number
26. +
Screen sharing tips: tools
Turn off the beeps when people join
Turn off chat for observers
Turn off list of meeting attendees
27. +
Screen sharing tips: during the session
“Can
you show me…”
the issues you’re having
how you _______
Help with issues at end
In case of emergency: let participant control your screen
30. + Live recruit test = watching more natural
behavior
CC license
http://www.flickr.com/photos/evilpe
acock/6423510787/
31. +
Use a live recruit test when motivation is
key
Pretend that you are looking
for bookkeeping software for your
company…
32. +
Live recruiting tips: Ethnio
Use Ethnio to recruit
Read all demographics
(unqualified people are listed)
Answers may reveal unclear
screener questions
Pause the recruiting if you leave
33. +
Live recruiting tips
Decide how to handle people who have already started using
the site
Have sales or relevant phone numbers handy
See www.remoteresear.ch for templates
37. +
A goal-less survey
1.
How did you first become aware of Awesome Accounting?
2.
To the best of your knowledge, what does Awesome
Accounting do?
3.
To the best of your knowledge, what is the price of Awesome
Accounting?
4.
What software were you using in the running of your business when you first considered Awesome Accounting?
5.
What software were you looking to replace when you first considered Awesome Accounting, if any?
6.
Why did you first consider Awesome Accounting?
7.
Before subscribing to Awesome Accounting, did you do a free trial?
8.
Are you currently using Awesome Accounting in running your business?
9.
Which of the following parts of Awesome Accounting have you used, or do you intend to use, in running your business?
10.
How satisfied are you with the following aspects of Awesome
Accounting?
11.
Please use the spaces below to comment on each of the aspects of Awesome Accounting.
12.
How likely are you to continue to be a Awesome Accounting subscriber 12 months from now?
38. Used with CC permission
http://www.flickr.com/photos/stovak/
39. +
Tip: Survey process
Ask questions that people can accurately answer
What did you eat for breakfast 4 days ago? vs.
Do you typically organize your emails into folders?
Include an “N/A” option
Avoid mandatory questions
Ask permission to contact for follow-up questions
Pilot test your survey
41. +
Users record activities, problems, or
feelings over time
CC license
http://www.flickr.com/photos/bitchcakes/5457865373/
sizes/o/in/photostream/
42. +
Diary studies: when to use
Seeing experience over time
Researching and buying a car
Infrequent tasks / non-core tasks
To identify places to dig deeper with other studies
44. +
Diary studies: Tips
Reserve time to follow up on unclear entries
Interact with participants to maintain engagement
Continue participants who are useful
Ask for screenshots
45. +
Diary studies: general guidelines
Compensation
At least $25/week
Can increase over time
Length: Can keep people engaged for several weeks if not
months
Recruit: At least 10
More depending on your goals
47. +
General research tips: planning and
logistics
Start with goals
Pilot
Consider recording with permission
Have someone else take notes if possible
48. +
General research tips: facilitating
Ask open-ended questions
Ask neutral questions
Probe for more information
Ask for an example
Be genuine
Show that you are on the user’s side
51. +
Getting started
Start lean and build up
Ask forgiveness, not permission from stakeholders
Find users who will help for free
Get your stakeholders involved
Don’t worry about being a research expert
53. +
Quick read to get started
UX for Lean Startups: Faster, Smarter
User Experience Research and Design
by Laura Klein
Short. No UX background required.
54. + General references on qualitative research
Understanding Your Users: A practical
guide to user requirements
Elizabeth Goodman, Mike Kuniavsky, Andrea
Moed
by Catherine Courage & Kathy Baxter
700+ pages. Each method = 1 chapter.
Observing the User Experience,
Second Edition
500+ pages. Each method = 1 chapter
55. +
On field visits
User and task analysis by Hackos and Redish
Contextual Design by Beyer and Holtzblatt
56. + Live recruit testing and remote research
Remote Research: Real
Users, Real Time, Real
Research
by Nate Bolt, Tony Tulathimutte
The general reference books I
mentioned don’t seem to cover
this topic.
Look at www.remoteresear.ch for templates
58. +
On surveys
Coming in 2014: “Surveys that work” by Caroline Jarrett
Caroline’s podcast with UIE on surveys
59. +
Let me know how it goes!
anance@gmail.com
@amandaux
Editor's Notes
Across all parts of the lifecycle
Slides will be available
SLIDE IS NOT GOSPEL. (Checkmarks are a generalization.)Identify unknown needs = things the USER doesn’t realize they needNatural environment = USER (not necessarily researcher) is in their natural environmentReal motivation = the user needs or wants to do this task; not a task given by researcher.Minor problems apparent -- things like the user’s eyes looking around the screen for a couple seconds before finding a buttonThere IS overlap in these methods. And the data you get is not the only thing to consider; time, resources are also a big driver of which method is used.
If you skip this step, you could have a product that isn’t useful to people.
What to build? You could just ask people what their problems are. But people aren’t good at realizing some of their problems.
Example: We wanted to create a brand new product and didn’t know what kind of product we wanted to build. Interviewed small business owners about all aspects of their business. We asked business owners about their biggest pain points but decided to solve something they didn’t actually mention as a problem. We saw that they were using several different tools to manage their business and entering the same data into more than one application. We decided, what if wehandleall of those tasks in a single application?
Observation helps because people can’t describe in detail what they do. Observation helps because the researcher doesn’t always know the right questions to ask.In one study, I asked business owners lots of questions about their invoices: when do you send them, how do you handle overdue payments, etc. I even got copies of their invoices. And there were some aspects of bookkeeping that we watched users do. But I never watched a customer SEND an invoice. If I had watched enough customers, I would have seen that they were sometimes emailing an invoice to multiple people at their client’s company. When we designed the application, we assumed that invoices would need just one recipient. WRONG.
This person was NOT our customer. Do you think she would have showed us her business’s financial data if we had interviewed her on the phone instead of face to face?
Recruiting: Occasionally, I use an external recruiting company, but I still talk to the participant myself before visiting. Once when I called in advance, the woman who had been recruited as a “small business owner” told me she was not working. If I had not spoken with her personally, we would have visited her home and learned that we couldn’t complete the session with her. Saving work - so you can watch them work.Not cleaning up - seeing all the mess is part of understanding user behavior. Post it notes and cheat sheet may help you understand important pain points.
Example lets you verify that you understand what participant said. It also keeps them from generalizing, which can gloss over important details.
Sometimes, field visits can be put you in slightlyawkward situations, especially if meeting in people’s homes. Roll with the punches and always be professional regardless.
Remember, our users are business owners.
You make certain assumptions when you design. Users will always use products in ways that you didn’t expect. EXAMPLE: Participant’s invoice list: draft invoices with the SAME client name over and over. We didn’t account for users who do all of their work for the same client.Note that in a survey, she might have asked for a specific field to be added to invoice list, but we wouldn’t know why she needed that.
Call participant directly if needed
GoToMeeting - MOV for MacWebEx: Separate meetings = not really separate?WebEx: Recurring meetings must be at a same time daily, weeklyAdobe Connect: Users may want to call in with computer audio
Explain what an intercept test is. The version of this that involves a facilitator -- ad on the site asking for participants.Call the participant, ask them why they came to the site, ask them to continue and think out loud.
Even asking users to pretend can give you some good information, if they are target users. E.g., does the information make sense?Live recruit results: Some people just watched the video. Usability test in lab: More people were thorough. Less variation in the amount of time spent on site.
Story: went to the copier. Gone maybe 4 minutes. Participant was already done looking at the site.
*** You may get people filling out the form who don’t really want to participate in the study
Focus = questions must support survey goalsShort = 10 questions or less, depending on how hard it is to answer the questions
The first question relevant to the survey goals is #10.
You must focus the survey because ifusers get bored, they will:Get lazy and give you bad information ORStop the survey altogether
People WILL answer your questions, for better or worseN/A: If it doesn’t apply and you force a choice, your data will be skewedPilot - I have never piloted a survey and found it to be perfect.
Purpose of one of my diary studies was to learn what problems people were having while first starting to use the software. Had them answer a few basic questions in a Word document each time they encountered an issue. Emailed document to me at the end of each week x up to 8 weeks.
Sample from an accounting software diary study
Pilot - not just to test out the test questions and prototype, but also the technology. The one time I didn’t pilot a remote test, first participant couldn’t even access the product on our staging server.
Despite Dumas and Loring’s advice to respond to participants in a neutral way, I feel that I may start to sound robotic if I’m too strict about this. So, I error on the side of being genuine / human when responding to user suggestions and comments.Show that you are on the user’s side = validate what they say. Tell them that what they do or think makes sense, even if they are being self-deprecating. Validation is important for maintaining rapport and getting the user to talk freely.
Start lean: 1) takes less time, 2) helps you build credibility and get buy inGet stakeholders involved; if they watch any part of what you’re doing and see value, you may start to get their buy in. You do need some sense of what you are doing. But don’t be afraid that you’re going to ask a question the wrong way or not do everything perfectly. Read what you can, then TRY IT and learn from your mistakes.
UX for Lean Startups: Faster, Smarter User Experience Research and Design, by Laura Klein