SLIDE IS NOT GOSPEL. (Checkmarks are a generalization.)Identify unknown needs = things the USER doesn’t realize they needNatural environment = USER (not necessarily researcher) is in their natural environmentReal motivation = the user needs or wants to do this task; not a task given by researcher.Minor problems apparent -- things like the user’s eyes looking around the screen for a couple seconds before finding a buttonThere IS overlap in these methods. And the data you get is not the only thing to consider; time, resources are also a big driver of which method is used.
If you skip this step, you could have a product that isn’t useful to people.
What to build? You could just ask people what their problems are. But people aren’t good at realizing some of their problems.
Example: We wanted to create a brand new product and didn’t know what kind of product we wanted to build. Interviewed small business owners about all aspects of their business. We asked business owners about their biggest pain points but decided to solve something they didn’t actually mention as a problem. We saw that they were using several different tools to manage their business and entering the same data into more than one application. We decided, what if wehandleall of those tasks in a single application?
Observation helps because people can’t describe in detail what they do. Observation helps because the researcher doesn’t always know the right questions to ask.In one study, I asked business owners lots of questions about their invoices: when do you send them, how do you handle overdue payments, etc. I even got copies of their invoices. And there were some aspects of bookkeeping that we watched users do. But I never watched a customer SEND an invoice. If I had watched enough customers, I would have seen that they were sometimes emailing an invoice to multiple people at their client’s company. When we designed the application, we assumed that invoices would need just one recipient. WRONG.
This person was NOT our customer. Do you think she would have showed us her business’s financial data if we had interviewed her on the phone instead of face to face?
Recruiting: Occasionally, I use an external recruiting company, but I still talk to the participant myself before visiting. Once when I called in advance, the woman who had been recruited as a “small business owner” told me she was not working. If I had not spoken with her personally, we would have visited her home and learned that we couldn’t complete the session with her. Saving work - so you can watch them work.Not cleaning up - seeing all the mess is part of understanding user behavior. Post it notes and cheat sheet may help you understand important pain points.
Example lets you verify that you understand what participant said. It also keeps them from generalizing, which can gloss over important details.
Sometimes, field visits can be put you in slightlyawkward situations, especially if meeting in people’s homes. Roll with the punches and always be professional regardless.
Remember, our users are business owners.
You make certain assumptions when you design. Users will always use products in ways that you didn’t expect. EXAMPLE: Participant’s invoice list: draft invoices with the SAME client name over and over. We didn’t account for users who do all of their work for the same client.Note that in a survey, she might have asked for a specific field to be added to invoice list, but we wouldn’t know why she needed that.
Call participant directly if needed
GoToMeeting - MOV for MacWebEx: Separate meetings = not really separate?WebEx: Recurring meetings must be at a same time daily, weeklyAdobe Connect: Users may want to call in with computer audio
Explain what an intercept test is. The version of this that involves a facilitator -- ad on the site asking for participants.Call the participant, ask them why they came to the site, ask them to continue and think out loud.
Even asking users to pretend can give you some good information, if they are target users. E.g., does the information make sense?Live recruit results: Some people just watched the video. Usability test in lab: More people were thorough. Less variation in the amount of time spent on site.
Story: went to the copier. Gone maybe 4 minutes. Participant was already done looking at the site.
*** You may get people filling out the form who don’t really want to participate in the study
Focus = questions must support survey goalsShort = 10 questions or less, depending on how hard it is to answer the questions
The first question relevant to the survey goals is #10.
You must focus the survey because ifusers get bored, they will:Get lazy and give you bad information ORStop the survey altogether
People WILL answer your questions, for better or worseN/A: If it doesn’t apply and you force a choice, your data will be skewedPilot - I have never piloted a survey and found it to be perfect.
Purpose of one of my diary studies was to learn what problems people were having while first starting to use the software. Had them answer a few basic questions in a Word document each time they encountered an issue. Emailed document to me at the end of each week x up to 8 weeks.
Sample from an accounting software diary study
Pilot - not just to test out the test questions and prototype, but also the technology. The one time I didn’t pilot a remote test, first participant couldn’t even access the product on our staging server.
Despite Dumas and Loring’s advice to respond to participants in a neutral way, I feel that I may start to sound robotic if I’m too strict about this. So, I error on the side of being genuine / human when responding to user suggestions and comments.Show that you are on the user’s side = validate what they say. Tell them that what they do or think makes sense, even if they are being self-deprecating. Validation is important for maintaining rapport and getting the user to talk freely.
Start lean: 1) takes less time, 2) helps you build credibility and get buy inGet stakeholders involved; if they watch any part of what you’re doing and see value, you may start to get their buy in. You do need some sense of what you are doing. But don’t be afraid that you’re going to ask a question the wrong way or not do everything perfectly. Read what you can, then TRY IT and learn from your mistakes.
UX for Lean Startups: Faster, Smarter User Experience Research and Design, by Laura Klein
Beyond usability testing: Getting a holistic view of the user experience
Beyond usability testing
Getting a holistic view of the user
Amanda Nance, User experience researcher
Atlanta Web Design Group
Aug. 29, 2013
In UX field since 2002
Studied human-centered communication & psychology, Mercer
M.S. in Human-computer interaction, Georgia Tech
Women’s barbershop quartet
Large, international corporation (14,000 employees)
Accounting products for small businesses
Desktop and web-based
Product Management team - 5
Developers - 16
Dev Operations - 5
QA - 6
Interaction & visual designers - 3
Writers - 2 part-time
Researcher - 1
31 studies since Jan 2010
Broaden your toolkit for including users
Convince you to start right away
Usability testing is great. Try it!
See “Rocket Surgery Made Easy” by Steve Krug
An ideal usability test would tell you
Identify unknown needs
User's own data
Major usability problems apparent
Minor usability problems apparent
Watch and interview users in their
Use field visits before designing
to learn what people need
Before designing can mean:
Before finalizing the design
After launch to improve product
Asking users about problems limits the
problems you’ll uncover
Used under CC license
+ The problem we solved was one users hadn’t
“I hadn’t even thought that far out of the box.”
-User reaction to our product concept
Observation gives you details about
what people need
Use field visits for learning about
Tips: Before the visit
Ask about pets
Talk to the participant yourself before the visit
Ask to save up some work
Even if using an external recruiter
Tell them not to clean up!
Ask for driving directions / verify Google Maps accuracy
Tips: During the visit
Say “Show me” and ask for specific examples
Tips: During the visit
Verify your understanding in the moment
Always respond professionally
Field visits: general guidelines
We pay $300 for 3 hours
Limit is 3 hours
Can divide into one-hour sessions amongst 3 users
Use informal interviews for (perceived)
Email and phone calls: easy and quick!
Use screen sharing interviews to learn
how the product is (really) used
We didn’t account for users who repeatedly
subcontract for the same company.
Screen sharing tips: recruiting
Find customers who are interested in the topic
Explain logistics: ear buds or speakerphone required
Get participant’s phone number
Questionnaire completed by users to learn about
demographics, opinions, etc.
Use when you need quantitative data
Keep it short
A goal-less survey
How did you first become aware of Awesome Accounting?
To the best of your knowledge, what does Awesome
To the best of your knowledge, what is the price of Awesome
What software were you using in the running of your business when you first considered Awesome Accounting?
What software were you looking to replace when you first considered Awesome Accounting, if any?
Why did you first consider Awesome Accounting?
Before subscribing to Awesome Accounting, did you do a free trial?
Are you currently using Awesome Accounting in running your business?
Which of the following parts of Awesome Accounting have you used, or do you intend to use, in running your business?
How satisfied are you with the following aspects of Awesome
Please use the spaces below to comment on each of the aspects of Awesome Accounting.
How likely are you to continue to be a Awesome Accounting subscriber 12 months from now?
Used with CC permission
Tip: Survey process
Ask questions that people can accurately answer
What did you eat for breakfast 4 days ago? vs.
Do you typically organize your emails into folders?
Include an “N/A” option
Avoid mandatory questions
Ask permission to contact for follow-up questions
Pilot test your survey
Users record activities, problems, or
feelings over time
Diary studies: when to use
Seeing experience over time
Researching and buying a car
Infrequent tasks / non-core tasks
To identify places to dig deeper with other studies
Diary studies will show you the
Diary studies: Tips
Reserve time to follow up on unclear entries
Interact with participants to maintain engagement
Continue participants who are useful
Ask for screenshots
Diary studies: general guidelines
At least $25/week
Can increase over time
Length: Can keep people engaged for several weeks if not
Recruit: At least 10
More depending on your goals
Start lean and build up
Ask forgiveness, not permission from stakeholders
Find users who will help for free
Get your stakeholders involved
Don’t worry about being a research expert
Quick read to get started
UX for Lean Startups: Faster, Smarter
User Experience Research and Design
by Laura Klein
Short. No UX background required.
+ General references on qualitative research
Understanding Your Users: A practical
guide to user requirements
Elizabeth Goodman, Mike Kuniavsky, Andrea
by Catherine Courage & Kathy Baxter
700+ pages. Each method = 1 chapter.
Observing the User Experience,
500+ pages. Each method = 1 chapter
On field visits
User and task analysis by Hackos and Redish
Contextual Design by Beyer and Holtzblatt
+ Live recruit testing and remote research
Remote Research: Real
Users, Real Time, Real
by Nate Bolt, Tony Tulathimutte
The general reference books I
mentioned don’t seem to cover
Look at www.remoteresear.ch for templates
Steve Portigal’s “Interviewing users”
Coming in 2014: “Surveys that work” by Caroline Jarrett
Caroline’s podcast with UIE on surveys
Let me know how it goes!