+
Beyond usability testing
Getting a holistic view of the user
experience

Amanda Nance, User experience researcher
Atlant...
+

About me


In UX field since 2002



Studied human-centered communication & psychology, Mercer
University



M.S. in...
+

Context


Large, international corporation (14,000 employees)



Accounting products for small businesses





Des...
+

Context


31 studies since Jan 2010



120+ users
+

About you
+

My goal


Broaden your toolkit for including users



Convince you to start right away
+

Usability testing is great. Try it!

See “Rocket Surgery Made Easy” by Steve Krug
+

An ideal usability test would tell you
everything, but…
Ideal

In practice

Real users

yes!

Identify unknown needs
Na...
+

So we have these other methods
+
Field visits
+

Watch and interview users in their
natural habit
+

Use field visits before designing
to learn what people need


Before designing can mean:


Before anything



Before...
+

Asking users about problems limits the
problems you’ll uncover

Used under CC license
Leah Gregg
+ The problem we solved was one users hadn’t
thought about
“I hadn’t even thought that far out of the box.”
-User reaction...
+

Observation gives you details about
what people need
+

Use field visits for learning about
sensitive topics
+

Tips: Before the visit


Ask about pets



Talk to the participant yourself before the visit



Ask to save up some...
+

Tips: During the visit


Say “Show me” and ask for specific examples
+

Tips: During the visit


Verify your understanding in the moment



Always respond professionally
+

Field visits: general guidelines


We pay $300 for 3 hours



Limit is 3 hours



Can divide into one-hour sessions ...
+
Informal interviews
Calls & emails, screen sharing
+

Use informal interviews for (perceived)
requirements


Email and phone calls: easy and quick!
+

Use screen sharing interviews to learn
how the product is (really) used

We didn’t account for users who repeatedly
sub...
+

Screen sharing tips: recruiting


Find customers who are interested in the topic


Analytics



Survey responses


...
+

Screen sharing tips: tools

Join.Me - recommended by colleague
+

Screen sharing tips: tools


Turn off the beeps when people join



Turn off chat for observers



Turn off list of ...
+

Screen sharing tips: during the session

 “Can

you show me…”



the issues you’re having



how you _______



Hel...
+

Screen sharing: general guidelines


We pay $75/hour



$25 for 20-30 min



Limit to 1 hour
+
Live recruiting test
AKA intercept test
+ Live recruit test = watching more natural
behavior

CC license
http://www.flickr.com/photos/evilpe
acock/6423510787/
+

Use a live recruit test when motivation is
key

Pretend that you are looking
for bookkeeping software for your
company…
+

Live recruiting tips: Ethnio


Use Ethnio to recruit



Read all demographics
(unqualified people are listed)



Ans...
+

Live recruiting tips


Decide how to handle people who have already started using
the site



Have sales or relevant ...
+
Surveys
+

Surveys


Questionnaire completed by users to learn about
demographics, opinions, etc.



Use when you need quantitat...
+

Tips
FOCUS

CC license
http://www.flickr.com/photos/44456430
@N04/6825380102/

Keep it short
+

A goal-less survey
1.

How did you first become aware of Awesome Accounting?

2.

To the best of your knowledge, what d...
Used with CC permission
http://www.flickr.com/photos/stovak/
+

Tip: Survey process


Ask questions that people can accurately answer


What did you eat for breakfast 4 days ago? vs...
+
Diary studies
+

Users record activities, problems, or
feelings over time

CC license
http://www.flickr.com/photos/bitchcakes/5457865373...
+

Diary studies: when to use


Seeing experience over time


Researching and buying a car



Infrequent tasks / non-co...
+

Diary studies will show you the
bigger issues
+

Diary studies: Tips


Reserve time to follow up on unclear entries



Interact with participants to maintain engageme...
+

Diary studies: general guidelines


Compensation


At least $25/week



Can increase over time



Length: Can keep ...
+
General research tips
+

General research tips: planning and
logistics


Start with goals



Pilot



Consider recording with permission



...
+

General research tips: facilitating


Ask open-ended questions



Ask neutral questions



Probe for more informatio...
+
How to get started
+

What is keeping you from getting
started?
+

Getting started


Start lean and build up



Ask forgiveness, not permission from stakeholders



Find users who wil...
+
More info
Books, links, and stuff
+

Quick read to get started



UX for Lean Startups: Faster, Smarter
User Experience Research and Design
by Laura Klein
...
+ General references on qualitative research



Understanding Your Users: A practical
guide to user requirements



Eliz...
+

On field visits


User and task analysis by Hackos and Redish



Contextual Design by Beyer and Holtzblatt
+ Live recruit testing and remote research


Remote Research: Real
Users, Real Time, Real
Research
by Nate Bolt, Tony Tul...
+

On interviewing


Steve Portigal’s “Interviewing users”
+

On surveys



Coming in 2014: “Surveys that work” by Caroline Jarrett



Caroline’s podcast with UIE on surveys
+

Let me know how it goes!


anance@gmail.com



@amandaux
Upcoming SlideShare
Loading in …5
×

Beyond usability testing: Getting a holistic view of the user experience

1,202 views

Published on

Talk given to Atlanta Web Design Group, August 2013

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,202
On SlideShare
0
From Embeds
0
Number of Embeds
26
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Across all parts of the lifecycle
  • Slides will be available
  • SLIDE IS NOT GOSPEL. (Checkmarks are a generalization.)Identify unknown needs = things the USER doesn’t realize they needNatural environment = USER (not necessarily researcher) is in their natural environmentReal motivation = the user needs or wants to do this task; not a task given by researcher.Minor problems apparent -- things like the user’s eyes looking around the screen for a couple seconds before finding a buttonThere IS overlap in these methods. And the data you get is not the only thing to consider; time, resources are also a big driver of which method is used.
  • If you skip this step, you could have a product that isn’t useful to people.
  • What to build? You could just ask people what their problems are. But people aren’t good at realizing some of their problems.
  • Example: We wanted to create a brand new product and didn’t know what kind of product we wanted to build. Interviewed small business owners about all aspects of their business. We asked business owners about their biggest pain points but decided to solve something they didn’t actually mention as a problem. We saw that they were using several different tools to manage their business and entering the same data into more than one application. We decided, what if wehandleall of those tasks in a single application?
  • Observation helps because people can’t describe in detail what they do. Observation helps because the researcher doesn’t always know the right questions to ask.In one study, I asked business owners lots of questions about their invoices: when do you send them, how do you handle overdue payments, etc. I even got copies of their invoices. And there were some aspects of bookkeeping that we watched users do. But I never watched a customer SEND an invoice. If I had watched enough customers, I would have seen that they were sometimes emailing an invoice to multiple people at their client’s company. When we designed the application, we assumed that invoices would need just one recipient. WRONG.
  • This person was NOT our customer. Do you think she would have showed us her business’s financial data if we had interviewed her on the phone instead of face to face?
  • Recruiting: Occasionally, I use an external recruiting company, but I still talk to the participant myself before visiting. Once when I called in advance, the woman who had been recruited as a “small business owner” told me she was not working. If I had not spoken with her personally, we would have visited her home and learned that we couldn’t complete the session with her. Saving work - so you can watch them work.Not cleaning up - seeing all the mess is part of understanding user behavior. Post it notes and cheat sheet may help you understand important pain points.
  • Example lets you verify that you understand what participant said. It also keeps them from generalizing, which can gloss over important details.
  • Sometimes, field visits can be put you in slightlyawkward situations, especially if meeting in people’s homes. Roll with the punches and always be professional regardless.
  • Remember, our users are business owners.
  • You make certain assumptions when you design. Users will always use products in ways that you didn’t expect. EXAMPLE: Participant’s invoice list: draft invoices with the SAME client name over and over. We didn’t account for users who do all of their work for the same client.Note that in a survey, she might have asked for a specific field to be added to invoice list, but we wouldn’t know why she needed that.
  • Call participant directly if needed
  • GoToMeeting - MOV for MacWebEx: Separate meetings = not really separate?WebEx: Recurring meetings must be at a same time daily, weeklyAdobe Connect: Users may want to call in with computer audio
  • Explain what an intercept test is. The version of this that involves a facilitator -- ad on the site asking for participants.Call the participant, ask them why they came to the site, ask them to continue and think out loud.
  • Even asking users to pretend can give you some good information, if they are target users. E.g., does the information make sense?Live recruit results: Some people just watched the video. Usability test in lab: More people were thorough. Less variation in the amount of time spent on site.
  • Story: went to the copier. Gone maybe 4 minutes. Participant was already done looking at the site.
  • *** You may get people filling out the form who don’t really want to participate in the study
  • Focus = questions must support survey goalsShort = 10 questions or less, depending on how hard it is to answer the questions
  • The first question relevant to the survey goals is #10.
  • You must focus the survey because ifusers get bored, they will:Get lazy and give you bad information ORStop the survey altogether
  • People WILL answer your questions, for better or worseN/A: If it doesn’t apply and you force a choice, your data will be skewedPilot - I have never piloted a survey and found it to be perfect.
  • Purpose of one of my diary studies was to learn what problems people were having while first starting to use the software. Had them answer a few basic questions in a Word document each time they encountered an issue. Emailed document to me at the end of each week x up to 8 weeks.
  • Sample from an accounting software diary study
  • Pilot - not just to test out the test questions and prototype, but also the technology. The one time I didn’t pilot a remote test, first participant couldn’t even access the product on our staging server.
  • Despite Dumas and Loring’s advice to respond to participants in a neutral way, I feel that I may start to sound robotic if I’m too strict about this. So, I error on the side of being genuine / human when responding to user suggestions and comments.Show that you are on the user’s side = validate what they say. Tell them that what they do or think makes sense, even if they are being self-deprecating. Validation is important for maintaining rapport and getting the user to talk freely.
  • Start lean: 1) takes less time, 2) helps you build credibility and get buy inGet stakeholders involved; if they watch any part of what you’re doing and see value, you may start to get their buy in. You do need some sense of what you are doing. But don’t be afraid that you’re going to ask a question the wrong way or not do everything perfectly. Read what you can, then TRY IT and learn from your mistakes.
  • UX for Lean Startups: Faster, Smarter User Experience Research and Design, by Laura Klein
  • Beyond usability testing: Getting a holistic view of the user experience

    1. 1. + Beyond usability testing Getting a holistic view of the user experience Amanda Nance, User experience researcher Atlanta Web Design Group Aug. 29, 2013
    2. 2. + About me  In UX field since 2002  Studied human-centered communication & psychology, Mercer University  M.S. in Human-computer interaction, Georgia Tech  Women’s barbershop quartet @amandaux #beyondutest
    3. 3. + Context  Large, international corporation (14,000 employees)  Accounting products for small businesses    Desktop and web-based No analytics My product        Product Management team - 5 Developers - 16 Dev Operations - 5 QA - 6 Interaction & visual designers - 3 Writers - 2 part-time Researcher - 1
    4. 4. + Context  31 studies since Jan 2010  120+ users
    5. 5. + About you
    6. 6. + My goal  Broaden your toolkit for including users  Convince you to start right away
    7. 7. + Usability testing is great. Try it! See “Rocket Surgery Made Easy” by Steve Krug
    8. 8. + An ideal usability test would tell you everything, but… Ideal In practice Real users yes! Identify unknown needs Natural environment User's own data Real motivation debatable Non-core tasks Major usability problems apparent yes! Minor usability problems apparent yes! Quantitative information sometimes
    9. 9. + So we have these other methods
    10. 10. + Field visits
    11. 11. + Watch and interview users in their natural habit
    12. 12. + Use field visits before designing to learn what people need  Before designing can mean:  Before anything  Before finalizing the design  After launch to improve product
    13. 13. + Asking users about problems limits the problems you’ll uncover Used under CC license Leah Gregg
    14. 14. + The problem we solved was one users hadn’t thought about “I hadn’t even thought that far out of the box.” -User reaction to our product concept
    15. 15. + Observation gives you details about what people need
    16. 16. + Use field visits for learning about sensitive topics
    17. 17. + Tips: Before the visit  Ask about pets  Talk to the participant yourself before the visit   Ask to save up some work   Even if using an external recruiter Tell them not to clean up! Ask for driving directions / verify Google Maps accuracy
    18. 18. + Tips: During the visit  Say “Show me” and ask for specific examples
    19. 19. + Tips: During the visit  Verify your understanding in the moment  Always respond professionally
    20. 20. + Field visits: general guidelines  We pay $300 for 3 hours  Limit is 3 hours  Can divide into one-hour sessions amongst 3 users
    21. 21. + Informal interviews Calls & emails, screen sharing
    22. 22. + Use informal interviews for (perceived) requirements  Email and phone calls: easy and quick!
    23. 23. + Use screen sharing interviews to learn how the product is (really) used We didn’t account for users who repeatedly subcontract for the same company.
    24. 24. + Screen sharing tips: recruiting  Find customers who are interested in the topic  Analytics  Survey responses  Email  Explain logistics: ear buds or speakerphone required  Get participant’s phone number
    25. 25. + Screen sharing tips: tools Join.Me - recommended by colleague
    26. 26. + Screen sharing tips: tools  Turn off the beeps when people join  Turn off chat for observers  Turn off list of meeting attendees
    27. 27. + Screen sharing tips: during the session  “Can you show me…”  the issues you’re having  how you _______  Help with issues at end  In case of emergency: let participant control your screen
    28. 28. + Screen sharing: general guidelines  We pay $75/hour  $25 for 20-30 min  Limit to 1 hour
    29. 29. + Live recruiting test AKA intercept test
    30. 30. + Live recruit test = watching more natural behavior CC license http://www.flickr.com/photos/evilpe acock/6423510787/
    31. 31. + Use a live recruit test when motivation is key Pretend that you are looking for bookkeeping software for your company…
    32. 32. + Live recruiting tips: Ethnio  Use Ethnio to recruit  Read all demographics (unqualified people are listed)  Answers may reveal unclear screener questions  Pause the recruiting if you leave
    33. 33. + Live recruiting tips  Decide how to handle people who have already started using the site  Have sales or relevant phone numbers handy  See www.remoteresear.ch for templates
    34. 34. + Surveys
    35. 35. + Surveys  Questionnaire completed by users to learn about demographics, opinions, etc.  Use when you need quantitative data
    36. 36. + Tips FOCUS CC license http://www.flickr.com/photos/44456430 @N04/6825380102/ Keep it short
    37. 37. + A goal-less survey 1. How did you first become aware of Awesome Accounting? 2. To the best of your knowledge, what does Awesome Accounting do? 3. To the best of your knowledge, what is the price of Awesome Accounting? 4. What software were you using in the running of your business when you first considered Awesome Accounting? 5. What software were you looking to replace when you first considered Awesome Accounting, if any? 6. Why did you first consider Awesome Accounting? 7. Before subscribing to Awesome Accounting, did you do a free trial? 8. Are you currently using Awesome Accounting in running your business? 9. Which of the following parts of Awesome Accounting have you used, or do you intend to use, in running your business? 10. How satisfied are you with the following aspects of Awesome Accounting? 11. Please use the spaces below to comment on each of the aspects of Awesome Accounting. 12. How likely are you to continue to be a Awesome Accounting subscriber 12 months from now?
    38. 38. Used with CC permission http://www.flickr.com/photos/stovak/
    39. 39. + Tip: Survey process  Ask questions that people can accurately answer  What did you eat for breakfast 4 days ago? vs.  Do you typically organize your emails into folders?  Include an “N/A” option  Avoid mandatory questions  Ask permission to contact for follow-up questions  Pilot test your survey
    40. 40. + Diary studies
    41. 41. + Users record activities, problems, or feelings over time CC license http://www.flickr.com/photos/bitchcakes/5457865373/ sizes/o/in/photostream/
    42. 42. + Diary studies: when to use  Seeing experience over time  Researching and buying a car  Infrequent tasks / non-core tasks  To identify places to dig deeper with other studies
    43. 43. + Diary studies will show you the bigger issues
    44. 44. + Diary studies: Tips  Reserve time to follow up on unclear entries  Interact with participants to maintain engagement  Continue participants who are useful  Ask for screenshots
    45. 45. + Diary studies: general guidelines  Compensation  At least $25/week  Can increase over time  Length: Can keep people engaged for several weeks if not months  Recruit: At least 10  More depending on your goals
    46. 46. + General research tips
    47. 47. + General research tips: planning and logistics  Start with goals  Pilot  Consider recording with permission  Have someone else take notes if possible
    48. 48. + General research tips: facilitating  Ask open-ended questions  Ask neutral questions  Probe for more information  Ask for an example  Be genuine  Show that you are on the user’s side
    49. 49. + How to get started
    50. 50. + What is keeping you from getting started?
    51. 51. + Getting started  Start lean and build up  Ask forgiveness, not permission from stakeholders  Find users who will help for free  Get your stakeholders involved  Don’t worry about being a research expert
    52. 52. + More info Books, links, and stuff
    53. 53. + Quick read to get started  UX for Lean Startups: Faster, Smarter User Experience Research and Design by Laura Klein  Short. No UX background required.
    54. 54. + General references on qualitative research  Understanding Your Users: A practical guide to user requirements  Elizabeth Goodman, Mike Kuniavsky, Andrea Moed by Catherine Courage & Kathy Baxter  700+ pages. Each method = 1 chapter. Observing the User Experience, Second Edition  500+ pages. Each method = 1 chapter
    55. 55. + On field visits  User and task analysis by Hackos and Redish  Contextual Design by Beyer and Holtzblatt
    56. 56. + Live recruit testing and remote research  Remote Research: Real Users, Real Time, Real Research by Nate Bolt, Tony Tulathimutte  The general reference books I mentioned don’t seem to cover this topic. Look at www.remoteresear.ch for templates
    57. 57. + On interviewing  Steve Portigal’s “Interviewing users”
    58. 58. + On surveys  Coming in 2014: “Surveys that work” by Caroline Jarrett  Caroline’s podcast with UIE on surveys
    59. 59. + Let me know how it goes!  anance@gmail.com  @amandaux

    ×