#UXDclass
Why you should
always test with users
@quiffboy @amazeltd #UXDclass
Tuesday 10th Feb 2015
‘guerrilla’ is
and why...
#UXDclass
Leisa is Head of User Research @ GDS, GOV.UK
#UXDclass
“…testing with customers…”Me
Customers have a monetary value
#UXDclass
#UXDclass
@quiffboy
#UXDclass
#UXDclass
#UXDclass
#UXDclass
#UXDclass
nuxuk.org @nuxuk
#UXDclass
Common types of
usability testing
• Moderated lab testing
• Remote testing
• Eye tracking
• Guerrilla testing
#UXDclass
Moderated lab testing
“Observe people using the product to discover…how well test
subjects respond in four areas...
#UXDclass
#UXDclass
Remote usability testing
#UXDclass
Eye tracking
“Eye tracking is the process of measuring either the point of gaze
(where one is looking) or the mo...
#UXDclass
Eye tracking: complex & intrusive hardware, right?
fffff.at
acuity-ets.com
Not so scary anymore
#UXDclass
Eye tracking: findings are still scary though
• Generates lots of complex graphs and diagrams
• Needs experts to...
#UXDclass
But it’s a no brainer, right?
You should test your stuff with real people
#UXDclass
• http://econsultancy.com/uk/reports/us
er-experience-survey-report
• http://nuxuk.org/2013/09/05/ux-
challenge-...
#UXDclass
Why test?
#UXDclass
Why test?
Improve
decision
making
Remove
guesswork
Remove
experience
bias
#UXDclass
So why don’t we always test?
#UXDclass
So why don’t we always test?
Too
expensive
Time or
resource
consuming
Not a primary
concern
#UXDclass
£15 for 15 minutes
• Based on moderated lab testing
• Uses think-aloud protocols
• Test in guerrilla environment...
#UXDclass
http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
5 = 85%
#UXDclass
Case Study
ASICS America
#UXDclassCase study: ASICS America
#UXDclass
#UXDclass
+
+
Axure RP Pro Silverback
(optional)
#UXDclass
#UXDclass
What we did
Prep
• Recruit from same office building – posters in lobby
• Identified 2-3 key user journeys & fea...
#UXDclass
What we did
Prep • 3 days
Test • 1 day
Report • 1 day
#UXDclass
What we did
Prep • 3 days
Test • 1 day
Report • 1 day
1 week
start to finish
Easily
digestible
report
Actionable...
#UXDclass
• Make your report a one-pager
• Bullet-point list not reams of copy
• Make findings actionable
• Highlight when...
#UXDclass
Success: All participants were able to create a new
to-do list item.
Observation: A number of participants comme...
#UXDclass
One day to write
your test report
#UXDclass
10 Considerations
and/or gotchas
#UXDclass
1) Avoid recruiting experts
#UXDclass
2) Test small,
test early,
test often
#UXDclass
3) If you can, test in
real environments
#UXDclass
4) Focus on what needs
fixing (or refining)
#UXDclass
5) Give tasks,
not instructions
#UXDclass
6) Rotate order of tasks
#UXDclass
7) Your prototype’s
fidelity matters,
but it’s not critical
#UXDclass
8) Five people’s opinions
are not conclusive proof
#UXDclass
9) People say the
darndest things
#UXDclass
10) “Testing with 1 person
is 100% better than
testing with none”
Steve Krug
“Rocket Surgery Made Easy”
#UXDclass
Useful links
• Why You Only Need to Test with 5 Users
http://www.nngroup.com/articles/why-you-only-need-to-test-...
#UXDclass
Thank you 
t: @quiffboy
t: @amazeltd
w: www.amaze.com
Upcoming SlideShare
Loading in …5
×

Why you should always test with users...

11,994 views

Published on

...and why ‘guerrilla’ is better than nothing

Published in: Design, Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
11,994
On SlideShare
0
From Embeds
0
Number of Embeds
9,144
Actions
Shares
0
Downloads
20
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • Pet hate - “User testing”

    GDS = Government Digital Service in the Cabinet Office
  • I sometimes like to reframe it as Testing with Customers (not “users”)
    Makes it commercial
    Customers = revenue
  • 3 of my favourite things….
  • Senior UX Consultant at amaze
    Used to be a front end dev
    Drifted into UX in 2005
    Been doing this for 10 years
    Joined amaze in April 2012

    Amaze founded in 1995
    Lots and lots of people (about 220!)
    Offices in Manchester, Liverpool & London
    This is the Manchester office opposite the old Hacienda
    Full service consultancy for B2C and B2B: Strategy & creative, Design & build, Digital commerce and technology, Integrated comms, Global implementation support
    We’re quite good…
  • One of the NUX organisers
    Free UX meet-ups and big all-day conferences in Manchester, Leeds and Newcastle
  • Most common form of usability testing
    1 hour sessions
    Lots of findings

    Lots of dedicated consultancies for this
    Also many digital agencies do it (we do it)
    Number of one hour session, with full written reports
    Audio and video capture, edited highlights etc etc
    Often clients watch from behind one way mirror
    Worked with BBC for Cbeebies
    William Hill anecdote of screaming at screen – needs to be soundproofed
  • This sort of thing

    Pros:
    Users are often in a more informal environment (comfortable and relaxed)
    Great for bringing in teams together to observe users (and discuss)
    Help and support when defining tasks and scenarios
    Less likely to introduce moderator bias
    Detailed external provider analysis and reports
    More likely to use advanced data collection (Eye tracking)

    Cons:
    Can be costly
    Less likely to want to invest in early stage testing (Prototypes and early beta code)
    Less likely to perform iterative tests
  • Browser toolbar & plugin gives users tasks and captures screen & video/audio
    Consultancies have a pool of people they use
    Good for quantitative testing – large generate enormous amount of data and feedback from participants
    Answering specific questions about how people use a user interface for relatively simple and straightforward tasks
  • “sciency”
    Clients love it
  • Perception vs reality
  • Despite the complicated stuff, we know it should be done
    But how can we make it manageable?
  • What Users Do & Econsultancy
    Report into attitudes and approaches to UX
  • One area was why don’t you test?

    More reasons:
    Findings not actionable
    Will create more work
    “It can’t be that bad”
    “You should already know”
  • One area was why don’t you test?

    More reasons:
    Findings not actionable
    Will create more work
    “It can’t be that bad”
    “You should already know”
  • Our answer to this is 15 15

    Environments – has ecological validity, behaviours that occur in natural settings, not lab environment

    We managed to achieve some of these, but not all
  • Sweetspot
    5 users = 85% of issues / Multiple iterations

    Saturation at about 10 users

    Exceptions:
    Quantitative studies (aiming at statistics, not insights): Test at least 20 users to get statistically significant numbers; tight confidence intervals require even more users
    Card sorting: Test at least 15 users
    Eyetracking: Test 39 users if you want stable heatmaps

    Any users are better than none
  • The solution
  • Take ASICS from non-transactional to transactional
    Requires whole sales, warehouse and distribution network, and customer services establishing
    America - live
    Japan - live
    Europe - soon
    Australia
    Then on to China, South America and rest of world
  • Put these up in the lifts and the lobby
  • Use these
    Only need Vimeo if you’re sharing the videos
    Only need the Pro version of Vimeo if you want to password-protect them
  • We did some stuff – it looked like this
  • Identified 2-3 key user journeys & features
    Prototype new features for journeys – medium fidelity
    Recruit from same office building – posters in lobby

    Test in spare meeting room in Amaze office
    Slots 12 - 2pm & 4 - 6pm

    Brief, focussed report - bullet point list will suffice
    Export & upload videos & prototype

  • Identified 2-3 key user journeys & features
    Prototype new features for journeys – medium fidelity
    Recruit from same office building – posters in lobby

    Test in spare meeting room in Amaze office
    Slots 12 - 2pm & 4 - 6pm

    Brief, focussed report - bullet point list will suffice
    Export & upload videos & prototype

  • Identified 2-3 key user journeys & features
    Prototype new features for journeys – medium fidelity
    Recruit from same office building – posters in lobby

    Test in spare meeting room in Amaze office
    Slots 12 - 2pm & 4 - 6pm

    Brief, focussed report - bullet point list will suffice
    Export & upload videos & prototype

  • Eg: IT and design bods
    Unless they really are your audience
  • Eg: IT and design bods
    Unless they really are your audience
  • 15 minutes isn’t long - 2 mins will be intro - People like to talk
    Focus on 2 or 3 key areas for iteration
    You will have more iterations to test other stuff

    Small guerrilla throughout the project to keep it on track, but then a big lab test at the end to rubber stamp it for key stakeholders
  • 15 minutes isn’t long - 2 mins will be intro - People like to talk
    Focus on 2 or 3 key areas for iteration
    You will have more iterations to test other stuff

    Small guerrilla throughout the project to keep it on track, but then a big lab test at the end to rubber stamp it for key stakeholders
  • Basic principle of testing

    “You need a new pair of running shoes, what would you do?”
    not
    “Click on ‘running’ and find a pair of green size 8s suitable for road or trail”
  • ABC
    BAC
    CAB
    Etc

    Eg: find returns info – could have seen during “buy a shoe” task

    Order effects

  • Important thing is prototype should look realistic enough to be usable
    Don’t test wireframes
    Having said that, you can “explain away” inconsistencies or quirks during the session
  • You’re looking for something you can make a decision based on
    Not statistically valid rigorous data
    Consider remote testing if you need to thrown numbers at something
  • Never cease to be amazed by what people say
    Have open mind
    Be flexible
    Its fun, amusing, and exhausting
  • Why you should always test with users...

    1. 1. #UXDclass Why you should always test with users @quiffboy @amazeltd #UXDclass Tuesday 10th Feb 2015 ‘guerrilla’ is and why gorillas are better than nothing
    2. 2. #UXDclass Leisa is Head of User Research @ GDS, GOV.UK
    3. 3. #UXDclass “…testing with customers…”Me Customers have a monetary value
    4. 4. #UXDclass
    5. 5. #UXDclass @quiffboy
    6. 6. #UXDclass
    7. 7. #UXDclass
    8. 8. #UXDclass
    9. 9. #UXDclass
    10. 10. #UXDclass nuxuk.org @nuxuk
    11. 11. #UXDclass Common types of usability testing • Moderated lab testing • Remote testing • Eye tracking • Guerrilla testing
    12. 12. #UXDclass Moderated lab testing “Observe people using the product to discover…how well test subjects respond in four areas: efficiency, accuracy, recall, and emotional response.” Wikipedia cabinetoffice.gov.uk
    13. 13. #UXDclass
    14. 14. #UXDclass Remote usability testing
    15. 15. #UXDclass Eye tracking “Eye tracking is the process of measuring either the point of gaze (where one is looking) or the motion of an eye relative to the head.” Wikipedia grokdotcom.com
    16. 16. #UXDclass Eye tracking: complex & intrusive hardware, right? fffff.at acuity-ets.com Not so scary anymore
    17. 17. #UXDclass Eye tracking: findings are still scary though • Generates lots of complex graphs and diagrams • Needs experts to interpret results • Time consuming & expensive Heatmap from kunocreative.com Gaze plot from uxmatters.com
    18. 18. #UXDclass But it’s a no brainer, right? You should test your stuff with real people
    19. 19. #UXDclass • http://econsultancy.com/uk/reports/us er-experience-survey-report • http://nuxuk.org/2013/09/05/ux- challenge-choose-right-approach- guest-blog-ian-franklin-freelance-ux- consultant/ User Experience Survey 2013
    20. 20. #UXDclass Why test?
    21. 21. #UXDclass Why test? Improve decision making Remove guesswork Remove experience bias
    22. 22. #UXDclass So why don’t we always test?
    23. 23. #UXDclass So why don’t we always test? Too expensive Time or resource consuming Not a primary concern
    24. 24. #UXDclass £15 for 15 minutes • Based on moderated lab testing • Uses think-aloud protocols • Test in guerrilla environments • Recruit “in the wild” • Prototype “good enough” • Focus on key tasks • Report on problem areas • Client expectations are focussed
    25. 25. #UXDclass http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ 5 = 85%
    26. 26. #UXDclass Case Study ASICS America
    27. 27. #UXDclassCase study: ASICS America
    28. 28. #UXDclass
    29. 29. #UXDclass + + Axure RP Pro Silverback (optional)
    30. 30. #UXDclass
    31. 31. #UXDclass What we did Prep • Recruit from same office building – posters in lobby • Identified 2-3 key user journeys & features • Prototype new features for journeys – medium fidelity Test • Test in spare meeting room in Amaze office • Slots 12-2pm and 4-6pm Report • Brief, focussed report - bullet point list • Export & upload videos & prototype • Talk client through findings
    32. 32. #UXDclass What we did Prep • 3 days Test • 1 day Report • 1 day
    33. 33. #UXDclass What we did Prep • 3 days Test • 1 day Report • 1 day 1 week start to finish Easily digestible report Actionable findings
    34. 34. #UXDclass • Make your report a one-pager • Bullet-point list not reams of copy • Make findings actionable • Highlight when a test passes • Focus on the things you can fix The report
    35. 35. #UXDclass Success: All participants were able to create a new to-do list item. Observation: A number of participants commented that they expected the newly created to-do list item to automatically appear on all their devices. Issue: 3 participants did not understand the flag icon ⚐ was for setting a reminder for their to-do list item. Consider adding a label or using a different icon.
    36. 36. #UXDclass One day to write your test report
    37. 37. #UXDclass 10 Considerations and/or gotchas
    38. 38. #UXDclass 1) Avoid recruiting experts
    39. 39. #UXDclass 2) Test small, test early, test often
    40. 40. #UXDclass 3) If you can, test in real environments
    41. 41. #UXDclass 4) Focus on what needs fixing (or refining)
    42. 42. #UXDclass 5) Give tasks, not instructions
    43. 43. #UXDclass 6) Rotate order of tasks
    44. 44. #UXDclass 7) Your prototype’s fidelity matters, but it’s not critical
    45. 45. #UXDclass 8) Five people’s opinions are not conclusive proof
    46. 46. #UXDclass 9) People say the darndest things
    47. 47. #UXDclass 10) “Testing with 1 person is 100% better than testing with none” Steve Krug “Rocket Surgery Made Easy”
    48. 48. #UXDclass Useful links • Why You Only Need to Test with 5 Users http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ • The Art of Guerrilla Usability Testing http://www.uxbooth.com/articles/the-art-of-guerilla-usability-testing/ • Remote Usability and UX Research Tools http://remoteresear.ch/tools/ • New to Usability Testing? You’re not alone! http://userexperiencerocks.wordpress.com/2013/07/16/getting-started-with-usability- testing/ • User Experience Survey 2013 http://econsultancy.com/uk/reports/user-experience-survey-report http://nuxuk.org/2013/09/05/ux-challenge-choose-right-approach-guest-blog-ian- franklin-freelance-ux-consultant/ • Steve Krug: “Don’t make me think” & “Rocket Surgery Made Easy” and more… http://www.sensible.com
    49. 49. #UXDclass Thank you  t: @quiffboy t: @amazeltd w: www.amaze.com

    ×