Planning and running usability tests


Published on

In October 2013, I gave the industry element of the lecture for the User Experience Design module at Manchester Metropolitan University.

To fit in with the overall programme, this covered thinking about objectives for an evaluation, goals, questions to be answered, choosing a method, arranging participants, and thinking about practicalities. It also touched on running a test, preparing students for future lectures covering specific methods.

Published in: Design, Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Given a probability of finding a usability problem in a given UI of 0.31 (their long term average), with 5 users you’ll find 85% of the usability problems
  • Planning and running usability tests

    1. 1. Usability testing Planning & running Chris Collingridge (@ccollingridge) 22 October 2013 Manchester Metropolitan University 1
    2. 2. 1.0 2.0 3.0 4.0 Who are we and who am I? What type of test? Fitting it in Planning and doing Manchester Metropolitan University 2
    3. 3. Sage and me Who we are and what we do Manchester Metropolitan University 3
    4. 4. Sage – Global • 6 million customers • 13,300 employees • Major offices in UK, Ireland, France, Germany, Spain, USA, Canada, Austrailia, & Brazil Manchester Metropolitan University – Small business accounting – Payroll – Customer relationship management (CRM) – Taxation and accountancy – Electronic payments 4
    5. 5. Sage – UK • Only software company in the FTSE 100 • 800,000 UK businesses use Sage • #1 in small business accounting • 1 in 4 people in the UK are paid by Sage Payroll Manchester Metropolitan University 5
    6. 6. Sage – Manchester • Software for accountants in practice • On-premise and online software • 300,000 sets of company accounts filed using Sage each year • 200,000 corporate tax submissions • 520,000 personal tax submissions Manchester Metropolitan University – Final accounts production – Corporate and personal taxation – Practice management – Time recording and billing – Accountant/client collaboration 6
    7. 7. Summary We’re big Manchester Metropolitan University 7
    8. 8. Me – Degree in economics (obviously!) – Worked in a shop – Decided there must be a career in computers …and mainly self-taught 15 years later… – Senior User Experience Specialist! Manchester Metropolitan University 8
    9. 9. Day-to-day • • • • • • User research – understanding the problem What do people know? What are they trying to do? Where do they do things? What do they value? What troubles them? • • • • Interaction design – solving the problem Information architecture User flows Patterns Low-level interaction (controls etc.) • • Usability testing – evaluating solutions Manchester Metropolitan University 9
    10. 10. Day-to-day • • • • • • User research – understanding the problem What do people know? What are they trying to do? Where do they do things? What do they value? What troubles them? • • • • Interaction design – solving the problem Information architecture User flows Patterns Low-level interaction (controls etc.) • • Usability testing – evaluating solutions Manchester Metropolitan University 10
    11. 11. What type of test? Manchester Metropolitan University 11
    12. 12. Formative vs. Summative • Summative • Formative “evaluation of a product with “a type of usability evaluation that helps to representative users and tasks designed "form" the design for a product or service. to measure the usability (defined as Formative evaluations involve evaluating effectiveness, efficiency and satisfaction) a product or service during of the complete product…the main development, often iteratively, with the purpose of a summative test is to evaluate goal of detecting and eliminating a product through defined usability problems.” measures, rather than diagnosis and correction of specific design problems” Ref: Usability Body of Knowledge Manchester Metropolitan University 12
    13. 13. Formative vs. Summative • Summative Analytics Customers surveying and feedback “Voice of customer” Usability test = large scale, expensive, scientific, resource intensive, low ROI Manchester Metropolitan University • Formative Most of our usability testing is formative Blended with research Often conceptual Usability test = small scale, iterative, pragmatic, high ROI 13
    14. 14. Fitting it in Manchester Metropolitan University 14
    15. 15. Overall process Plan Prepare Do Analyse Act This is the usability test bit Manchester Metropolitan University 15
    16. 16. Planning Manchester Metropolitan University 16
    17. 17. Fact* 96% of #### usability tests are a result of poor planning* * Entirely made up Manchester Metropolitan University 17
    18. 18. Failure • • • • • • • Unclear goals Test not addressing goals Method produces unusable results Too many variables Unrepresentative participants Inconsistent moderation Inconsistent/incomplete notes/recording = a whole load of rubbish Let’s plan! Manchester Metropolitan University 18
    19. 19. Planning Manchester Metropolitan University 19
    20. 20. My template for planning Objectives Goals Questions Method Participants Practicalities Manchester Metropolitan University 20
    21. 21. 1 – Objectives What will you be able to do as a consequence of this test? The point is always to stimulate valuable action Example: Redesign the delivery elements of the shopping cart Manchester Metropolitan University 21
    22. 22. 1 – Objectives Your objective is not a report Manchester Metropolitan University 22
    23. 23. 2 – Goals What will you know about or understand? Examples: • Know how people enter addresses for other people if they’re sending gifts • Understand how people react to different default delivery prices Manchester Metropolitan University 23
    24. 24. 3 – Questions What specific questions will you have answers to? Examples: • When asked to enter the address of a friend or sibling, where do they get the address from? • Do postcode lookups help people enter addresses other than their own? • Are people less willing to continue if higher priced delivery options are the default? Manchester Metropolitan University 24
    25. 25. 4 – Method How will you get answers to the questions? Hint: • Don’t think only about checking something • Compare one thing to another (but control your variables) • Evaluate the perceived value of something • Gain contextualise insight into something Manchester Metropolitan University 25
    26. 26. 4 – Method Some things to think about when defining your method (1) • Do you need to be able to see the user? • Do they need to be using their own equipment? • (PC vs. Apple, desktop vs. laptop, multi-screen, browser, etc.) • • • • How much time will they have? How much time do you have? What equipment/software will you be able to use? How much help might they need to access the software? Manchester Metropolitan University 26
    27. 27. 4 – Method Some things to think about when defining your method (2) • Where can you run it? • What do you need to be able to measure? • Time*, success rate, errors, comments? • How will you avoid order effects? • Are you able to access/build software to support the tasks? * Using “talk aloud” invalidates objective measures of time. But does it matter? Manchester Metropolitan University 27
    28. 28. 4 – Method Your requirements drive your method Onsite usability test In-person usability test Remote moderated test Or something else entirely… Manchester Metropolitan University Remote unmoderated test Survey Tip: The answer almost certainly is *not* a focus group 28
    29. 29. 5 – Participants Who will you get to participate in your test? Manchester Metropolitan University 29
    30. 30. 5 – Participants – How many? Manchester Metropolitan University 30
    31. 31. 5 – Participants – How many? 5 is the classic answer, and it may or may not be right for your situation. • Practical experience is that you do start to see a lot of repetition about here • Repeating “like” tasks gives you a lot more data • An explicit assumption is that you will iterate – this is not a 1-time activity never to be repeated Manchester Metropolitan University 31
    32. 32. 5 – Participants – Representativeness “Just go out to a coffee shop, buy someone a coffee, and sit down with them for 5 minutes” – Almost everyone on every blog on the internet Manchester Metropolitan University 32
    33. 33. 5 – Participants – Representativeness Manchester Metropolitan University 33
    34. 34. 5 – Participants – Representativeness Do people you will encounter in a coffee shop represent the important characteristics of your users? • If so, great! Head down to the coffee shop of your choice • If they are owner-managers of scrap metal merchants, consider whether they are likely to be hanging out in Starbucks Manchester Metropolitan University 34
    35. 35. 5 – Participants – Representativeness One of the single most important things you can do to generate valid insight is recruit people who represent your target users • What are the important characteristics of your users (for this test)? • Where can you find people like that? • When will people like that be available? • Will you need to incentivise them? • Are there any special ethics? (Children, etc) Manchester Metropolitan University 35
    36. 36. 6 – Practicalities What do you need to sort out to make it happen? (1) • A space somewhere • A time • Equipment and software • Script • Note recording sheets and/or recording software • Task sheets for people to follow • Participant availability (and overbooking) • Prototypes, live sites, login details… Manchester Metropolitan University 36
    37. 37. 6 – Practicalities What do you need to sort out to make it happen? (2) • Attendees to help you • Remember you are trying to stimulate valuable action. Who can help make action come about? (A: People who set priorities and choose what work to do) • Development Managers • Product Managers • Developers • Etc… Manchester Metropolitan University 37
    38. 38. Example 1. A plan 2. A “method” (prototype) Manchester Metropolitan University 38
    39. 39. Doing Manchester Metropolitan University 39
    40. 40. This morning, 9am – Remote, moderated test – Web-hosted prototype plus alpha build – Skype with video, + Evaer to record – Note recording sheets (formatted) – Participant sent task sheets and URLs in advance – Chris – pictured – comoderating & notetaking Manchester Metropolitan University 40
    41. 41. Running a test - Prepare Be ready Do run-throughs in advance to check everything works Make sure all co-moderators know what’s going to happen and what they need to do Think about if you need to do “resets” between participants if data or state can be persisted Allow yourself time to set things up Manchester Metropolitan University 41
    42. 42. Running a test - Introduce Usability tests can be stressful for the participant • It’s not them on test – it’s the product or site • Everything that goes “wrong” is the fault of the designers, and the most useful bits of the session • People just like them are particularly interesting – you want to know how easy they find it to use • Reprise what to expect, and how long the session will take • Get any permissions (e.g. informed consent, consent for recording etc…) Manchester Metropolitan University 42
    43. 43. Running a test - Prepare Recommendations • 1 moderator (talker), 1 note taker • Lets one person remain engaged and keep the conversation going • Should result in better notes • Notes vs. audio/video • Need to be clear on what makes good notes • Notes are faster to analyse than video, if they’re good • Video is good to go back to if the notes are lacking Manchester Metropolitan University 43
    44. 44. Running a test – Run! Talk aloud? • Probably – good insight • Completely invalidates any objective measures of efficiency (but does it matter?) • Might not be appropriate for tasks where “flow” is important Manchester Metropolitan University 44
    45. 45. Running a test – Dealing with questions Manchester Metropolitan University 45
    46. 46. Running a test – Questions DO NOT ANSWER QUESTIONS Manchester Metropolitan University 46
    47. 47. Running a test – Questions Some suggestion for avoidance • “What do you expect that to do?” • “What do you think you could do next?” • “If I wasn’t here, what would you do?” • “Try doing whatever you think might help” • If necessary, admit you’re being awkward and unhelpful, but you need to know what they’d do if you weren’t here • Let people struggle for a while, but rescue them before they’re suicidal Manchester Metropolitan University 47
    48. 48. Running a test – Finish Show your appreciation • You should be grateful • Be grateful • Be genuine Manchester Metropolitan University 48
    49. 49. Analysis and reporting Manchester Metropolitan University 49
    50. 50. Analysis and reporting For another day…but remember We are doing this to stimulate valuable action. NOT: Manchester Metropolitan University 50
    51. 51. Thanks Chris Collingridge @ccollingridge Manchester Metropolitan University 51