Remote Usability Testing


Published on

It is important to learn to be flexible in your work and to learn tools that will empower usability analysts to be more effective in their roles. Also, in an age of diminishing corporate spending and rapid prototyping and testing, remote usability can save companies a lot of time and money.

Published in: Technology, Design
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Here are the top 5 things you all told me and my team to focus on last year when we first met at Insights: Starting with #5…you told us to get more aggressive in the market ….you felt we needed better lead generation, more competitive pricing, and better brand awareness for Accpac in the US 4 th , you told us to make it easier to do business with Sage…and in particular, repatriate our old operating model for CRM so that you could purchase it from just ONE part of the Sage organization. The 3 rd thing you told me was that despite the tremendous capabilities of Accpac, we had become overcomplicated …our Pricing was too complicated, we were nickling and diming, we were spending a lot of time on things that were of low value, and not enough time on things that really mattered to you The 2 nd priority you gave us was to synchronize our releases . We had been on a 24 month release cycle for our core product but many of our options and E2E products were not available until several months after the core ERP..causing you and your clients to delay in upgrading. And, FIRST on your list was…we want a VOICE…is anyone actively listening to what we are saying and prioritizing our ideas for future development cycles. I remember one of my very first conversations at last year’s Insights was with Pat Wade and her team…and they very clearly told me that while our technology vision was great, that we needed to spend an equal amount of time fixing known pain points for existing customers. So let’s talk now about how we have responded to all of this with our latest release!
  • Synchronous One on one and real time Shared technology Asynchronous No moderator Self-paced Special tools record activities & feedback
  • Same number of issues as face-to-face testing.
  • Remote Usability Testing

    1. 1. Remote Usability Testing December 5, 2009 Elizabeth Snowdon Usability Analyst, Sage Software [email_address]
    2. 2. About me Senior Usability Analyst at Sage Software Previously worked in usability at high-tech company Conducting usability studies since 2003
    3. 3. Principles of Scientific Management
    4. 4. Traditional Lab Testing
    5. 5. <ul><li>Remote usability </li></ul><ul><li>Advantages / Disadvantages </li></ul><ul><li>Case study </li></ul><ul><li>Tools </li></ul>What I will talk about today
    6. 6. What is Remote Testing? <ul><li>Testing sessions in which you and participant are not physically in the same place but are communicating via electronic technology </li></ul>
    7. 7. Types of Remote Testing <ul><li>One on one and real time </li></ul><ul><li>Shared technology </li></ul><ul><li>No moderator </li></ul><ul><li>Self-paced </li></ul><ul><li>Special tools record activities & feedback </li></ul>
    8. 8. Current Usability Testing Trends <ul><li>Movement away from lab testing </li></ul><ul><li>Increased interest in testing in users’ environments </li></ul><ul><li>Agile development methods </li></ul>
    9. 9. Advantages <ul><li>Lab not required </li></ul>University of Leeds Human Perception and Performance: Usability Laboratory
    10. 10. Advantages <ul><li>Expands sampling population </li></ul><ul><li>Ease of recruitment </li></ul><ul><li>Users in their own natural browsing environment </li></ul><ul><li>Share their desktop with you </li></ul>
    11. 11. Advantages <ul><li>Save money </li></ul><ul><ul><li>Travel </li></ul></ul><ul><ul><li>Equipment </li></ul></ul><ul><ul><li>Recruiting costs </li></ul></ul><ul><ul><li>Compensation </li></ul></ul>
    12. 12. Disadvantages <ul><li>Difficult to see participant’s screen </li></ul><ul><li>Cannot see participants </li></ul><ul><li>Problems downloading or installing desktop sharing </li></ul><ul><li>Task scenarios disseminated in advance </li></ul><ul><li>Participants interrupted or distracted </li></ul>
    13. 13. When not to use <ul><li>When security is important </li></ul><ul><li>When reliable high speed internet connection is unavailable </li></ul><ul><li>For eye tracking studies </li></ul><ul><li>When it is important to attend to user’s physical movement </li></ul>
    14. 14. When to use <ul><li>Recruiting valuable individuals </li></ul><ul><li>Minimal set-up </li></ul><ul><li>For lone wolf consultants </li></ul><ul><li>Rapid iterative development (Agile) </li></ul>
    15. 15. Recruiting Users
    16. 16. Case Study <ul><li>Baseline study for Accpac </li></ul><ul><li>Three personas </li></ul><ul><ul><li>clerks, controllers, operations </li></ul></ul><ul><li>24 participants </li></ul><ul><ul><li>20 remote </li></ul></ul><ul><ul><ul><li>US, South Africa, Australia, UK, Canada </li></ul></ul></ul><ul><li>One moderator, one note taker, observers </li></ul>
    17. 17. Results <ul><li>Accpac study completed in 4 weeks </li></ul><ul><li>Cost savings </li></ul><ul><ul><li>Travel </li></ul></ul><ul><ul><li>Lab set-up </li></ul></ul><ul><li>Time savings </li></ul><ul><ul><li>Recruiting </li></ul></ul><ul><ul><li>Scheduling </li></ul></ul><ul><ul><li>Less overtime </li></ul></ul><ul><li>Bottom line - $50,000 saved </li></ul>
    18. 18. Recruiting <ul><li>Internal recruitment, customer lists, Craigslist, email </li></ul><ul><li>Interested parties must pass screener </li></ul>
    19. 19. Other issues to sort out <ul><li>Obtaining informed consent and confidentiality agreement </li></ul><ul><li>What computing and communications equipment do candidates have? </li></ul><ul><li>Are they willing and able to download and run collaboration software? </li></ul>
    20. 20. Typical Set-up <ul><li>Moderator / Note-taker </li></ul><ul><ul><li>Screen sharing: WebEx </li></ul></ul><ul><ul><li>Recording: Morae </li></ul></ul><ul><ul><li>Speakerphone </li></ul></ul><ul><li>Participant </li></ul><ul><ul><li>High speed internet access </li></ul></ul><ul><ul><li>Speakerphone or headset telephone </li></ul></ul>
    21. 21. Meeting room set-up
    22. 22. What can go wrong? <ul><li>User forgets </li></ul><ul><ul><li>Send reminder one day in advance </li></ul></ul><ul><li>No shows </li></ul><ul><ul><li>Provide your cell # and get participant’s cell and work # </li></ul></ul><ul><li>User cannot download plug in </li></ul><ul><ul><li>Send participant WebEx compatibility test check email in advance </li></ul></ul>
    23. 23. What can go wrong? <ul><li>User takes a long time to log onto WebEx </li></ul><ul><ul><li>Add extra 10 minutes for WebEx set-up </li></ul></ul><ul><ul><li>Log onto session 15 minutes in advance </li></ul></ul><ul><ul><li>Get participant to log onto WebEx 10 minutes in advance </li></ul></ul><ul><li>User brings along his co-worker </li></ul><ul><ul><li>Explain 1to1 test </li></ul></ul><ul><ul><li>Reschedule secondary participant </li></ul></ul>
    24. 24. What can go wrong? <ul><li>Can’t see what the user is doing </li></ul><ul><ul><li>Ask user to point with mouse </li></ul></ul><ul><li>Conversations </li></ul><ul><ul><li>Mute your line </li></ul></ul>
    25. 25. WebEx Limitations <ul><li>Test! Test! Test! – Dry run </li></ul><ul><li>Delays </li></ul><ul><ul><li>Explain that there is a slight delay between what they are seeing and what you see </li></ul></ul><ul><li>Cannot view hover overs </li></ul><ul><ul><li>Verbally tell users what they are hovering over </li></ul></ul>
    26. 26. Pre-test / Test checklist <ul><li>Email scenarios </li></ul><ul><li>Set-up system or web site </li></ul><ul><li>Open up required documents </li></ul><ul><li>Dial into conference bridge or participant </li></ul><ul><li>Sign onto WebEx </li></ul><ul><li>Facilitator script </li></ul><ul><li>Start recording </li></ul><ul><li>Hand WebEx control to participant </li></ul>
    27. 27. Incentives <ul><li>Challenging for international users </li></ul><ul><li>Email local Amazon gift certificates </li></ul>
    28. 28. Resources <ul><li>IBM - Experience remote usability testing </li></ul><ul><li> </li></ul><ul><li>Remote Usability </li></ul><ul><li>User Experience magazine </li></ul><ul><li>Moderating Usability Tests: Principles and Practices for Interacting by Loring and Dumas </li></ul><ul><li>Usability Engineering by Jakob Neilsen </li></ul>
    29. 29. Remote tools <ul><li>Liz Bacon listing </li></ul><ul><li>Bolt Peters </li></ul>
    30. 30. Questions <ul><li>Feel free to connect with me at [email_address] or </li></ul><ul><li> </li></ul>