Elizabeth Snowdon is a senior usability analyst at Sage Software who has been conducting usability studies since 2003. She discusses remote usability testing, which involves testing sessions where the participant and moderator are not physically in the same place but communicate electronically. Some advantages of remote testing are that it expands recruitment populations, saves costs compared to in-person lab testing, and allows testing in users' natural environments. However, remote testing has disadvantages like not being able to see participants' screens and potential technical issues. Snowdon provides an example case study where remote testing saved $50,000 compared to in-lab testing.

1. Remote Usability Testing December 5, 2009 Elizabeth Snowdon Usability Analyst, Sage Software [email_address]

2. About me Senior Usability Analyst at Sage Software Previously worked in usability at high-tech company Conducting usability studies since 2003

3. Principles of Scientific Management

4. Traditional Lab Testing

5. Remote usability Advantages / Disadvantages Case study Tools What I will talk about today

6. What is Remote Testing? Testing sessions in which you and participant are not physically in the same place but are communicating via electronic technology

7. Types of Remote Testing One on one and real time Shared technology No moderator Self-paced Special tools record activities & feedback

8. Current Usability Testing Trends Movement away from lab testing Increased interest in testing in users’ environments Agile development methods

9. Advantages Lab not required University of Leeds Human Perception and Performance: Usability Laboratory

10. Advantages Expands sampling population Ease of recruitment Users in their own natural browsing environment Share their desktop with you

11. Advantages Save money Travel Equipment Recruiting costs Compensation

12. Disadvantages Difficult to see participant’s screen Cannot see participants Problems downloading or installing desktop sharing Task scenarios disseminated in advance Participants interrupted or distracted

13. When not to use When security is important When reliable high speed internet connection is unavailable For eye tracking studies When it is important to attend to user’s physical movement

14. When to use Recruiting valuable individuals Minimal set-up For lone wolf consultants Rapid iterative development (Agile)

15. Recruiting Users

16. Case Study Baseline study for Accpac Three personas clerks, controllers, operations 24 participants 20 remote US, South Africa, Australia, UK, Canada One moderator, one note taker, observers

17. Results Accpac study completed in 4 weeks Cost savings Travel Lab set-up Time savings Recruiting Scheduling Less overtime Bottom line - $50,000 saved

18. Recruiting Internal recruitment, customer lists, Craigslist, email Interested parties must pass screener

19. Other issues to sort out Obtaining informed consent and confidentiality agreement What computing and communications equipment do candidates have? Are they willing and able to download and run collaboration software?

20. Typical Set-up Moderator / Note-taker Screen sharing: WebEx Recording: Morae Speakerphone Participant High speed internet access Speakerphone or headset telephone

21. Meeting room set-up

22. What can go wrong? User forgets Send reminder one day in advance No shows Provide your cell # and get participant’s cell and work # User cannot download plug in Send participant WebEx compatibility test check email in advance

23. What can go wrong? User takes a long time to log onto WebEx Add extra 10 minutes for WebEx set-up Log onto session 15 minutes in advance Get participant to log onto WebEx 10 minutes in advance User brings along his co-worker Explain 1to1 test Reschedule secondary participant

24. What can go wrong? Can’t see what the user is doing Ask user to point with mouse Conversations Mute your line

25. WebEx Limitations Test! Test! Test! – Dry run Delays Explain that there is a slight delay between what they are seeing and what you see Cannot view hover overs Verbally tell users what they are hovering over

26. Pre-test / Test checklist Email scenarios Set-up system or web site Open up required documents Dial into conference bridge or participant Sign onto WebEx Facilitator script Start recording Hand WebEx control to participant

27. Incentives Challenging for international users Email local Amazon gift certificates

28. Resources IBM - Experience remote usability testing www.ibm.com/developerworks/library/wa-rmusts1/ Remote Usability remoteusability.com/ User Experience magazine ww.usabilityprofessionals.org/upa_publications/user_experience/past_issues/2008-3.html Moderating Usability Tests: Principles and Practices for Interacting by Loring and Dumas Usability Engineering by Jakob Neilsen

29. Remote tools Liz Bacon listing http://remoteusability.com/remote-testing-tools-round-up-by-liz-bacon/ Bolt Peters http://remoteusability.com/tools/

30. Questions Feel free to connect with me at [email_address] or elizabethsnowdon.wordpress.com/