Remote User Research Methods, tools and practicalities Breakfast Briefing 1 October 2014
@UserVision

Remote User Research
What do we mean by remote user research?
Types of remote usability testing
Practicalities and tips
What to use when
Usability testing is the best way to get empirical evidence of the likely appeal or success of your product

What is remote usability testing?
•Participant far away – can’t interact in person
•It is NOT
…collecting analytics, Split testing or MVT
VWO, Optimisely, Google Experiments
… where user is unaware they are being tested
Background playback tools – Mouseflow, Sessioncam
Heatmaps of where user clicked – Clicktale, Crazy egg
… only a survey of opinion
Eg. Surveymonkey, primarily based on opinion
… remote ethnography, diary study

Some things that we’re not talking about
•Various ways to record clicks, mouse movements, scrolling and secretly record users using your site.
•Crazy Egg
•Clicktale
•Inspectlet
•Sessioncam
•Mouseflow
•Anyone using these?

Remote research
•Advantages
 Access users anywhere in the world
 Remove travel costs
 Faster turnaround – results within an hour
 Natural environment –at home
 Can ‘live recruit’ to capture ‘in the moment’ experience
 More honest / representative?
•Disadvantages
 No chance to talk with user (unless remote moderated)
 Less rich results compared to standard F2F tests
 Types of tasks may be constrained
 Potential for technical glitches
 Getting the right audience may be difficult

Other considerations
•Still need a good research plan
What do you want to find out?
Who participates? Are they familiar with your product?
Do you want qualitative or quant insights?
•Make it personal, friendly
•Be clear in task scenarios, and success states
•Consider the audience:
Will technology overly complicate the process?
How can we best replicate real-life circumstances?
Online research audience may be more web savvy or ‘professional testers’

Remote User Research
What do we mean by remote user research?
Types of remote usability testing
Practicalities and tips
What to use when

Types of remote research
1.Remote moderated – Like normal testing but aided by screenshare / audio share – live discussion
2.Remote unmoderated – User goes through tasks to collect behavioural & opinion data, no discussion

Remote Moderated Testing

Remote moderated testing
•Like a face to face test – over distance
•See their screen and hear the audio
•Captures experience from where they normally use the web
•Can allow 3rd party observation
•Technical issues may occur

Remote Moderated Testing - example
Edinburgh, London
Mexico
Brazil
Nigeria
Egypt
Pakistan
Bangladesh
Indonesia

Remote observation – with translation
Tests conducted in China – in Chinese
Simultaneous translation to English
Observed by UX consultant in the UK, listening to the English translation
Observed by the client in Dubai
Observed by the agency in NY

Practicalities and tips for remote moderated tests
•Five common practical questions:
What can you test?
Who should you test with?
Where can participants be located?
How should you conduct the testing?
When will the sessions happen?
•Already covered why…

What can you test?

What can you test?
•Website/web services
Most common
‘Best fit’ for this form of testing

What can you test?
•Website/web services
•Corporate IT systems/Intranets
Can present some security/firewall issues
Some participants can feel constrained when taking part from their employer’s environment

What can you test?
•Website/web services
•Corporate IT systems/Intranets
•Highly graphical/animated interfaces e.g. Flash
Can suffer in transmission due to frame-rate/image quality limitations

What can you test?
•Website/web services
•Corporate IT systems/Intranets
•Highly graphical/animated interfaces e.g. Flash
•Mobile
Much bigger challenge
No single, cross-platform solution
Many security-related restrictions (and risks) at OS level regarding screen-sharing

What can you test?
•Mobile: ‘Laptop hugging’ technique

Who should you test with?

Who should you test with?
•Anyone…
…with some caveats

Who should you test with?
IT Confidence
Context
High
Low
Unrepresentative
Representative

Where can participants be located?

Where can participants be located?
•Anywhere
…with some caveats:
Language requirements

Where can participants be located?
•Anywhere
…with some caveats:
Language requirements
Geographical regions with ‘endemic infrastructure limitations’

How should you conduct the testing?

How should you conduct the testing?
•Many screen sharing apps available
Join.me
Webex
GoToMeeting
Adobe Connect
Skype

How should you conduct the testing?

How should you conduct the testing?
•Skype:
 Good audio quality

How should you conduct the testing?
•Skype:
 Good audio quality
 Relatively low bandwidth requirements

How should you conduct the testing?
•Skype:
 Good audio quality
 Relatively low bandwidth requirements
 ‘Set up’ already done by the participant

How should you conduct the testing?
•Skype:
 Good audio quality
 Relatively low bandwidth requirements
 ‘Set up’ already done by the participant
 Implies some level of IT comfort/capability

How should you conduct the testing?
•Skype:
 Good audio quality
 Relatively low bandwidth requirements
 ‘Set up’ already done by the participant
 Implies some level of IT comfort/capability
 Gets the communications technology ‘out of the way’

How should you conduct the testing?
•Skype:
 Good audio quality
 Relatively low bandwidth requirements
 ‘Set up’ already done by the participant
 Implies some level of IT comfort/capability
 Gets the communications technology ‘out of the way’
 Risk that it restricts pool of potential participants

How should you conduct the testing?
•Skype:
 Good audio quality
 Relatively low bandwidth requirements
 ‘Set up’ already done by the participant
 Implies some level of IT comfort/capability
 Gets the communications technology ‘out of the way’
 Risk that it restricts pool of potential participants
 Shifting sands of features/account types

How should you conduct the testing?
•Skype:
 Good audio quality
 Relatively low bandwidth requirements
 ‘ ‘Set up’ already done by the participant
 Implies some level of IT comfort/capability
 Gets the communications technology ‘out of the way’
 Risk that it restricts pool of potential participants
 Shifting sands of features/account types
And…
ALWAYS have a back-up plan!

When will the sessions happen?

When will the sessions happen?
•Time.is:

When will the sessions happen?
•Time.is:

When will the sessions happen?
•Time.is
•Calendly:

When will the sessions happen?
•Time.is
•Calendly:

When will the sessions happen?
•Time.is
•Calendly:

When will the sessions happen?
•Time.is
•Calendly
•Google docs:
Calendar
Spreadsheet

Remote Un-moderated Testing

Self-moderated (video) testing
•Participants video themselves doing task & talking
•Can’t discuss in real time with the participant
•Recruited from provider’s panel, crowdsourcing / social media or your own list
•Participants set up own web cam
•Still need time to review the videos, analyse , recommend solution
•Better for general audiences rather than specialist

Self-moderated (video) testing
•Usertesting.com
Panel focused on US, Canada, UK
Can annotate, edit video clips
Peek as a free trial
•WhatUsersDo
Panel covering 26 countries, can be filtered
•Both can cover mobile platforms
Example - mobile - http://youtu.be/zb0HigU_rys

Remote unmoderated testing
•Various tools, most capturing quantitative & qual
•Track success by specifying success pages
•May be able to test competitor sites
•Both can test on mobile

Other Mobile variations
•Forward facing camera captures user’s face
•Apsee
a mobile iOS app analytics platform
Requires line of code inserted to the App
Captures user recordings, creates touch heatmaps
•Lookback
•UX Recorder
Mobile website (not Apps) testing

Quick Exposure tests
•Five Second Tests
•UI Tests – 10 people for 10 Seconds for $9
www.uitests.com/t/aXmgpXE
•Usability Tools
Click Testing
Web Testing
Card Sorting

Optimal workshop

Verify
•Test screenshots / designs to gain insights on expectations and reactions
1.Preference test – which of 2 options prefer and why
2.Yes / no test –Mark areas, see if click right place
3.Click test – See where people click based on instructions
4.Multiclick test – As above but can compare different sites
5.Memory test – like the 5 seconds test – what you remember
6.Annotate test – allow free form notes to be marked on page
7.Mood test –show screenshot, select from emotion options
8.Label test – Check users understanding of what link will do

Example output - Verify

Usabilla
•Usabilla Visual Survey
Collect visual feedback on wireframes, mockups or any other visual
Feedback shows where people clicked in a heatmap

Choosing remote methods
Qualitative
Concrete
Quantitative
Conceptual
Video Self- moderated
Remote Moderated
IA, Short tests, Verify
Remote Unmoderated

Resources
•http://remoteresear.ch/
•Books
Beyond the Usability Lab – Bill Albert
Remote research – Nate Bolt

55 North Castle Street
Edinburgh
EH2 3QA
Tel: 0131 225 0853
www.uservision.co.uk
info@uservision.co.uk
@UserVision
Contact us