With the realities of social distancing likely impacting society for months, if not years, those of us who do physical product evaluation with users have spent the last few weeks reconfiguring our study designs to keep moving forward with a user-centered product development process. At Priority Designs, we’ve spent the last few weeks getting innovative and being realistic about how we move forward with design research in a socially distanced world. We’d like to share our experiences with remote research over the last few weeks, along with a perspective of 20 years of experience in design research. In this presentation, our intent is to help you understand when remote research supports good decision making, what to consider regarding your unique project, and when it might be best to wait for in-person research. This session took place live at the Greenlight Guru True Quality Virtual Summit, a three-day event for medical device professionals to learn to get their devices to market faster, stay ahead of regulatory changes, and use quality as their multiplier to grow their device business.

1. Why Remote Research Is (Or Isn’t)
Right For Your Project

2. REMOTE RESEARCH
INTRODUCTIONS
Colton Turner, PhD
Sr. Human Factors Specialist
Katie Mowery
Sr. Human Factors Specialist

3. REMOTE RESEARCH
PRESENTATION OUTLINE
1 HAVE YOU HEARD THERE’S A PANDEMIC?
2 BENEFITS OF REMOTE RESEARCH
3 HURDLES OF REMOTE RESEARCH
4 WHEN IT’S BEST TO WAIT
5 KEY TAKEAWAYS

4. HAVE YOU HEARD THERE’S A PANDEMIC?

5. IMPACTS OF COVID-19
• Identifying whether to keep moving or press
pause
• Re-thinking research methods
COVID-19 has greatly impacted the way
we conduct research, but this does not
mean user research must stop.

6. BENEFITS OF REMOTE RESEARCH

7. IN-HOME CONTEXT
• Devices intended for home use
• No strangers in the participant’s home
Users may feel more comfortable in their home with
the research team in a remote location.

8. There are many commercially available tools for
users to document their experiences in real-time.
REAL-TIME DOCUMENTATION
Digital Diary Data Collection
• Real-time documentation of an experience
• Priming for the interview

9. LIVE INTERVIEWS
• Video conferencing can capture
interactions as well as a live interview
• Real-time probing
Video conferencing allows for the researcher to
guide the participant through a series of tasks,
asking questions along the way.

10. HURDLES OF REMOTE RESEARCH

11. HURDLES OF REMOTE RESEARCH
1 BUILDING RAPPORT
2 OBSERVER ANONYMITY
3 CONTEXTS OF USE
4 EXPERIMENTAL CONTROL
5 LOGISTICAL CHALLENGES
6 CONFIDENTIALITY

12. BUILDING RAPPORT
Problems with technology could lead
to awkwardness or even frustration

13. MITIGATION | BUILDING RAPPORT
1. Add time to build rapport

14. MITIGATION | BUILDING RAPPORT
1. Add time to build rapport
2. Acknowledge the awkwardness

15. MITIGATION | BUILDING RAPPORT
1. Add time to build rapport
2. Acknowledge the awkwardness
3. Be human, humble, and humorous

16. HURDLES OF REMOTE RESEARCH
1 BUILDING RAPPORT
2 OBSERVER ANONYMITY
3 CONTEXTS OF USE
4 EXPERIMENTAL CONTROL
5 LOGISTICAL CHALLENGES
6 CONFIDENTIALITY

17. OBSERVER ANONYMITY
• Participants may be able to see the volume of
observers, which can be intimidating
• Email or location of each of the observers may
be visible
• Meeting chat is visible to the participant
• Participants may be able to identify the research
sponsor based on displayed locations

19. MITIGATION | OBSERVER ANONYMITY
1. Use only first names, and last initial if required

20. MITIGATION | OBSERVER ANONYMITY
1. Use only first names, and last initial if required
2. Do not use a service that requires company name or email address

21. MITIGATION | OBSERVER ANONYMITY
1. Use only first names, and last initial if required
2. Do not use a service that requires company name or email address
3. Dial in at least 5 minutes early

22. MITIGATION | OBSERVER ANONYMITY
1. Use only first names, and last initial if required
2. Do not use a service that requires company name or email address
3. Dial in at least 5 minutes early
4. Keep your mic on mute and camera off

23. HURDLES OF REMOTE RESEARCH
1 BUILDING RAPPORT
2 OBSERVER ANONYMITY
3 CONTEXTS OF USE
4 EXPERIMENTAL CONTROL
5 LOGISTICAL CHALLENGES
6 CONFIDENTIALITY

24. CONTEXTS OF USE
Environment: Products not intended for home use
Group Use: Shared by more than one user
Intended Use: Active use can be hindered

25. MITIGATION | CONTEXTS OF USE
1. Environment: Consider whether aspects of the environment can be
simulated (e.g. surgical suite sounds)

26. MITIGATION | CONTEXTS OF USE
1. Environment: Consider whether aspects of the environment can be
simulated (e.g. surgical suite sounds)
2. Group Use: Consider whether a researcher can “role play” as the other
user to simulate the experience (e.g. medical assistant)

27. MITIGATION | CONTEXTS OF USE
1. Environment: Consider whether aspects of the environment can be
simulated (e.g. surgical suite sounds)
2. Group Use: Consider whether a researcher can “role play” as the other
user to simulate the experience (e.g. medical assistant)
3. Intended Use: Consider whether mounting a mobile device or camera
will mitigate the issue (e.g. tripod or head mounted camera)

28. HURDLES OF REMOTE RESEARCH
1 BUILDING RAPPORT
2 OBSERVER ANONYMITY
3 CONTEXTS OF USE
4 EXPERIMENTAL CONTROL
5 LOGISTICAL CHALLENGES
6 CONFIDENTIALITY

29. EXPERIMENTAL CONTROL
When the participant is remote, even research with a strong
plan cannot control for changes in lighting, noise, and other
distractions. This impacts:
• Time on Task
• Visibility
• Audio cues

30. 1. Consider the impact of these variables that you cannot
control
MITIGATION | EXPERIMENTAL CONTROL

31. 1. Consider the impact of these variables that you cannot
control
2. Probe to the root cause to understand if it’s an
environmental factor
MITIGATION | EXPERIMENTAL CONTROL

32. 1. Consider the impact of these variables that you cannot
control
2. Probe to the root cause to understand if it’s an
environmental factor
3. Recruit extra participants to account for potential outliers
or data loss
MITIGATION | EXPERIMENTAL CONTROL

33. HURDLES OF REMOTE RESEARCH
1 BUILDING RAPPORT
2 OBSERVER ANONYMITY
3 CONTEXTS OF USE
4 EXPERIMENTAL CONTROL
5 LOGISTICAL CHALLENGES
6 CONFIDENTIALITY

34. LOGISTICAL CHALLENGES
1. Participants’ personal wi-fi and tech-setup
2. Increased time to ship and disinfect products between
participants
3. Prototypes and test materials could break, be delayed, or
lost in transit

35. 1. Plan additional time for shipping and prototype delivery
MITIGATION | LOGISTICAL CHALLENGES

36. 1. Plan additional time for shipping and prototype delivery
2. Perform a “tech-check” call to ensure any technical
difficulties are sorted (or gain permission to setup in the
user’s home)
MITIGATION | LOGISTICAL CHALLENGES

37. 1. Plan additional time for shipping and prototype delivery
2. Perform a “tech-check” call to ensure any technical
difficulties are sorted (or gain permission to setup in the
user’s home)
3. Consider using the laptop and web cam as an incentive
MITIGATION | LOGISTICAL CHALLENGES

38. 1. Plan additional time for shipping and prototype delivery
2. Perform a “tech-check” call to ensure any technical
difficulties are sorted (or gain permission to setup in the
user’s home)
3. Consider using the laptop and web cam as an incentive
4. Develop extra prototypes in case of damage
MITIGATION | LOGISTICAL CHALLENGES

39. 1. Plan additional time for shipping and prototype delivery
2. Perform a “tech-check” call to ensure any technical
difficulties are sorted (or gain permission to setup in the
user’s home)
3. Consider using the laptop and web cam as an incentive
4. Develop extra prototypes in case of damage
5. Optimize prototype design for easier disinfection
MITIGATION | LOGISTICAL CHALLENGES

40. HURDLES OF REMOTE RESEARCH
1 BUILDING RAPPORT
2 OBSERVER ANONYMITY
3 CONTEXTS OF USE
4 EXPERIMENTAL CONTROL
5 LOGISTICAL CHALLENGES
6 CONFIDENTIALITY

41. CONFIDENTIALITY
Remote testing lacks control over actions the participant
can take with the product that could impact your intellectual
property (e.g., taking photos or taking screenshots) and sharing
them

42. 1. Non-disclosure agreements
MITIGATION | CONFIDENTIALITY

43. 1. Non-disclosure agreements
2. Share, do not email, electronic prototypes
MITIGATION | CONFIDENTIALITY

44. 1. Non-disclosure agreements
2. Share, do not email, electronic prototypes
3. Modify URL for web prototypes after testing
MITIGATION | CONFIDENTIALITY

45. WHEN IT’S BEST TO WAIT

46. QUESTIONS TO CONSIDER
1. Is the environment in which this product will be used critical in understanding the user
experience?
If yes, it might be better to wait.

47. QUESTIONS TO CONSIDER
1. Is the environment in which this product will be used critical in understanding the user
experience?
If yes, it might be better to wait.
2. Are there other users who might simultaneously use the product that can’t be present or
simulated for testing?
If the user’s portion of tasks can still yield good data, move forward. If the experience
is largely interactive, it might be better to wait.

48. QUESTIONS TO CONSIDER
1. Is the environment in which this product will be used critical in understanding the user
experience?
If yes, it might be better to wait.
2. Are there other users who might simultaneously use the product that can’t be present or
simulated for testing?
If the user’s portion of tasks can still yield good data, move forward. If the experience
is largely interactive, it might be better to wait.
3. Does my research require a distraction-free, controlled environment in order to yield
confidence in the results?
If yes, you should wait.

49. KEY TAKEAWAYS

50. FORGING AHEAD
Candidate research topics that can succeed with remote testing:
1 2-D Concept Sketches / Storyboards
2 GUI Formative Testing
3 Product/IFU/QRG Formative Testing

51. PUMP THE BRAKES
Research topics that should wait for a controlled setting
1 Summative Usability Testing
2 Performance Testing (product or user performance)

52. THANK YOU