Distributed Remote Psychophysiological Data Collection for UX Evaluation: A Pilot Project

Distributed Remote
Psychophysiological Data
Collection for UX Evaluation
A PILOT PROJECT
2021 HCI International virtual conference, July 24-29 2021
Vanessa Georges
Audrey Valiquette
David Brieugne
Emma Rucco
Constantinos K. Coursaris
Marc Fredette
Sylvain Sénécal
HEC Montréal,
Montréal, Canada
Aurélie Vasseur
Pierre-Majorique Léger
François Courtemanche
Elise Labonte-Lemoyne

© Copyright Vasseur et al. (2021)
Constantinos K. Coursaris,
Ph.D., M.B.A., B.Eng.
ASSOCIATE PROFESSOR, DEPARTMENT OF INFORMATION TECHNOLOGIES
Co-Director Tech3Lab
ckc@hec.ca
Google Scholar
www.bitly.com/scholarcoursaris/
PH.D. INFORMATION
SYSTEMS & M.B.A.
E-COMMERCE
B. ENG. AEROSPACE
RESEARCH LABORATORY
& CHAIR
VISITING PROFESSOR
FUNDING

Research Motivation & Goal
1 1
2 2
Develop a rigorous and contextually relevant
protocol for remote physiological data collection
in UX evaluations
Provide guidance through methodological
contributions to fellow UX researchers along
with opportunities for future research.
Validate feasibility and reliability of developed
protocol
Uncover key success factors in remote collection
of UX physiological data
MOTIVATIONS GOALS:

Marc Fredette, Ph.D.
Data Sciences
Sylvain Sénécal, Ph.D.
Marketing
Pierre-Majorique Léger, Ph.D.
Information technologies

Powered by:

▪ Electroencephalography

▪ Functional near-infrared spectroscopy

▪ Electrocardiogram

▪ Electrodermal activity

▪ Eye tracking

▪ Facial expression analysis
STATE OF THE ART
NEUROPHYSIOLOGICAL TOOLS

Industrial research partners
from a wide range of industries
INSURANCE
ONLINE GROCERY
MEDIA
RAILWAYS
BANKING
AERONAUTICS
LOGISTICS
FINANCE
r partenaire de
PLUSIEURS BOURSES DE RECHERCHE
À TOUS LES NIVEAUX :
chaire_ux.hec.ca
INSCRIVEZ-VOUS COMME
PARTICIPANTS À NOS ÉTUDES :
panel.hec.ca
Organismes subventionnaires :
partenaire de

est fier partenaire de
SEVERAL RESEARCH SCHOLARSHIPS
AVAILABLE AT ALL LEVELS:
chaire_ux.hec.ca
REGISTER AS A PARTICIPANT
IN OUR STUDIES:
panel.hec.ca
a proud partner of
Powered by: Financial support:

1 2 3 4
UX Evaluation
today
UX Evaluation
with neuroscientific
methods
How to evaluate UX
methods during
COVID-19?
What are
the lessons
learned?
A B
?

Immobilier.ca
Please
test!

Immobilier.ca

Immobilier.ca
Testez-le
svp!
Ease
of use?

Immobilier.ca
3 of 5
Testez-le
svp!
Ease
of use?

Immobilier.ca
Testez-le
svp!
?
?
?
?
?
?
?
?

?
Conclusion?

1 2 3 4
UX Evaluation
today
How to evaluate UX
methods during
COVID-19?
What are
the lessons
learned?
A B
?
2
UX Evaluation
methods

NeuroIS:
The Basic Idea
IT Behavior
BIOLOGY

▪ Body physiology

▪ Brain anatomy & functionality

▪ Hormones

▪ Genes

Traditional Approach
EXAMPLE: SHOPPING BEHAVIOR
INDEPENDENT VARIABLE DEPENDENT VARIABLE
e.g., perceived
trustworthiness of ebay offer
or
purchase intention

NeuroIS Approach
EXAMPLE: SHOPPING BEHAVIOR
IT ARTIFACT
MEDIATOR VARIABLE
IS RELEVANT VARIABLES

“NeuroIS is a subfield in the IS literature that relies on neuroscience and neurophysiological
theories and tools to better understand the development, use, and impact of information
technologies (IT). NeuroIS seeks to contribute to
(i) the development of new theories that make possible accurate predictions
of IT-related behaviors, and
(ii) the design of IT artifacts that positively affect economic and non-economic variables
(e.g., productivity, satisfaction, adoption, well being).”
What is NeuroIS?
Riedl, R. et al. (2010): On the foundations of NeuroIS: Reflections on the Gmunden Retreat 2009. Communications of the Association for Information Systems, 27, 243-264.

Immobilier.ca
COGNITION
ATTENTION
EMOTIONS
1 gig/hour
ENCYCLOPEDIA
ENCYCLOPEDIA
ENCYCLOPEDIA
ENCYCLOPEDIA
ENCYCLOPEDIA
ENCYCLOPEDIA
ENCYCLOPEDIA
ENCYCLOPEDIA

COGNITION
ATTENTION
EMOTIONS

Mesuring emotional
arousal Calm Excited

Mesuring emotional
arousal
Calm Excitd

Remote neurophysiological data collection

Mesuring emotional
valence Sad Happy

NEUTRAL
HAPPY
SAD
DISCUSTED
ANGRY
SURPRISED
SCARED

VALENCE
EMOTIONAL
Fruitless
search
using the
tool bar
Inscription:
Entering
personal data
Adding filters
to choose
the product
Search
results
Products
without photos
Click tio
“buy”
Submitting
tel. number
Emotional
Intensity
Intensité émotionnelle
Emotional valence
Identification
of friction points
for each step

1 2 3 4
UX Evaluation
today
UX Evaluation
methods
What are
the lessons
learned?
A B
?
3
How to evaluate UX
methods during
COVID-19?

Remote UX testing
PARTICIPANT'S SCREEN MODERATOR PARTICIPANT

Remote UX testing
PARTICIPANT SCREEN INSTRUCTION PARTICIPANT

Participant recruitment Participation prerequisites
Chrome
web browser
Can be downloaded and installed
with this link:
https://www.google.com/chrome/
1
A microphone
integrated or connected
to your computer
2
A webcam
integrated or connected
to your computer
3
4
Isolated workspace
An isolated work space
(in order to reduce noise and
the presence of other people
during the test)
Good lighting
(your face must be well lit
for video recording)
No glasses
You must be able to work on a
computer without seeing glasses
5 6
Access to Wi-Fi
If available, select your
5Ghz Wi-Fi network

Participant set-up To do the day before the test
Test your webcam and microphone
3
Read and complete the consent form
1
Conduct a Wi-Fi speed test
2
Install the LookBack extension in Chrome
4
1. Open the “Consent Form” PDF
in your participation confirma-
tion e-mail
2. Carefully read the whole
document, and complete it by
answering all of the questions
3. Send the completed consent
form to panel.admin@hec.ca
1. If you are connected to Wi-Fi,
and you have two networks
available, please select your
5Ghz network.
2. Go to the following website :
https://fast.com/ and click “GO”
3. Click “Show More Info”
1. Go to https://webcamera.io/ and
make a recording, lasting at least
15 seconds, in which you talk.
2. Watch the recording, making sure
that both video and sound are of
good quality.
1. Open your Chrome
web browser
2. Install the Lookback extension
using the link provided in your
Participation Confirmation email
3. Accept the use of your microphone,
webcam, and screen recording.
4. Please contact the panel.admin@hec.ca
if your Wi-Fi does not meet the following
thresholds :
INSTALL
ACCEPT
2.4Ghz 5Ghz
GO
OVER
Mbps
5
OVER
Mbps
5
UNDER
ms
300
Internet Speed Upload Speed Latency

Preparation for the test
1
Wi-Fi access
If available, select your
5Ghz Wi-Fi network
To do 15 minutes before
your scheduled participation
Close all unnecessary
windows and software
Quit all software and close all
web pages on your computer,
except for your e-mail
access point.
2
Hide
personal information
Please make sure that no
personal information is visible
on your desktop
(for example, photos or documents)
3
Copy the Lookback link
Copy and paste the Lookback link
included in your Participation
Confirmation email.
4
Set up the study
Click
Begin the session
Stay on the Lookback Page until your scheduled participation time.
5
6
Open the study in Lookback
GET STARTED!
1. Only enter your first name
2. Enter your email address
(the one used to communicate
with Panel HEC)
3. Accept the use of your microphone
4. Accept the use of your camera
5. Accept the recording of your
screen
to start the test setup:

Procedure
1 2 3
SET-UP TASK AND QUESTIONNAIRE INTERVIEW

Guidelines for
collecting automatic
facial expression in
remote moderated user
test
REMOTE NEUROPHYSIOLOGICAL UX TESTING
Giroux, F., Léger, P. M., Brieugne, D., Courtemanche, F., Bouvier, F.,
Chen, S.L., Tazi, S., Rucco, E., Fredette, M., Coursaris, C., Sénécal,
S.: Guidelines for collecting automatic facial expression detection
data synchronized with a dynamic stimulus in remote moderated
user tests. In: International Conference on Human-Computer Inter-
Giroux, F., Léger, P. M., Brieugne, D., Courtemanche, F., Bouvier, F., Chen, S.L., Tazi,
Guidelines for collecting automatic facial expression
detection data synchronized with a dynamic stimulus in
remote moderated user tests
Félix Giroux1
, Pierre-Majorique Léger1,2
, David Brieugne1
, François Courtemanche1
,
Frédérique Bouvier1
, Shang-Lin Chen1
, Salima Tazi1
, Emma Rucco1
, Marc
Fredette1,4
, Constantinos Coursaris1,2
and Sylvain Sénécal1,3
1 Tech3Lab, HEC Montréal, Montréal, Québec, Canada
2
Department of Information Technologies, HEC Montréal, Montréal, Québec, Canada
3
Department of Marketing, HEC Montréal, Montréal, Québec, Canada
4
Department of Decision Sciences, HEC Montréal, Montréal, Québec, Canada
Abstract. Because of the COVID-19 pandemic, telework policies have required many user
experience (UX) labs to restrict their research activities to remote user testing. Automatic Facial
Expression Analysis (AFEA) is an accessible psychophysiological measurement that can be
easily implemented in remote user tests. However, to date, the literature on Human Computer
Interaction (HCI) has provided no guidelines for remote moderated user tests that collect facial
expression data and synchronize them with the state of a dynamic stimulus such as a webpage.
To address this research gap, this article offers guidelines for effective AFEA data collection that
are based on a methodology developed in a concrete research context and on the lessons learned
from applying it in four remote moderated user testing projects. Since researchers have less
control over test environment settings, we maintain that they should pay greater attention to
factors that can affect face detection andor emotion classification prior, during, and after remote
moderated user tests. Our study contributes to the development of methods for including
psychophysiological and neurophysiological measurements in remote user tests that offer
promising opportunities for information systems (IS) research, UX design, and even digital health
research.
Keywords: NeuroIS, User Experience, Remote User Test, Automatic Facial Expression
Analysis, Psychophysiological Data, Human-Computer Interaction
1 Introduction
In Human Computer Interaction (HCI) research, conducting a user experience (UX)
study remotely can improve access to participants and provide an ecologically valid
environment for tests in remote environments such as a person’s living room [1].
Nonetheless, remote user tests are limited in terms of the equipment and measurement
tools that can be used and installed during this distributed setup, preventing scholars
from typically collecting psychophysiological data. Therefore, lab-based user tests are
still preferable as they allow researchers to enrich their understanding of the user’s
experience by triangulating traditional self-reported measures via survey scales and
interviews with psychophysiological measurements such as automatic facial expression
analysis (AFEA) [2, 3]. However, due to the COVID-19 pandemic and the telework
policies that have been introduced as part of the public health response, user experience

How to use neuroscience in a remote context ?
REMOTE PROCEDURE TECH3LAB COBALT
+

Tech3Lab
COBALT initiative

100% Tech3lab
DESIGNED AND PRINTED AT TECH3LAB

Pilot study utilizing new measuring instrument and
tailored protocol in 2 consecutive experiments:
1 Website usability study
(search tasks on hotel booking site)
2 Video experience study
(physiological reaction to video stimuli)
Sample size:

▪ 46 test sessions

▪ 92 completed questionnaires
(participants and moderators)
Methodology

1 Physiological measures EDA & ECG
(not reported here as it is outside
of this paper's scope)
2 Post-test closed questionnaire question
on Effort
3 Post-test open-ended interview questions
regarding experience
Measures

▪ Installation of sensors

▪ Interactions with participant /
moderator

▪ Completion the study

▪ Emotional state during study

Data Analysis
CONTENT ANALYSIS
ROUND 0

Inspecting
the data

▪ Outcome:
initial thematic
map
ROUND 1

Coding the
extracts

▪ Outcome:
list of codes
ROUND 2

Grouping
the codes

▪ Outcome:
list of
categories
ROUND 3

Revising the codes
and categories

▪ Outcome:
final thematic
map
FINAL
THEMATIC
MAP

Results
INITIAL ANALYSIS
Two emergent categories:
human and technical factors.
Facilitators
of success
Visual
feedback
Technical
capabilities
Training
Support
Collaboration
HUMAN FACTORS TECHNICAL FACTORS

Results
CONTENT ANALYSIS
Human factors contributed to positive experience:

▪ Respondents appreciative of training received &
of remote support

▪ Moderators and participants: collaboration was key
in experience
Technical factors limited or disrupted aspects of experience:

▪ Network issues or technical problems from the
test interface: stress source or reason to stop
the experiment

▪ Since information is shared by voice or through visual
feedback obtained from the camera of the participant,
the technical capabilities of participants and modera-
tors seem to be an important factor to consider.

Results
REFINED ANALYSIS
Four key success factors in
relation to a successful remote
physiological data collection:
1 Support
2 Collaboration
3 Individual characteristics
4 Technological capabilities

Results
SUPPORT
Training
Amount of training available
for moderators and participants
with the device, the protocol,
and the test interface.
Documentation
Clarity and completeness
of available documentation.
Operational Involvement
Clarity and completeness
of available documentation.

Results
COLLABORATION
Trust
How much the moderator and
participant trust each other and
remain calm during study
Knowledge
How well does the moderator
know the protocol
How well they can communicate
with the participant
Presence
Availability of moderator to assist
participant only when needed

Results
INDIVIDUAL
CHARACTERISTICS
Technical know-how
Level of general technical
knowledge possessed by
moderators and participants.
Experience in remote tools
Amount of experience in
videoconferencing.
Level of preparation
How well-prepared the modera-
tors and the participants are at
the beginning of the experiment.

Results
TECHNOLOGICAL
CAPABILITIES
of the moderator
Regarding technical setup on their
end (hardware and network).
of the participant
Regarding technical setup on their
end (hardware and network).
of the data-collection device
(measuring instrument)
Extent to which it supports the
remote user (e.g., visual feed-
back, accessories).

Conclusion
This pilot study was an important step in supporting
the remote collection of UX physiological data by:
1 Validating the feasibility and reliability
of our developed protocol
2 Uncovering key success factors, both human
and technological
Next step:

▪ Guidelines for collecting automatic
facial expression data in remote
moderated user tests
Guidelines for collecting automatic facial expression
detection data synchronized with a dynamic stimulus in
remote moderated user tests
Félix Giroux1
, Pierre-Majorique Léger1,2
, David Brieugne1
, François Courtemanche1
,
Frédérique Bouvier1
, Shang-Lin Chen1
, Salima Tazi1
, Emma Rucco1
, Marc
Fredette1,4
, Constantinos Coursaris1,2
and Sylvain Sénécal1,3
1 Tech3Lab, HEC Montréal, Montréal, Québec, Canada
2 Department of Information Technologies, HEC Montréal, Montréal, Québec, Canada
3 Department of Marketing, HEC Montréal, Montréal, Québec, Canada
4
Department of Decision Sciences, HEC Montréal, Montréal, Québec, Canada
Abstract. Because of the COVID-19 pandemic, telework policies have required many user
experience (UX) labs to restrict their research activities to remote user testing. Automatic Facial
Expression Analysis (AFEA) is an accessible psychophysiological measurement that can be
easily implemented in remote user tests. However, to date, the literature on Human Computer
Interaction (HCI) has provided no guidelines for remote moderated user tests that collect facial
expression data and synchronize them with the state of a dynamic stimulus such as a webpage.
To address this research gap, this article offers guidelines for effective AFEA data collection that
are based on a methodology developed in a concrete research context and on the lessons learned
from applying it in four remote moderated user testing projects. Since researchers have less
control over test environment settings, we maintain that they should pay greater attention to
factors that can affect face detection andor emotion classification prior, during, and after remote
moderated user tests. Our study contributes to the development of methods for including
psychophysiological and neurophysiological measurements in remote user tests that offer
promising opportunities for information systems (IS) research, UX design, and even digital health
research.
Keywords: NeuroIS, User Experience, Remote User Test, Automatic Facial Expression
Analysis, Psychophysiological Data, Human-Computer Interaction
1 Introduction
In Human Computer Interaction (HCI) research, conducting a user experience (UX)
study remotely can improve access to participants and provide an ecologically valid
environment for tests in remote environments such as a person’s living room [1].
Nonetheless, remote user tests are limited in terms of the equipment and measurement
tools that can be used and installed during this distributed setup, preventing scholars
from typically collecting psychophysiological data. Therefore, lab-based user tests are
still preferable as they allow researchers to enrich their understanding of the user’s
experience by triangulating traditional self-reported measures via survey scales and
interviews with psychophysiological measurements such as automatic facial expression
analysis (AFEA) [2, 3]. However, due to the COVID-19 pandemic and the telework

Thank you!
Vanessa Georges
Audrey Valiquette
David Brieugne
Emma Rucco
Constantinos K. Coursaris
Marc Fredette
Sylvain Sénécal
HEC Montréal,
Montréal, Canada
Aurélie Vasseur
Pierre-Majorique Léger
François Courtemanche
Elise Labonte-Lemoyne
MORE INFORMATION:
Constantinos.Coursaris@hec.ca

Distributed Remote Psychophysiological Data Collection for UX Evaluation: A Pilot Project

Recommended

Recommended

More Related Content

What's hot

What's hot (10)

Similar to Distributed Remote Psychophysiological Data Collection for UX Evaluation: A Pilot Project

Similar to Distributed Remote Psychophysiological Data Collection for UX Evaluation: A Pilot Project (20)

Recently uploaded

Recently uploaded (20)

Distributed Remote Psychophysiological Data Collection for UX Evaluation: A Pilot Project