The importance of Debriefing in online experiment. Delivered by Harryadin Mahardika at Universitas Indonesia

1. Debriefing in Online Experiment
Harryadin Mahardika, PhD

2. Harryadin Mahardika
• Pop Economist
• FEUI & laporsuap.com
• Research objective:
– “to liberate and empower consumer...”
• Current research:
– Consumer empowerment
– Consumer intervention/engineering
– Mobile advertising
• Contact:
– harryadin.mahardika@ui.ac.id / harryadin@gmail.com
– @HarrySastro
2

3. Discussion Agenda
• Informed consent
• Debriefing
• Randomization
• Participant incentives

4. Risk of experiment on human subject
• Deception, manipulation, priming, scenario.
– May harm participants’ mind.
– Mental fatigue.
• Examples?

5. Informed consent
• Information about the experiment and its risk.
– As a basis for participants to decide after they
understand what the research involves (risks and
benefits ).
• Written consent vs unwritten consent.
– Written:
• Investigators must typically obtain and document voluntary
informed consent from research subjects.
– Unwritten:
• button on an online form to indicate they have read and
understood the consent form.

6. Informed consent in online experiment
• Limited interaction with participants
– investigator often cannot tell whether a subject understood the
informed consent statement.
• Online form:
– Researchers can increase the likelihood that subjects are granting truly
informed consent by requiring feedback from subjects about their
level of understanding,
• Example:
– by requiring a “click to accept” for each element in an informed
consent statement or even administering short quizzes to establish
that a subject understood.
• Reduce response rate:
– Increase nonresponse to sensitive items (Singer, 1978)
– Possibly produce biased data (Trice, 1987).

7. Risk in online experiment
• It exposes subjects to innocuous questions and benign
or transient experiences with little lasting impact.
• In general, online experiments is no more risky than
any of their offline counterparts.
• In some respects, they may be less risky:
– The reduced social pressure (Sproull & Kiesler, 1991) in
online surveys or experiments makes it easier for subjects
to quit whenever they feel discomfort.
– This freedom to withdraw is no trivial benefit, given the
strong pressures to continue in face-to-face studies (e.g.,
Milgram, 1963) and even telephone calls.

8. Risk in online experiment
• Although risk in online settings is typically low,
the actual risk depends on the specifics of the
study.
• For example:
– Some questions in a survey or feedback from an
experiment may cause subjects to reflect on
unpleasant experiences or to learn something
unpleasant about themselves
– e.g., Nosek et al.’s, 2002b, research on automatic
stereotyping.

9. Risk in online experiment
• Experiments that deliberately manipulate a:
– subject’s sense of self-worth,
– reveal a lack of cognitive ability,
– challenge deeply held beliefs or attitudes, or
– disclose some other real or perceived
characteristic
..... may result in mental or emotional harm to some
subjects.

10. Debriefing
• American Psychological Association (2002)
ethical guidelines call for debriefing subjects:
– “Providing an explanation of the nature, results,
and conclusions of the research—as soon after
their participation as practical.“
• If deception was involved:
– Researcher needs to explain the value of the
research results and why deception was
necessary.

11. Debriefing in online experiment
• When conducting research online:
– Researchers can post debriefing materials at a
Web site,
– Provide debriefing materials to those who leave
before completing the research (Nosek, Banaji, &
Greenwald, 2002a).
– For example, researchers can deliver debriefing
material through a link to a “leave the study”
button or through a pop-up window, which
executes when a subject leaves a defined Web.

12. Debriefing in online experiment
• Appropriate debriefing in online research may
be difficult:
– The absence of a researcher in the online setting
makes it difficult to assess a subject’s state.
– Difficult to determine whether an individual has
been upset by an experimental procedure or
understands feedback received.

13. Extraneous Variables cont.
Selection Bias
Improper assignment of test units to
treatment conditions
[sampling error]

14. Extraneous Variables cont.
Mortality
Loss of test units while the experiment is in
progress
[respondents selected no longer wish to
participate]

15. Controlling Extraneous Variables
Randomisation
 Randomly assigning test units to experimental groups by using random
numbers
Matching
 Comparing test units on a set of key background variables before assigning
them to the treatment
Statistical Control
 Measuring the extraneous variables and adjusting for their effects through
statistical analysis
Design Control
 Use of experiments designed to control specific extraneous variables

16. Case: Beauty is in the eye of the beholder
Scientists have found a link between drinking
alcohol and perceptions of beauty
80 students were shown colour photographs of
120 male and female students and were asked to
rate the aesthetic properties on a 7-point scale
from high unattractive to highly attractive
Half the students had drunk up to four units of
alcohol, the other half had no alcohol.
The students who had consumed alcohol rated
the people in the photographs as more attractive
than the student who did not consume alcohol.
Source: http://www.theage.com.au/articles/2002/09/091031115991721.html