Debriefing in online experiment


Published on

The importance of Debriefing in online experiment. Delivered by Harryadin Mahardika at Universitas Indonesia

Published in: Education, Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Debriefing in online experiment

  1. 1. Debriefing in Online Experiment Harryadin Mahardika, PhD
  2. 2. Harryadin Mahardika • Pop Economist • FEUI & • Research objective: – “to liberate and empower consumer...” • Current research: – Consumer empowerment – Consumer intervention/engineering – Mobile advertising • Contact: – / – @HarrySastro 2
  3. 3. Discussion Agenda • Informed consent • Debriefing • Randomization • Participant incentives
  4. 4. Risk of experiment on human subject • Deception, manipulation, priming, scenario. – May harm participants’ mind. – Mental fatigue. • Examples?
  5. 5. Informed consent • Information about the experiment and its risk. – As a basis for participants to decide after they understand what the research involves (risks and benefits ). • Written consent vs unwritten consent. – Written: • Investigators must typically obtain and document voluntary informed consent from research subjects. – Unwritten: • button on an online form to indicate they have read and understood the consent form.
  6. 6. Informed consent in online experiment • Limited interaction with participants – investigator often cannot tell whether a subject understood the informed consent statement. • Online form: – Researchers can increase the likelihood that subjects are granting truly informed consent by requiring feedback from subjects about their level of understanding, • Example: – by requiring a “click to accept” for each element in an informed consent statement or even administering short quizzes to establish that a subject understood. • Reduce response rate: – Increase nonresponse to sensitive items (Singer, 1978) – Possibly produce biased data (Trice, 1987).
  7. 7. Risk in online experiment • It exposes subjects to innocuous questions and benign or transient experiences with little lasting impact. • In general, online experiments is no more risky than any of their offline counterparts. • In some respects, they may be less risky: – The reduced social pressure (Sproull & Kiesler, 1991) in online surveys or experiments makes it easier for subjects to quit whenever they feel discomfort. – This freedom to withdraw is no trivial benefit, given the strong pressures to continue in face-to-face studies (e.g., Milgram, 1963) and even telephone calls.
  8. 8. Risk in online experiment • Although risk in online settings is typically low, the actual risk depends on the specifics of the study. • For example: – Some questions in a survey or feedback from an experiment may cause subjects to reflect on unpleasant experiences or to learn something unpleasant about themselves – e.g., Nosek et al.’s, 2002b, research on automatic stereotyping.
  9. 9. Risk in online experiment • Experiments that deliberately manipulate a: – subject’s sense of self-worth, – reveal a lack of cognitive ability, – challenge deeply held beliefs or attitudes, or – disclose some other real or perceived characteristic ..... may result in mental or emotional harm to some subjects.
  10. 10. Debriefing • American Psychological Association (2002) ethical guidelines call for debriefing subjects: – “Providing an explanation of the nature, results, and conclusions of the research—as soon after their participation as practical.“ • If deception was involved: – Researcher needs to explain the value of the research results and why deception was necessary.
  11. 11. Debriefing in online experiment • When conducting research online: – Researchers can post debriefing materials at a Web site, – Provide debriefing materials to those who leave before completing the research (Nosek, Banaji, & Greenwald, 2002a). – For example, researchers can deliver debriefing material through a link to a “leave the study” button or through a pop-up window, which executes when a subject leaves a defined Web.
  12. 12. Debriefing in online experiment • Appropriate debriefing in online research may be difficult: – The absence of a researcher in the online setting makes it difficult to assess a subject’s state. – Difficult to determine whether an individual has been upset by an experimental procedure or understands feedback received.
  13. 13. Extraneous Variables cont. Selection Bias Improper assignment of test units to treatment conditions [sampling error]
  14. 14. Extraneous Variables cont. Mortality Loss of test units while the experiment is in progress [respondents selected no longer wish to participate]
  15. 15. Controlling Extraneous Variables Randomisation  Randomly assigning test units to experimental groups by using random numbers Matching  Comparing test units on a set of key background variables before assigning them to the treatment Statistical Control  Measuring the extraneous variables and adjusting for their effects through statistical analysis Design Control  Use of experiments designed to control specific extraneous variables
  16. 16. Case: Beauty is in the eye of the beholder Scientists have found a link between drinking alcohol and perceptions of beauty 80 students were shown colour photographs of 120 male and female students and were asked to rate the aesthetic properties on a 7-point scale from high unattractive to highly attractive Half the students had drunk up to four units of alcohol, the other half had no alcohol. The students who had consumed alcohol rated the people in the photographs as more attractive than the student who did not consume alcohol. Source:
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.