1. Controlled Experiments
Tomohiro Nagashima, CC BY-NC-SA
Interactive Data Science; HCII; Fall 2018
05-839: Interactive Data Science

2. A bit about me
4/4/2019
2
• Tomohiro Nagashima
• Second-year PhD student in HCII
• Learning sciences, instructional design,
Open educational resources, learning
analytics

3. The tragedy
4/4/2019
3
• https://tinyurl.com/yakdc25t

4. Learning objectives
4/4/2019
4
• Describe what an experiment is
• Differentiate between experiments and
observational studies
• Identify different types of experiments

5. Interactive class?
4/4/2019
5
• Poll Everywhere
• https://pollev.com/tnagashima649

6. Interactive class?
4/4/2019
6
https://pollev.com/tnagashima649
Do you like data science?

7. 10/30/18
7

8. Research
4/4/2019
8
Lady scientist doing experiment vector clipart. CC0. https://www.goodfreephotos.com/vector-images/lady-scientist-doing-experiment-vector-clipart.png.php

9. Quantitative vs. Qualitative Research
4/4/2019
McGill University. https://www.mcgill.ca/mqhrg/resources/what-difference-between-qualitative-and-quantitative-research
9
Quantitative research aims to test
hypotheses, examine cause and effect
relationships, and establish general principles
using numeric data.
Qualitative research aims to understand
phenomena that cannot be directly measured
with numbers. Research questions are usually
exploratory.

10. Qualitative research: example
4/4/2019
10
Miyake, N. (1986). Constructive interaction and the iterative process of understanding. Cognitive Science 10, 151-177.
https://onlinelibrary.wiley.com/doi/abs/10.1207/s15516709cog1002_2

11. 4/4/2019
11
Miyake, N. (1986). Constructive interaction and the iterative process of understanding. Cognitive Science 10, 151-177.
https://onlinelibrary.wiley.com/doi/abs/10.1207/s15516709cog1002_2

12. Qualitative research: example
4/4/2019
12
Miyake, N. (1986). Constructive interaction and the iterative process of understanding. Cognitive Science 10, 151-177.
https://onlinelibrary.wiley.com/doi/abs/10.1207/s15516709cog1002_2

13. Transforming qualitative data into quantitative
4/4/2019
13
Coding interview data

14. Experiment
4/4/2019
14
https://pollev.com/tnagashima649
Lady scientist doing experiment vector clipart. CC0. https://www.goodfreephotos.com/vector-images/lady-scientist-doing-experiment-vector-clipart.png.php

15. 10/30/18
15

16. What is an experiment?
Observational Studies: we observe individuals
and measure variables of interest but do not
attempt to influence the responses.
Experimental Studies: we deliberately impose
some treatment on individuals and we observe their
responses.
4/4/2019
Dalene Stangl, Experimental design concepts, Experimental Design for Behavioral and Social Sciences
16

17. Example of an observational study
Recruit participants for a study. Give them
journals to record hour by hour their
activities the following day, including when
they watch TV and when they consume
snacks. Determine if snack consumption is
higher during TV times.
4/4/2019
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/
17

18. Example of an experimental study
Recruit participants for a study. Half of the
individuals sit in a waiting room with snacks
available and a TV on. The other half sit in a
waiting room with snacks available and no
TV, just magazines. Researchers determine
whether people consume more snacks in
the TV setting.
4/4/2019
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/
18

19. Exercise 1
• https://pollev.com/tnagashima649
A delivery company that owns a fleet of trucks performs
a study to determine whether rotating tires at specific
intervals has any effect on the number of miles a set of
tires lasts.
Observational study or experiment?
4/4/2019
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/
19

20. 10/30/18
20

21. Exercise 2
• https://pollev.com/tnagashima649
Researchers compare the rates of autism for children
who did and did not receive the standard measles-
mumps-rubella vaccine, to see if the vaccine might be
responsible for autism in some children.
Observational study or experiment?
4/4/2019
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/
21

22. 10/30/18
22

23. Exercise 3
• https://pollev.com/tnagashima649
Researchers classified pregnant women as being
nondrinkers or light, moderate, or heavy drinkers; they
examine the weights of their children at regular age
intervals to see if alcohol during pregnancy results in
poor growth.
Observational study or experiment?
4/4/2019
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/
23

24. 10/30/18
24

25. Exercise 3
• https://pollev.com/tnagashima649
Researchers classified pregnant women as being
nondrinkers or light, moderate, or heavy drinkers;
they examine the weights of their children at regular
age intervals to see if alcohol during pregnancy results in
poor growth.
Observational study or experiment?
4/4/2019
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/
25

26. Independent and dependent variables
• Independent variable ( explanatory variable)
• A variable that claims to explain, predict, or affect the
response
• Dependent variable (response variable)
• Outcome of the study
4/4/2019
26Independent variable Dependent variable
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/

27. Independent and dependent variables
• Exercise 4
• https://pollev.com/tnagashima649
Are the smoking habits of a person (yes, no) related to the
person's gender? – Is the gender the independent or
dependent variable?
4/4/2019
27
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/

28. 10/30/18
28

29. Independent and dependent variables
• Exercise 5
• https://pollev.com/tnagashima649
Can you predict a person's favorite type of
music (classical, rock, jazz) based on his/her IQ level? -- Is
favorite type of music the independent or dependent
variable?
4/4/2019
29
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/

30. 10/30/18
30

31. True experiment
• Researcher-manipulated variable
• Independent (explanatory) variable and dependent (response)
variable
• Control condition and experimental condition
• no intervention, works as a baseline group
• Random assignment
• Each participant has an equal chance of being assigned to each
condition.
4/4/2019
Grand Canyon University, Types of experimental design: https://cirt.gcu.edu/research/developmentresources/research_ready/experimental/design_types
31

32. Types of study designs
• Pre-experimental
• No randomization, no control group
• Experimental
• Random assignment of participants, treatment and control
groups
• Quasi-experimental
• Alternative to experiments
• Ex-post facto
• After the “treatment”
4/4/2019
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/
32

33. Types of study designs
• Pre-experimental
• No randomization
• No control group
• No causal inference can be made
4/4/2019
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/
33

34. Types of study designs
• Experimental
• Random assignment of participants
• Treatment and control groups
4/4/2019
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/
34

35. Types of study designs
• Experimental – adding a pretest
4/4/2019
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/
35

36. Types of study designs
• Experimental – two or more independent variables
4/4/2019
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/
36

37. Types of study designs
• Experimental – two or more independent variables
4/4/2019
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/
37
Low-performing
students
High-performing
students
Discovery learning A B
Direct instruction C D

38. Types of study designs
• Quasi-experimental
• When it’s hard to do randomization
4/4/2019
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/
38

39. Types of study designs
• Ex-post facto
• After the treatment has already been “administered”
4/4/2019
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/
39

40. Exercise 6
4/4/2019
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/
40
• Pre-experimental
• No randomization, no control group
• Experimental
• Random assignment of participants, treatment and control
groups
• Quasi-experimental
• Alternative to experiments
• Ex-post facto
• After the “treatment”
• https://pollev.com/tnagashima649

41. 10/30/18
41

42. Exercise 7
4/4/2019
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/
42
• Pre-experimental
• No randomization, no control group
• Experimental
• Random assignment of participants, treatment and control
groups
• Quasi-experimental
• Alternative to experiments
• Ex-post facto
• After the “treatment”
• https://pollev.com/tnagashima649

43. 10/30/18
43

44. Study design activity
4/4/2019
OLI at Stanford University, CC-BY-NC-SA: http://oli.stanford.edu/
44
https://tinyurl.com/ydbk3y29