Coursework; analysis of experimental designs

1. Experimental Methods
Amelia W. Cole
October 2017
1

2. Experimental Methods
2
Experimental Quasi-Experimental
A Bvs

3. BOOKS TO READ
3
Experimental Design: From User Studies
to Psychophysics
Christian Wallraven
Douglas W. Cunningham
How to Design and Report
Experiments
Andy Field
Graham Hole

4. HISTORY OF EXPERIMENTAL METHOD IN PSYCHOLOGY
David Hume (1730s & 40s).
“conventional wisdom on cause and effect stems from David Hume’s ideas about causality.”
(1) Cause and effect occur close in time.
(2) The cause must occur before the effect.
(3) The effect should never occur without the presence of the cause.
John Stuart Mill (1865)
“The only way to infer causality is through comparison of two controlled situations.”
(1) Cause has to precede effect.
(2) Cause and effect should correlate.
(3) All other explanations must be ruled out.*
4
Field, A., & Hole, G. (2003). How to Design and Report Experiments. London, UK: SAGE Publications. p10-13.

5. VARIABLES
Independent Variable (IV)
• The variable you manipulate.
Dependent Variable (DV)
• The measured variable.
Confounding Variables
• Another factor(s) that may cause or influence a result.
5

6. AIMS OF EXPERIMENTAL RESEARCH
Reliability
• The study can be easily replicated
• Requires a precise measurement of the dependent variable.
• To generalize the results, it needs a large number of participants.
Validity
• Internal validity: the ability to claim that the results aren’t influenced by other factors.
• (e.g., no randomization, measurement instruments, maturation, etc…)
• External validity: representative of the entire population.
• (e.g., overuse of a special population)
6

7. 7
WHEN TO USE IT
Establish a causal relationship: You have complete
control over the independent variable, so you can
determine what is causing the change in the
dependent variable.
Access to a large number of people: This method
requires a lot of people to be successful.
Simplicity: The design itself is relatively simple.
POTENTIAL ISSUES
Difficulty randomizing: Randomization of people or
the order of conditions (or both!) helps rule out other
factors and random influences in a study.
Not enough participants: To generalize the
findings, a large number of participants are
required.
Bad research question: Experiments depend on a
well-written research question and series of
measurable hypotheses.
EXPERIMENTAL
A Bvs
!

8. QUASI-EXPERIMENTAL
8
WHEN TO USE IT
Studying subgroups: the quasi-independent
variables involves predefined groups.
Can't randomly assign participants: One or more
independent variables are predefined (male/female) or
cannot be assigned due to ethical issue (smoking/
smoking free)
POTENTIAL ISSUES
No causal relationship: The researcher does not
have complete control over the independent
variable with regard to timing of the IV, or participant
allocation to groups.
Inequivalent study group: The lack of randomly
assigned participants may create bias in the results.
!

9. RESEARCH QUESTIONS & HYPOTHESES
TOPIC: HACT+L
My topic investigates designing to support children (humans) living in poverty
(context), learning to self-regulate their emotions (activity), using an EEG headset and
simple mind-full games on an android tablet (tech), at the Nepal House Kaski school
in Nepal (location).
RESEARCH QUESTION:
Does 10 hours of Mind-full game play in one week increase children’s ability to self-
regulate their emotions?
9

10. 10
A B
B A
Randomly
allocated
Measurement
RESEARCH DESIGNS: REPEATED MEASURES

11. 11
A B
B A
Randomly
allocated
Measurement
RESEARCH DESIGNS: REPEATED MEASURES

12. 12
WHEN TO USE IT
Save time and money: Using the same people for
every condition reduces the sample size and other
costly expenses for research.
Reduces the noise of other factors: Measures the
effects of the independent variables better than any
other design.
POTENTIAL ISSUES
Carry-over effects: People transfer information from
one condition to the next, influencing the results.
Order of the conditions: All conditions in the
experiment must be able to occur in any order.
Fatigue and boredom: Hitting a hammer for five hours
in one condition and again in the other is boring!
!
RESEARCH DESIGNS: REPEATED MEASURES

13. RESEARCH DESIGNS: INDEPENDENT MEASURES
13
Two-group independent measures
Treatment
Control
Randomly
allocated
Measurement
Pre- and Post- Test
Measurement Measurement
Randomly
allocated
Treatment
Control

14. RESEARCH DESIGNS: INDEPENDENT MEASURES
14
WHEN TO USE IT
Keeps It Simple: A/B conditions.
Need to reduce task fatigue: If the participants are
likely to tire of the task easily or if they occur too
frequently.
Can randomly allocate participants: Random
assignment to the treatment and control groups is
beneficial.
POTENTIAL ISSUES
Expense: It's an expensive design and you may need
a grant and a long timeline.
Too many participants: For every condition, you need
about the same number of participants in each group,
doubling the amount of participants.
Random noise: Participants are not all equal and you
could end up with some random factors influencing the
results.
!

15. 15
Research Articles

16. How learning shapes the empathic brain.
16
Hein, G., Engelmann, J. B., Vollberg, M. C., & Tobler, P. N. (2016). How learning shapes the empathic brain. Proceedings of the National Academy of Sciences,
113(1), 80–85. https://doi.org/10.1073/pnas.1514539112
Hypothesized that positive associations
with in-group and out-group people result in
increased empathy.
In-Group Out-Group
HYPOTHESIS

17. How learning shapes the empathic brain.
17
Hein, G., Engelmann, J. B., Vollberg, M. C., & Tobler, P. N. (2016). How learning shapes the empathic brain. Proceedings of the National Academy of Sciences,
113(1), 80–85. https://doi.org/10.1073/pnas.1514539112
STUDY DESIGN
Pre-
intervention
IV1: Name
DV1: fMRI / neural activation related to empathy for pain
DV2: Likert-scaled question
Sessions
40
Males
Results
Intervention Post-
Intervention
Swiss
Name
Balkan
Name
Control
Treatment
Survey Groups

18. The “findings show that empathy with an out-group
member can be learned and generalizes to other out-
group individuals.”
The “study provides a mechanistic account of how
positive contacts with the out-group can counteract
empathy deficits.”
18
How learning shapes the empathic brain.
RESULTS
Hein, G., Engelmann, J. B., Vollberg, M. C., & Tobler, P. N. (2016). How learning shapes the empathic brain. Proceedings of the National Academy of Sciences,
113(1), 80–85. https://doi.org/10.1073/pnas.1514539112

19. 19
RESEARCH QUESTION
How does interface complexity impact
children’s touchscreen interactions and
how do they differ from adults?
Characterizing How Interface Complexity
Affects Children’s Touchscreen Interactions.
Woodward, J., Shaw, A., Luc, A., Craig, B., Das, J., Hall, P., Jr., … Anthony, L. (2016). Characterizing How Interface Complexity Affects Children’s Touchscreen Interactions. In
Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 1921–1933). New York, NY, USA: ACM. https://doi.org/10.1145/2858036.2858200

20. 20
Woodward, J., Shaw, A., Luc, A., Craig, B., Das, J., Hall, P., Jr., … Anthony, L. (2016). Characterizing How Interface Complexity Affects Children’s Touchscreen Interactions. In
Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 1921–1933). New York, NY, USA: ACM. https://doi.org/10.1145/2858036.2858200
STUDY DESIGN
IV1: Gestures vs. Touching
IV2: Task difficulty
DV1: System data (e.g., target misses, pressure, size, location, response time, etc…)
30
Adults
Results
Simple Tasks Complex Tasks
Touching
Targets
Drawing
Gestures
Touching
Targets
Drawing
Gestures
Pre-Survey
Characterizing How Interface Complexity
Affects Children’s Touchscreen Interactions.
30
Children

21. 21
RESULTS
Characterizing How Interface Complexity
Affects Children’s Touchscreen Interactions.
Onscreen feedback must be prominent and immediately
noticeable.
Avoid small objects near screen edges
Response time to animations remains an open question.
Younger children need specific gestures and time to learn;
and algorithms are needed to recognize children’s gestures.
Woodward, J., Shaw, A., Luc, A., Craig, B., Das, J., Hall, P., Jr., … Anthony, L. (2016). Characterizing How Interface Complexity Affects Children’s Touchscreen Interactions. In
Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 1921–1933). New York, NY, USA: ACM. https://doi.org/10.1145/2858036.2858200

22. Design an Experiment
We’re going to get into two groups and go to the whiteboard.
Using one of your own research areas, think of a research question and hypothesis. Write it
down (or use an existing one from a study or one provided here).
Design an experiment that might answer your research question.
22

23. SAMPLE RESEARCH QUESTIONS
23
How does interface complexity impact children’s touchscreen
interactions and how do they differ from adults?
Do positive associations with in-group and out-group people
result in increased empathy.
Does 10 hours of Mind-full game play in one week increase
children’s ability to self-regulate their emotions?

24. REFERENCES
Woodward, J., Shaw, A., Luc, A., Craig, B., Das, J., Hall, P., Jr., … Anthony, L. (2016). Characterizing How
Interface Complexity Affects Children’s Touchscreen Interactions. In Proceedings of the 2016 CHI Conference on
Human Factors in Computing Systems (pp. 1921–1933). New York, NY, USA: ACM. https://doi.org/
10.1145/2858036.2858200
Hein, G., Engelmann, J. B., Vollberg, M. C., & Tobler, P. N. (2016). How learning shapes the empathic brain.
Proceedings of the National Academy of Sciences, 113(1), 80–85. https://doi.org/10.1073/pnas.1514539112
Creswell, J. W. (2013). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (4th ed.).
Thousand Oaks, United States: SAGE Publications, Inc.
Cunningham, D. W., & Wallraven, C. (2011). Experimental Design: From User Studies to Psychophysics. CRC
Press.
Field, A., & Hole, G. (2003). How to Design and Report Experiments. London, UK: SAGE Publications.
Icons sourced from The Noun Project
Images sourced from Pixabay
24

25. SCIENTIFIC STATEMENTS
“A scientific statement can be confirmed or disconfirmed empirically.” - Field & Hole (2003)
EXAMPLES OF SCIENTIFIC STATEMENTS
1. There are five key dimensions to personality (Costa & MacRa, 1955)
2. Dogs can learn that a certain sound predicts being fed (Pavlov, 1927)
3. Social phobic people dwell on the bad aspects of social events more than non-anxious
individuals do (Clark & Wells, 1995)
EXAMPLES OF NON-SCIENTIFIC STATEMENTS
4.We sometimes behave in a way opposite to how we feel, but sometimes we don’t (Freud, 1901)
• Cannot be tested
5.Humans and apes are descended from a common ancestor (Darwin, 1859)
• Relies on corroborative evidence
25
Field, A., & Hole, G. (2003). How to Design and Report Experiments. London, UK: SAGE Publications. p.18