The document discusses issues in experimental design for studying atypical language development. It covers the differences between controlled experiments, field experiments, and natural experiments. Controlled experiments allow for precise control but lack generalizability, while field and natural experiments are more reflective of real life but have less control. The document also addresses within-subject repeated measures designs versus between-group independent measures designs. It notes the importance of counterbalancing to control for order effects. Additional topics covered include matching groups on relevant variables, checking the distribution of data to select appropriate statistical tests, and keeping thorough lab notes.

1. 08/10/2013
Issues in experimental
design for the study of
atypical language
development
David Saldaña
DESIGNING YOUR STUDY: CONTROL
1

2. 08/10/2013
Controlled experiments
This type of experiment is conducted in a well-controlled environment –
not necessarily a laboratory – and therefore accurate measurements are
possible.
Easier to replicate
Precise control of
extraneous and
independent
variables.
The artificiality of
the setting and lack
of generalization
Demand
characteristics or
experimenter effects
may bias the results
McLeod, S. A. (2012).
Field experiments
The experimenter still manipulates the independent variable, but in a reallife setting (so cannot really control extraneous variables).
More likely to reflect real life
because of it natural setting
Less control over extraneous
variables
Less likelihood of demand
characteristics affecting the
results
Less replicable
Can be used in situations in
which it would be ethically
unacceptable to manipulate
the independent variable, e.g.
researching stress.
McLeod, S. A. (2012).
2

3. 08/10/2013
Natural experiments
Natural Experiments are conducted in the everyday environment of the participants
but here the experimenter has no control over the IV as it occurs naturally in real
life.
More likely to reflect real life because No control over extraneous variables
of it natural setting
Less replicable
Less likelihood of demand
Possible more expensive
characteristics affecting the results
Can be used in situations in which it
would be ethically unacceptable to
manipulate the independent variable,
e.g. researching stress.
Less likelihood of demand
characteristics affecting the results,
as participants may not know they
are being studied.
McLeod, S. A. (2012).
A perhaps “silly” research question
Does vocabulary
influence the
comprehension of
questions more than the
comprehension of
imperatives?
3

4. 08/10/2013
A question that is a little more complicated
Do children with SLI
understand questions
better than typically
developing children?
Do children with SLI
understand questions
better than imperatives?
DESIGNING YOUR STUDY:
CONDITIONS
4

5. 08/10/2013
How is your IV distributed?
Independent Measures
Repeated Measures
Matched Pairs
Independent measures (between-group)
• Each group gets one condition
• Different participants in each
group
• Avoids practice
• More people needed
• Participant variables could affect
results.
5

6. 08/10/2013
Repeated measures (within-group)
Task 1
Task 2
• Measurement
• Measurement
- Less participants
- Precision determined by variation within same
subject
- May be the only design that answers the
questions of interest.
- Order effects: practice and fatigue effect
Counterbalancing
Moment 1
Moment 2
A
B
B
A
6

7. 08/10/2013
Back to our “silly” study
Lets design the tasks and
the data collection
Does vocabulary
influence the
comprehension of
questions more than the
comprehension of
imperatives?
A little bit more interesting (perhaps)
research question
Does vocabulary
influence the
comprehension of
questions more than the
comprehension of
imperatives in SLI?
7

8. 08/10/2013
Mixed-design
Group 1
Task 1
Task 2
• Measurement
• Measurement
Group 2
Task 1
Task 2
• Measurement
• Measurement
Some imaginative results
SLI
Typical
120
100
Standard Scores
100
99
95
92
90
80
77
60
70
73
90
90
90
75
40
20
0
IQ
Vocabulary
Comprehension
Questions
Word Reading
SES
Comprehension
Imperatives
8

9. 08/10/2013
MATCHING
Matching your groups: why?
SLI
Typical
120
100
Standard Scores
100
99
95
92
90
80
77
60
70
73
90
90
90
75
40
20
0
IQ
Vocabulary
Comprehension
Questions
Word Reading
SES
Comprehension
Imperatives
9

10. 08/10/2013
Matching your groups: on what?
6
8 10
2
4
6
8
2
4
6
8
Imp.Comp
6
8 10
IQ
8
5
10 15
QuestComp
4
6
Voc
8 10
2
4
6
SES
2
4
6
8
5
10 15
4
6
8 10
Your groups: three group design
• Group of interest
• Control group, matched on chronological age
• Control group, matched on variable of interest
10

11. 08/10/2013
Some issues
• Match on variables of interest to
the study
• Match average and distribution
• Study carefully your exclusions
Recruit clinical
sample
Decide
variables of
matching
Recruit age
controls
Recruit “level”
controls
Let’s do it! Matching exercise
Some issues
Recruit clinical
sample
• Match on variables of interest to
the study
• Match average and distribution
• Study carefully your exclusions
Decide
variables of
matching
Recruit age
controls
Recruit “level”
controls
Check and
report
11

12. 08/10/2013
What statistical test?
An ideal world
Facon et al. (2011)
What statistical test?
An ideal world
12

13. 08/10/2013
What statistical test?
An ideal world
The real world
Non-normal distributions
Different range and variance
Different distributions
What statistical test?
None: look at the picture first!
Back-to-back histogram
Back-to-back stem-and-leaf
13

14. 08/10/2013
What statistical test?
What statistical test?
Mann-Whitney?
Two-sample Kolmogorov-Smirnov test
For small sample sizes: exact version
• Does not assume prior “shape” in distributions
• Tests for the differences in distribution
14

15. 08/10/2013
Matching your groups: how much?
The easy-peasy way
Matching your groups: how much?
A brief recall of significance testing
I say
No Match
I say
Match
Match
No Match
http://intuitor.com/statistics/T1T2Errors.html
15

16. 08/10/2013
“Casual acceptance of the null hypothesis”
(Harcum, 1990)
< .2 too low
Frick (1995)
.2 to .5 ambiguous
> .5 fine
http://intuitor.com/statistics/T1T2Errors.html
Some things to remember
• Population comparisons are group comparisons
(Paradis, 2010)
16

17. 08/10/2013
Some things to remember
• Each group must be there for a reason
(Paradis, 2010)
An example: why each group?
Two groups of deaf adolescents
SAL
16 deaf (13-21) good oral language
SBL
16 deaf (13-21) poor oral language
Three groups of hearing adolescents
OCS
20 adolescents (13-21) task with sound
OSS
20 adolescents (13-21) task with no sound
NO
20 children (6-11) oral language equivalent to SAL
Torres (2013)
17

18. 08/10/2013
Some things to remember
• Check matching while you are recruiting
(Paradis, 2010)
Keep your lab notes up to date
•Some important
decisions to record:
•Changes and
discussions on criteria
•Subjects in – subjects
out.
•Transcriptions and
criteria if using MLU
(Paradis, 2010)
18