This document provides an overview of quantitative research methods for educational research. It defines key terms like quantitative vs qualitative research, variables, hypotheses, and different types of validity. Common quantitative study designs are described such as experimental, quasi-experimental, and non-experimental designs. Issues of practicality in educational research like sample size, random assignment, and intervention strength are also discussed.

1. Virginia Commonwealth University
ADLT 673 –Teaching as Scholarship in Medical Education
Kelly Lockeman, PhD

2.  Describe when a quantitative study is appropriate.
 Differentiate types of validity and define common
threats to validity.
 Recognize common quantitative study designs in
educational research.
 Identify general issues of practicality in
educational research.

4. QUANTITATIVE
 Specific
 Closed
 Static
 Outcome-oriented
 Specific variables
 May have hypotheses
QUALITATIVE
 General
 Open
 Evolving
 Process-oriented
 No specific variables
 No hypotheses

5.  From the Greek… to put under, suppose
 hypo- (under) + tithenai (to put)
 “to put under the microscope” and analyze
 A tentative statement about the expected
relationship between two or more variables
 Can be directional or non-directional

6.  Is it stated in declarative form?
 Is it consistent with known facts, previous research,
and theory?
 Does it state the expected relationship between
two or more variables?
 Is it testable?
 Is it clear?
 Is it concise?

7. Program ObservationCauses
What you do What you see
Intervention orTreatment
IndependentVariable
DependentVariable
Your hypothesis…
alternative
cause
alternative
cause
alternative
cause
alternative
cause

9. Conclusion
Internal
Construct
External
Validity
There is a relationship between observed variables
(e.g., between your intervention and the outcomes)
The observed intervention and outcomes
(measures) reflect what you think
(intended) them to
The implemented intervention caused the observed
outcome(s)
The results generalize to other
persons, places, times
 In any study there are many assumptions/assertions.
 For each assertion, there may be many reasons you are
wrong.These are called “threats” to the validity
of the assertion.
 You establish greater validity in your
research when you “rule out” or
minimize the more plausible
“threats” to validity
or plausible
alternative
explanations
to your
assertions.

10.  ConstructValidity: Am I implementing what
I think I am implementing?Am I measuring
what I think I am measuring?
 Experimental Validity:
 InternalValidity – Did the treatment cause
the outcome?
 ExternalValidity – Can I expect to see the
same results in other samples?

11.  A concept, model, or schematic idea
 A construct is the global notion of the measure, such as:
▪ Student motivation
▪ Intelligence
▪ Student learning
▪ Student anxiety
 The specific method of measuring a construct is called
the operational definition.
 For any construct, researchers can choose many
possible operational definitions.

12.  Example: What is “learning”? (Operational definition)
 How do we measure learning? (Proxy measures)
 Common measures of learning:
▪ Assessment/test scores
▪ Demonstration of skill/competency
 Common data collection methods:
▪ Self report (i.e., test/quiz)
▪ Observation
▪ Record review
 Proxy: approximates the real thing

13.  Measure constructs directly. Use clear operational
definitions. For example, “learning” is not enjoyment
or perceived learning.
 Align assessments with learning objectives.
 Use established scales whenever possible. Don’t
reinvent the wheel.
 Know how to score the measure before collecting
data. Consider what is reasonable (e.g., rubrics,
training, interrater reliability, etc.).

14.  Two types of experimental validity:
1. InternalValidity – the extent to which the
independent variable, and not other extraneous
variables, produce the observed effect on the
dependent variable
2. ExternalValidity – the extent to which the
results are generalizable
 Each types of experimental validity can be
threatened by certain factors.

15. Diffusion ofTreatment the treatment is [inadvertently] given to the control group
Instrumentation
poor technical quality (validity, reliability) or changes in
instrumentation
Selection
groups that are not equal due to differences in the
participants in those groups (e.g., positive and negative
attitudes, high and low achievers)
History
extraneous events (e.g., the crash of the stock market, 9/11)
have an effect on the participants' performance on the
dependent variable
Maturation participants' maturation over the course of the study
Attrition differential loss of participants from groups
Testing the effect of having taken a pretest
Statistical Regression the natural movement of extreme scores toward the mean

16. 1. Subjects
▪ Representativeness of the sample in comparison to the population
▪ Consistency of the results across subgroups within the sample
▪ Personal characteristics of the subjects
▪ Subject's awareness of being involved in a study
2. Situations - characteristics of the setting (e.g., specific
environment, special situation, particular school, etc.)
3. Time - explanations can change over time
4. Treatments - specific way in which an experimental treatment
is conceptualized, operationalized, and administered
5. Measures
▪ Different instruments measure content or constructs differently
▪ Measures change across studies

17.  It is the inference that is valid or invalid, not the
measure.
 An instrument can be valid for one use but not
another.
 Validity is a matter of degree.
 Validity involves an overall evaluative judgment
based on evidence.

19.  Experimental
 Pre-experimental
 Quasi-experimental
 True experimental
 Non-Experimental
 Descriptive
 Comparative
 Correlational
 Ex post facto
 Causal-comparative

20. EXPERIMENTAL NONEXPERIMENTAL
 There is an intervention that
the researcher manipulates.
 Researcher has direct control
over the variables.
 Some say this is the only way
to truly determine cause.
 The researcher does not
manipulate anything.
 Researcher does not have
direct control.
 Researcher can only describe
variables and relationships.
What’s the difference?

21.  R – random selection or random assignment
 O – an observation
▪ e.g., test score, observation score, survey measure, etc.
 X – a treatment or intervention
 A, B, C, ... – different groups

22.  Pre-experimental designs do not control threats
to internal validity very well.
 One-group, posttest only
▪ A X O
 One-group, pretest-posttest
▪ A O X O
 Two groups (non-equivalent), posttest only
▪ A X O B O
Key
R = Random
O = Observation
X =Treatment
A, B, C,… = Groups

23. Don’t Use
 If you want to make statement about causality
 If you want to make a comparison to another group
Use
 When your focus is to describe a treatment and not assessment.
 When you can’t have a pretest or a control group
 When you have a single group of students that cannot be divided
A X O
R = Random O = Observation X =Treatment A, B, C,… = Groups

24. Don’t Use
 If activities other than the treatment occur between assessments
 If the first assessment affects the second
 If participants are likely to change between assessments with no
treatment
Use
 When you have a small sample
 When you have single group that cannot be divided
 When you cannot have a control condition
A O X O
R = Random O = Observation X =Treatment A, B, C,… = Groups

25. Don’t Use
 If you have a small sample
 If the groups are very different
 If you have different assessments for each condition
Use
 If you are concerned about carryover effects
 If you are concerned about testing and instrumentation effects
 If you have multiple groups
 If you have only one session to collect data
A X O B O
R = Random O = Observation X =Treatment A, B, C,… = Groups

26.  Quasi-experimental designs do not control
threats to internal validity very well.
 Two-group pretest-posttest, experimental and control
groups
▪ A O X O B O O
 Two-group pretest-posttest, multiple treatment
groups
▪ A O X1 O B O X2 O
Key
R = Random
O = Observation
X =Treatment
A, B, C,… = Groups

27. Don’t Use
 If you have single group of students that cannot be divided
Use
 If you have multiple groups
 Use random assignment to improve internal validity
(1) A O X O B O O
(2) A O X1 O B O X2 O
R = Random O = Observation X =Treatment A, B, C,… = Groups

28.  Important components
 Random assignment
▪ Participants are placed into groups using a random procedure
▪ This ensures equivalency of the groups
 Random selection of subjects
▪ Participants are chosen from a population using random procedures
▪ This ensures generalizability to the population from which the
participants were selected (i.e., external validity)
 Effect on threats to internal validity
 Controls for selection, maturation, and statistical regression
 Likely to control for most other threats

29.  Types
 Randomized posttest only, experimental control groups
▪ R A X O R B O
 Randomized posttest only, multiple treatment groups
▪ R A X1 O R B X2 O
 Randomized pretest-posttest, experimental control groups
▪ R A O X O R B O O
 Randomized pretest-posttest,
multiple treatment groups
▪ R A O X1 O R B O X2 O
Key
R = Random
O = Observation
X =Treatment
A, B, C,… = Groups

30. Descriptive Designs Comparative Designs
Describe something using descriptive
statistics (frequencies, averages, graphs,
etc.). Usually used in the early stages of
research on a topic.
Compare the dependent variable between
two or more groups of participants.The
groups are the “levels” of the independent
variable.
Correlational Designs Predictive Designs
Show how two variables (in the same
participant) are related using a correlation
coefficient.Often difficult to discern the IV
or DV (unless they mistakenly infer cause!).
A variation of the correlational design. One
or more IVs (predictors) are used to predict
the DV through statistical correlation. More
than one predictor = multiple regression.
Causal-comparative Designs Ex post facto Designs
There is a naturally-occurring intervention
where the researcher has no control over
the conditions, but outcomes are
compared.
Analysis of data collected in the past.
Typically tries to makes causal inferences.A
special case of causal-comparative design.
Can also involve correlation/prediction.

31.  Figure out what you’re talking about!
 Define a concept
 Operationalize a definition with
measures
 Clarify ways to measure a concept
 Identify variables to include in a later
correlational study
 Determine items to include in a later
survey with a generalizable sample
Statistics like frequencies,
averages, and graphs.
Early stages of the
investigation of an area
The “building block” of all
quantitative research.

32.  You want to know whether there
are differences between currently
existing…
 Groups of people
 Situations
 Locations
 More on comparing two groups
“with greater confidence” when
we discuss experimental and
quasi-experimental research
designs next week.
Are they different?
(statistical significance)
How different are they?
(practical significance
or effect size)

33.  You want to know how variables are related among a
single group of participants.
 You might simply have two variables with no true IV or DV
(bivariate correlation) .
 Or there may be a predictor/IV
(if one of the conditions naturally
happens first) and an outcome/DV.
 Or there could be many predictors/
IVs that are related to an outcome/
DV, and you want to know which ones have the most
influence (multiple correlation/ regression).

34. What could be responsible for this
relationship? Some possibilities…
 Stress associated with larger cities
encourages people to drink more; larger
cities will have more religious leaders.
 Larger cities may have more efficient
distribution centers, and therefore the
price of beer may be lower. People may
buy more beer at lower prices.
 Alcohol consumption may encourage
more crime or depression. More religious
leaders may move to the area to address
the problem.
With correlation analysis, the relationship may be a causal relationship (independent
and dependent variable) or a non-causal relationship (variable 1 and variable 2).

35. Don’t Use
 If you want to make statement about causality
 If you have a small number of students
Use
 If you have single group of students that cannot be divided
 If you have only one session in which to collect data
 If you want to correlate many variables at the same time

36.  You can’t manipulate everything. Some
experiments are impractical or unethical.
 Some interventions occur naturally, and you want
to know whether they make a difference.
 You only know who the participants are,
or you only have access to them after
the fact.
 Historical events
Determine “causation” after the fact
by comparing two groups
with different preexisting conditions.

37. Special considerations when using
surveys in nonexperimental
research:
 Best practices in survey design,
e.g., TheTailored Design Method
(Dillman, 2009)
 Administration: Paper/Mail vs.
Online /Internet
 Cross-sectional vs. Longitudinal
With unique
opportunities and
challenges, surveys
have become one of the
most popular forms of
data collection in
nonexperimental
research.

38.  Trying to measure everything
 Small number of students = low statistical power
 Only a single class or group; limits type of design
 Difficulties in random assignment
 Difficulties in determining whether the treatment is
potent enough to have an effect (relates to power)
 Conducting an ethical study in a classroom or training
situation

39.  Each design has advantages and disadvantages.
 Often, there is no clear right way, although some
designs will be better than others.
 There is no single ideal study that eliminates all
potential problems and all alternative hypotheses.
 One study cannot answer all of your questions!