This document discusses quantitative research design and methods. It covers topics such as credibility of research design, sources of variability, design validity, designing quantitative research studies, population and sampling methods, test validity and reliability, threats to internal validity, and subject effects. The overall goal of research design is to provide trustworthy results that approximate reality by controlling sources of error and extraneous variables. Key aspects of design include subjects, instruments, procedures, and treatment administration. Probability and nonprobability sampling techniques are explained.

1. Quantitative Research Designs
and Methods
Notes from
McMillan & Schumacher
EDUU600

2. Credibility of Research Design
 Goal of sound research design
 Provide results that approximate reality
 Judged trustworthy and reasonable
 Eliminate or reduce sources of error
 Extraneous effects need to be controlled

3. Sources of Variability
Observations take on different values
 Systematic
 Two variables have high variability
Maximize
 Error
 Sampling and Measurement Errors
Minimize

4. Design Validity
Degree to which scientific explanations of
phenomena match the realities of the world.
 Internal Validity
 Extent to which extraneous variables have been
controlled and accounted for
 External Validity
 Generalizability of the results
 The extent to which the results and conclusions
can be generalized to other people and other
settings

5. Designing Quantitative Research
 Who will be assessed? – SUBJECTS
 What will they be assessed by –
INSTRUMENTS
 How will they be assessed –
PROCEDURES FOR DATA
COLLECTION
 In Experimental Designs – How are
treatments administered?

6. Plausible Rival Hypotheses
Is there anything that has
occurred or was done that
could provide an explanation
that is in addition to the
stated hypothesis or intent of
the research?

7. Possible Sources of Error
 Does the researcher have an existing bias about
subjects or topic researched?
 Are the subjects aware that they are being studied?
 Are the subjects responding honestly?
 Did both groups receive treatment as described?
 Does the sex of the interviewer matter?
 Did very many subjects drop out before the end of
the study?
 Did the time of day the research was done affect the
results?

8. Population
 Group of elements or cases (individuals,
objects, events) that conform to specific
criteria
 Group to which we intend to generalize the
results of the research
 Target population or Universe

9. Sample of Population-
Probability Sampling
 Subjects drawn from a larger population
 Estimate what is true from a smaller sample
of the general population
 Representative of the population
 Unbiased random sample
 Each member of the population has the
same chance of being selected as other
members of the group
See page 178-179

10. Simple Random Sampling
 Drawing names out of a hat
 Table of random numbers – set
of randomly assorted digits
 Each person randomly
assigned a number from 001
to ?
 Computer software random
selection

11. Systematic Sampling
 Every nth element selected from
list
 Weakness in sampling technique
 when numbers arranged in
systematic pattern
 rank order,
 by school, age, scores, etc.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

12. Stratified Random Sampling
 Population divided into subgroups on basis of strata
(age, gender, level of education, etc.)
 Proportional sampling – according to number in
strata
 Nonproportional sampling – equal numbers between
strata
 Cluster Sampling
 Groups of individuals are identified from
population
 Subjects are drawn from this group

13. Nonprobability Sampling
 Convenience Sampling
 Accessible subjects
 Convenience group – classroom of students, members of
a group
 Purposeful Sampling
 Judgment sampling
 Population that will be informative or representative
about a topic
 Quota Sampling
 Select subjects on a basis of certain characteristics of the
population
 Profiles of groups identified and selected non-randomly

14. Sample Size
 Number of subjects in a study
 What size of sample will provide sufficient
data to answer research question?

15. Factors to take into consideration
 Type of research
 Correlational – minimum of 30
 Comparing Groups – at least 15
 Survey – At least 100
 Research hypothesis
 Small differences – large sample
 Relatively small – but perhaps significant
differences
 Financial Constraints

16. Other Considerations
 Importance of Results
 Exploratory – smaller sample size
 When major potential effect on people or $$
spent – need large enough sample to minimize
error
 Number of variables studied – more variables mean
larger sample
 Methods of data collection – accuracy or
consistency
 Larger the population, the smaller the percentage
needed

17. Test Validity
 Extent to which inferences and uses made on the
basis of scores from an instrument are reasonable and
appropriate
 Appropriateness of measure for specific inferences,
decisions, consequences or uses that results from
scores are generated
 Dependent on the purpose, population, situational
factors
 Mental Measurements reports technical information
about instrument validity in terms of generalization

18. Test Reliability
 Consistency of the measurement
 Extent to which scores are similar over
different forms of the same instrument
 Scores consistent over different occasions
of data collection
 Minimize the influence on the scores of
change or other variables unrelated to intent
of the measure

19. Internal Validity of Design
 Control over extraneous variables
 Controls for error
 Source of errors – “threats”
 Threats may invalidate the study’s findings
 Various Sources of Threats

20. Threats to Internal Validity
History
 Incidents and events that occur during the
research
 Can occur within the study when
something outside or unrelated happens
during the treatment in an experiment
 Outside the research setting

21. Threats
 Selection
 Systematic difference in groups
 Existing differences
 Volunteers – different characteristics
 Statistical Regression – tendency for very high or
low scores in pretest to regress to the mean on
posttest
 Pretesting – Test itself has impact on subjects
 Stimulate thought
 Change attitude toward topic

22. More Validity Threats
 Instrumentation
 The way the instrument is used to collect data
 The person who uses the instrument
 Fatigue, boredom, way of recording data
 Subject Attrition – mortality or loss of subjects
during course of study
 Maturation –
 Changes in subjects over a period of extended
time
 Fatigue, boredom, concentration, hunger, time of
day

23. Threats, continued
 Diffusion of Treatment - When control
group/experimental group becomes aware of
differences in treatment
 Experimenter Effects
 Deliberate and intentional effect on subjects
 Tone of voice, gender, race, educational level,
clothing, attitude of experimenter

24. More Threats to Internal Validity
 Treatment Replications
 Number of subjects in study is not the same as
the number of treatments
 Comparing 2 classes – n=2 but number of
students may be 60.
 Group assignment - but individual treatment for
each student
 Statistical Conclusion – “use of impressive-
sounding statistics does not guarantee valid
results” (pp. 192-193)

25. Subject Effects
Change behavior because they are being studied
 Hawthorne effect – increase positive behavior
because they know they are being studied
 John Henry effect – Group tried harder to compete
with other group (compensatory rivalry)
 Resentful demoralization – not selected for the
group with preferred treatment
 Novelty effect – React with increased motivation
because they are doing something new and different