3. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
3
Conflicting paradigms about reality
Ontology
Epistemology
Human Nature
Methodology
Nature/essence of reality/Phenomenon
What is true/authentic? How to reach the truth?
What is it? Relationship with environment
How to explore reality?
Created, Product of
consciousness
Objectivist
Subjectivist
Personal, Subjective,
unique
Positivism: Hard, tangible,
empirically verifiable, Obj.
Voluntarism: act with free
will and creativity, produce
own environment
Determinism: Puppet,
responding mechanically
Reality Hard, real, external.
universal, Surveys, Exp.
Realism: External. Objective
independent, imposed
Soft, Personal, humanly
created, Participant
observation, accounts
4. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
4
Contrasting Paradigms:
Positivist vs. Naturalist
Reality is single, tangible,
fragmentable, and
measureable
Knower and known are
independent – objectivity
Time- and context-free
generalizations are possible
– Big “T” Truth, reliability,
internal and external validity
There are real causes,
precedent to or
simultaneous with their
effects
Inquiry is value-free
Realities are multiple,
socially constructed, and
holistic
Knower and known are
interactive and inseparable
Only time- and context-
bound generalizations are
possible – truths are
idiographic, credibility,
transferability, dependability,
and confirmability
Impossible to distinguish
causes from effects
Inquiry is value-laden
7. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
7
Common Designs in
Educational Research
Quantitative—involves use of numerical
indices to summarize, describe and
explore relationships among traits—
reliance on control, statistics,
measurements, and experiments
Qualitative—emphasis is on conducting
studies in natural settings using mostly
verbal descriptions, resulting in stories and
case studies, not statistical reports
8. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
8
Quantitative vs Qualitative
Methodologies
Quantitative
• Validation of facts,
estimates, predications,
relationships
• Descriptive and causal
designs
• Mostly structured
• Good representation of
target populations
• Statistical, descriptive,
causal predictions,
relationships analyses
possible
Qualitative
• Discovery of ideas,
feelings, preliminary
insights and understanding
of ideas
• Exploratory
• Open-ended, semi-
structured or unstructured
• Short time frames
• Small samples, limited
generalizability
• Subjective and content
analysis
9. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
9
Characteristics of a Quantitative
Research Plan
A quantitative plan will…
• state the hypothesis,
• determine the participants,
• select measuring instruments,
• choose a specific research design,
• specify procedures to conduct the study, and
• stipulate the statistical techniques.
10. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
10
Characteristics of a Qualitative
Research Plan
A qualitative plan will…
• identify the general research issue,
• explain how the researcher intends to gain entry
to the research site,
• identify the participants,
• estimate the time that will be spent in the field,
• determine the best ways to collect data, and
• identify appropriate ways to analyze the data.
12. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
13
Non-Experimental Designs
Four types of designs
• Descriptive
• Relationships
•Comparative
•Correlational
• Causal-comparative
• Ex Post Facto
• Survey
13. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
14
Descriptive Designs
Studies that describe a phenomena
• Statistical nature of the description
• Frequency
• Percentages
• Averages
• Graphs
Importance of these designs in the early
stages of the investigation of an area
14. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
15
Using Surveys for Descriptive
Research
A data collection method that is very
useful in descriptive and correlational
studies
• Versatile
• Efficient
• Generalizable
Two types of survey designs
• Cross sectional designs
• Longitudinal designs
15. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
16
Using Surveys
Cross sectional designs
• Information is collected from one or more groups at the
same time
• Examples
• Student’s, teacher’s, administrator’s, and parent’s opinions
regarding an extended school year
• Elementary, middle, and secondary teachers’ feelings toward
a new school board policy
• Issue of concern - comparisons across groups can be the
result of differences between participants within the groups
• Fifth and seventh graders opinions can be affected by a
changes in the attendance zones of a school
16. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
17
Using Surveys
Longitudinal designs - information is
collected from the same participants over
time
• Example - changes in the academic self-concept
of students from the sixth to the twelfth grade
• Issues of concern
• Loss of subjects over time
• Difficulty tracking participants over time
17. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
18
Using Surveys
Steps in designing a survey
• Define a purpose and objectives
• Identify the resources needed and the target population
• Cost of preparation, printing, mailing, and analyzing results
• Time needed to administer the survey
• Sample size
• Choose the method
• Paper
• Electronic
• Telephone
• Interview
18. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
19
Using Surveys
Steps in designing a survey
• Develop the items – guidelines
• Use clear, unbiased, non-ambiguous language
• Keep it short and simple
• Use grammatically correct language
• Do not write leading items
• Use the same response scale for all items
• Be consistent with wording
• Design the format of the survey
• White space
• Font size
19. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
20
Using Surveys
Steps in designing a survey
• Develop directions
• Make it clear with no ambiguity
• Indicate clearly how participants are to respond
• Indicate where responses are recorded
• Indicate what participants should do when finished
• Develop a letter of transmittal
• Keep it brief
• Include a statement of the purpose of the research
• Include a statement of the benefits of the research
20. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
21
Using Surveys
Steps in designing a survey
• Pilot test
• 15-20 representative participants
• Identify concerns
• Clarity
• Format
• Responding
• Directions
• Time to complete
21. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
22
Using Surveys
Response rates
• Low response rates are the major limitation of survey use
• Suggestions for increasing response rates
• Design the survey well
• Contact the subjects several times especially following-up
on non-respondents
• Include a self-addressed return envelope
• Use a good transmittal letter
• Use a telephone for follow-up
• Use incentives for completing the survey
22. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
23
Relationship Designs
Two types of designs
•Comparative
•Correlational
23. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
24
Comparative Designs
These designs investigate the relationship of
one variable to another by examining
differences on the dependent variable between
two groups of participants
• If math scores for males are significantly higher than
those for females, a relationship exists between
gender and math achievement
• If the academic self-concept scores for ninth graders
are significantly different than those for twelfth
graders, a relationship exists between grade level and
academic self-concept
24. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
25
Correlational Designs
Simple correlation designs
• Designs that examine the relationship between two
variables
• Two variables
• Predictor and criterion
• Use caution describing the variables as independent and
dependent
• Examples
• Math achievement and math attitudes
• Teacher effectiveness and teacher efficacy
25. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
26
Correlational Designs
Prediction designs
• Designs that examine the predictive nature of
the relationships between variables
• Two types of designs
• Simple prediction
• Multiple regression
26. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
27
Correlational Designs
Prediction designs
• Simple predictive studies
• Performance on one variable (i.e., the predictor) is used
to predict performance on a second variable (i.e., the
outcome or criterion)
• Examples
• Scholastic Aptitude Test (SAT) scores are used to predict
college freshmen grade point averages
• Scores from a mathematical attitude scale are used to
predict math achievement scores
• Importance of the time interval between collecting the
predictor and criterion variable data
27. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
28
Causal Comparative Designs
Ex-post-facto designs
• Studies that investigate the relationships between
independent and dependent variables in situations
where it is impossible or unethical to manipulate the
independent variable
• Example - what is the effect of pre-kindergarten (Pre-K)
attendance on first grade achievement
• Cannot mandate Pre-K attendance for children
• Characteristics and resources of families who do and do
not send their children to Pre-K may influence first grade
achievement
• Similarities with correlational and experimental
research designs
28. Wednesday, July 20,
2022
Center for Educational Research
and Assessment (CERA)
29
Causal Comparative Designs
Ex-post-facto designs
• Issues of concern
• Selecting participants who are as similar as possible
on all characteristics except the independent variable
• Generalizing beyond the participants studied
