The document summarizes a workshop on designing curriculum evaluations. It provides templates and examples for developing evaluation plans with key components: evaluation questions, indicators of success, data sources, data collection methods, and data analysis strategies. Participants will use the templates to design an evaluation plan for assessing the implementation and impact of a reading program in their schools. The workshop covers qualitative and quantitative evaluation approaches to determine if program elements are being delivered as intended, whether teaching and student outcomes have improved, and the degree to which program goals are being met.

1. IEFE 2013 Workshop
Designing a Curriculum
Evaluation
Dr. Dorothy Harnish

2. Workshop Outline
— Overview of Evaluation Design Components
— Hierarchy of evidence model
— Template for developing an evaluation plan
— Complete sample plan
◦ Identify the evaluation questions
◦ Identify indicators or evidence desired
◦ Describe sources of information
◦ Describe types of data from each source
◦ Describe methods of collecting information
◦ Describe how data will be analyzed
— Participants design an evaluation plan to use in their school
— Share several examples with group
— Questions, concerns, issues related to the evaluation process

3. Evalua&on
Ques&ons
Evidence/
Indicators
Sources
Type
of
data
Data
collec&on
Analysis

4. Evaluation Questions
Ø Is the program being implemented as it was designed/
intended? Have each of the key elements of the
program been put in place? If not, why not?
Ø What changes have occurred in teaching methods,
student behaviors, learning outcomes, or school
processes? What evidence exists for this?
Ø Are those implementing the program (e.g., teachers,
administrators) sufficiently trained to use the new
methods, materials, or equipment?

5. Evaluation Questions
Ø Are resources (e.g., materials, technology, time, staffing)
sufficient to implement all parts of the program?
Ø How satisfied are parents, students, teachers, and
administrators, with the program and results? What
problems have been identified by stakeholder groups?
Ø What changes are needed to improve the program?
Ø Do the results justify continued expenditure of funds
on this program or expansion to other sites?

6. Information Sources
— People
— Products
— Processes
— Documents/Materials
— Importance of “triangulation” through
multiple sources of information

7. Data Collection
Ø Methods determined by evaluation purpose
and questions
Ø Qualitative and quantitative methods
§ on-site field observations
§ individual interviews
§ focus groups
§ questionnaires
§ document or program materials review
§ student test or performance data

8. Data Analysis & Reporting
— Experimental, quasi-experimental,
longitudinal, and qualitative analyses
— Pre-post analyses to assess growth and
impact
— Comparison groups
— Randomization to control extraneous
variables
— Generalization of findings
— Difficulty of cause-effect conclusions

10. Evalua&on
Ques&ons
Evidence/Indicators
Sources
Type
of
data
Data
collec&on
Analysis
1. How
well
is
the
Reading
Excellence
program
being
implemented
in
elementary
classes?
(process
evalua>on,
implementa>on)
-­‐ Are
all
components
of
the
curriculum
being
taught?
-­‐ Are
new
teaching
approaches
being
used
effec>vely?
-­‐ Are
new
materials
being
used
as
intended?
Materials
are
available
in
all
classes
Teachers
are
trained
to
use
methods
Teachers
understand
the
new
program
goals,
materials,
methods
Teachers
use
program
strategies
in
their
daily
teaching
Teachers
Students
Administrators
Curriculum
experts/trainers
Curriculum,
training
materials
Field
observa>ons
of
reading
teachers
Ra>ng
scale
of
program
implementa>on
criteria
used
in
observa>on
visits
to
classrooms
Interviews
with
teachers
Survey
of
reading
teachers,
students,
school
administrators,
curriculum
experts
Training
workshop
documenta>on
Classroom
visits
by
evaluator(s)
4-­‐6
>mes/year
Classroom
visits
by
supervisor
or
principal
2-­‐3
>mes/
month
Interviews
with
teachers
conducted
by
evaluators
mid-­‐
term,
end-­‐of-­‐course
Teacher,
student
,
administrator,
curriculum
expert
surveys
administered
at
end
of
course
Implementa>on
criteria
achieved
by
teachers,
as
documented
by
observa>on
interview,
and
survey
data
(components,
methods,
materials)
Comparison
of
survey
feedback
from
4
sources

11. Evalua&on
Ques&ons
Evidence/Indicators
Sources
Type
of
data
Data
collec&on
Analysis
2.
To
what
extent
does
student
reading
skill
improve
with
use
of
Reading
Excellence
methods?
(outcomes
evalua>on,
impact)
-­‐ Do
students
read
with
greater
fluency
and
understanding?
-­‐ Do
students
have
a
greater
vocabulary?
-­‐ Do
students
read
more
frequently
in
and
out
of
class?
Results
of
student
reading
assessments
show
posi>ve,
significant
changes
in
vocabulary,
fluency,
comprehension
In-­‐class
reading
performance
has
improved
Out-­‐of-­‐class
reading
ac>vi>es
are
more
frequent
Student
achievement
measures
Test
results/in-­‐
class
student
assessments
Teachers
Parents
Tests
of
achievement
in
vocabulary,
fluency,
comprehension
(criterion
referenced)
Teacher
assessments/
grading
of
in-­‐class
reading
performance
Teacher
interviews
Parent
survey
(observa>on
of
out-­‐
of-­‐class
reading
frequency,
ability,
enjoyment)
Pre-­‐test;
mid-­‐
term;
post-­‐test
administra>on
Interviews
with
teachers
conducted
by
evaluators
mid-­‐
term,
end-­‐of-­‐
course
Student
grades
(mid-­‐term,
end-­‐of-­‐
course)
Parent
surveys
(beginning,
end
of
school
year)
Comparison
of
pre-­‐
post
standardized
test
scores,
amount/
significance
of
change
%
of
students
achieving
end-­‐of-­‐
course
passing
grade
vs.
previous
year;
%
of
students
receiving
A
–
F
grades;
%
improving
grades
from
mid
to
end
of
year
Comparison
of
pre-­‐
post
student
reading
frequency
reported
by
parents