The document discusses a system developed by Amy Cassata and Jeanne Century for measuring the implementation of instructional materials in educational interventions aimed at improving student outcomes. It outlines a theory of implementation considering various contextual factors that influence how instructional materials are used by teachers and their subsequent impact on student achievement. The researchers conducted a study on the Everyday Mathematics curriculum across multiple districts, identifying key instructional practices and their relationship to student math achievement.

1. Measuring
educa-onal
interven-ons
in
context:
A
system
for
iden-fying
how
and
why
instruc-onal
materials
are
effec-ve
Amy
Cassata,
PhD
and
Jeanne
Century,
PhD
Flash
Presenta5on
at
the
Third
Biennial
Global
Implementa5on
Conference
Dublin,
Ireland,
May
29,
2015

2. Measuring
educa-onal
interven-ons
in
context:
A
system
for
iden-fying
how
and
why
instruc-onal
materials
are
effec-ve
Amy
Cassata,
PhD
and
Jeanne
Century,
PhD
Flash
Presenta5on
at
the
Third
Biennial
Global
Implementa5on
Conference
Dublin,
Ireland,
May
29,
2015

3. Educa&onal
interven&ons
are
curricula,
tools,
teaching
approaches,
programs,
or
prac5ces
designed
to
improve
student
outcomes.

4. Instruc&onal
materials
are
one
type
of
educa5onal
interven5on.
They
are
resources
designed
for
use
by
teachers
in
classrooms
to
guide
their
instruc5on.

5. To
help
guide
selec5on
decisions,
instruc5onal
materials
are
evaluated
based
on
evidence
of
their
effec5veness
in
improving
student
achievement.

6. The
evidence
is
organized
and
summarized
to
help
school
and
district
leaders
decide
which
instruc5onal
materials
to
adopt
in
their
seLng.

7. The
expecta5on
is
that
teachers’
use
of
these
materials
will
lead
to
large-­‐scale
changes
in
prac5ce
and
in
turn,
increased
levels
of
student
achievement.

8. Evidence-­‐based
Instruc5onal
materials
Large-­‐scale
changes
in
teacher
prac5ce
Improved
student
achievement
But
how
do
schools
and
districts
get
from
point
A,
to
point
B,
to
point
C?

9. D
Evidence-­‐based
Instruc5onal
materials
Large-­‐scale
changes
in
teacher
prac5ce
Improved
student
achievement
Con5nued
posi5ve
outcomes
TIME
And,
how
do
they
ensure
that
any
changes
in
teacher
prac5ce,
and
effects
on
student
outcomes,
will
be
sustained
over
5me?

10. For
the
past
several
years,
our
research
team
has
asked
these
same
ques5ons.

11. We
developed
a
theory
of
implementa5on
to
understand
and
measure
interven5ons
in
prac5ce,
the
factors
that
influence
their
use,
and
their
rela5onships
to
outcomes.

12. Our
theory
begins
by
clearly
describing
an
interven5on
and
its
essen5al
components.

13. We
then
consider
the
context
in
which
the
interven5on
is
used,
which
also
has
many
elements.
THE
CONTEXT
THE
CONTEXT

14. Characteris5cs
of
individual
end-­‐users
include
demographics,
disposi5ons,
and
percep5ons.
Percep5ons
Individual
beliefs
Mo5va5on
Self-­‐efficacy
Innova5veness
Resourcefulness
Experience
Demographics
THE
CONTEXT

15. Characteris5cs
of
the
organiza5on
include
people,
structures,
and
strategies.
THE
CONTEXT
THE
CONTEXT
Ongoing
improvement
structures
Organiza5onal
innova5veness
Time
Locus
of
decision
making
Clarity
of
organiza5onal
goals
Learning
opportuni5es
Leadership

16. Elements
of
the
environment
include
policies,
poli5cs,
and
community
characteris5cs.
THE
CONTEXT
Poli5cal
environment
Extraneous
events
Community
characteris5cs
Community
beliefs
Learning
opportuni5es
THE
CONTEXT

17. Networks
are
systems
of
interconnected
people
or
organiza5ons.
THE
CONTEXT
THE
CONTEXT
Network
structure
Member
rela5onships
Trust
Communica5on
Change
and
evolu5on

18. The
interven5on
is
enacted
in
a
context
over
5me,
which
means
that
elements
of
the
context
change.
THE
CONTEXT
THE
CONTEXT
THE
CONTEXT

19. And
in
turn,
interven5on
components
may
be
adapted
to
fit
the
changing
context.
THE
CONTEXT
THE
CONTEXT
THE
CONTEXT

20. All
of
these
elements
together
make
up
the
IMPLEMENTATION
PROCESS.
THE
CONTEXT
THE
CONTEXT
THE
CONTEXT

21. In
short,
Implementa5on
is
complex!

22. D
Evidence-­‐based
Instruc5onal
materials
Large-­‐scale
changes
in
Teacher
prac5ce
Improved
student
achievement
Con5nued
posi5ve
outcomes
TIME
Based
on
this
theory,
we
created
a
suite
of
instruments
to
measure
the
implementa5on
of
instruc5onal
materials
in
school
systems.

23. D
Evidence-­‐based
Instruc5onal
materials
Large-­‐scale
changes
in
Teacher
prac5ce
Improved
student
achievement
Con5nued
posi5ve
outcomes
TIME
Part
of
this
work
involved
a
research
study
to
understand
the
implementa5on
of
the
Everyday
Mathema&cs
curriculum
in
five
school
districts.
“poten5ally
posi5ve
effects”
on
math
achievement
-­‐
What
Works
Clearinghouse
(2010)

24. D
Evidence-­‐based
Instruc5onal
materials
Large-­‐scale
changes
in
Teacher
prac5ce
Improved
student
achievement
Con5nued
posi5ve
outcomes
TIME
We
focused
in
par5cular
on
measuring
teacher
varia5on
in
EM
use,
and
understanding
why
teachers
used
EM
in
par5cular
ways.

25. We
administered
two
ques5onnaires
to
387
K-­‐5
teachers
to
collect
data
on
which
EM
components
they
were
using,
and
why.

26. EM
essen5al
components
The
first
ques5onnaire
asked
teachers
to
report
the
extent
to
which
they
used
each
of
the
EM
components
in
the
most
recent
unit
they
taught.

27. Items
measured
teachers’
use
of
EM
structures
(ac5vi5es,
procedures,
materials,
and
content),
their
interac5ons
with
students,
and
level
of
student
par5cipa5on.
EM
essen5al
components

28. The
second
ques5onnaire
asked
teachers
to
report
on
selected
teacher
and
school-­‐level
factors.
THE
CONTEXT
Percep5ons
about
EM
Beliefs
about
student
ability
&
behavior
Extrinsic
mo5va5on
for
EM
Self-­‐efficacy
for
EM
Innova5veness
Resourcefulness
Years
of
experience
Ongoing
improvement
structures
Organiza5onal
innova5veness
Time
Locus
of
decision
making
Clarity
of
organiza5onal
goals

29. Facilita5on
of
Small
Group
Par5cipa5on
Facilita5on
of
Cogni5vely
Demanding
Work
Facilita5on
of
Students
Taking
Risks
Facilita5on
of
Student
Interest
Use
of
Assessment
to
Inform
Instruc5on
Use
of
Differen5a5on
Through
confirmatory
factor
analysis,
we
iden5fied
six
specific
EM
instruc5onal
prac5ces.

30. We
conducted
mul5ple
regression
analysis
to
see
which
factors
were
most
strongly
related
to
which
EM
prac5ces.
THE
CONTEXT
Beliefs
about
student
ability
Beliefs
about
student
behavior
Perceived
effec5veness
for
EM
Extrinsic
mo5va5on
for
EM
Self-­‐efficacy
for
EM
Innova5veness
Resourcefulness
Years
of
experience
Ongoing
improvement
structures
Clarity
of
organiza5onal
goals
Organiza5onal
innova5veness
Locus
of
decision
making
Time
sufficiency
?
EM
prac5ces
?

31. We
found
that
teacher
characteris5cs
were
more
influen5al
than
school
characteris5cs.
THE
CONTEXT
Beliefs
about
student
ability
Beliefs
about
student
behavior
Perceived
effec5veness
for
EM
Extrinsic
mo5va5on
for
EM
Self-­‐efficacy
for
EM
Innova5veness
Resourcefulness
Years
of
experience
Ongoing
improvement
structures
Clarity
of
organiza5onal
goals
Organiza5onal
innova5veness
Locus
of
decision
making
Time
sufficiency
?
EM
prac5ces
?

32. D
Evidence-­‐based
Instruc5onal
materials
Large-­‐scale
changes
in
Teacher
prac5ce
Improved
student
achievement
Con5nued
posi5ve
outcomes
TIME
Next,
we
were
interested
in
which
of
these
EM
prac5ces
were
associated
with
changes
in
student
math
achievement.

33. We
conducted
a
path
analysis
to
predict
student
math
achievement
using
data
from
one
district
(N=125
teachers).

34. We
es5mated
a
path
model
to
predict
math
achievement
for
students
in
grades
2-­‐5.
Variables
controlled
for
Independent
variables
Dependent
variable

35. The
model
contained
the
6
EM
instruc5onal
prac5ces
we
iden5fied
in
CFA.
EM
Instruc-onal
prac-ces
Facilita5on
of
small
group
par5cipa5on
Facilita5on
of
cogni5vely
demanding
work
Facilita5on
of
students
taking
risks
Facilita5on
of
student
interest
Use
of
assessment
to
inform
instruc5on
Use
of
differen5a5on

36. The
model
also
contained
6
EM
structures.
EM
Structures
Number
lessons
taught
in
order
Number
of
lessons
with
parts
in
order
Number
of
lessons
omiked
Number
of
lesson
with
parts
omiked
Amount
of
content
covered
in
the
unit
Non-­‐EM
materials
supplementa5on

37. Significant,
posi5ve
direct
effects
on
student
achievement
included:
✔ Non-­‐EM
supplementa5on
✔ Prior
achievement

38. Significant,
posi5ve
indirect
effects
on
student
achievement
included:
✔ Facilita5on
of
cogni5vely
demanding
work
✔ Facilita5on
of
student
interest
✔ Number
of
lessons
taught
in
order
✔ Prior
achievement

39. For
more
informa5on,
contact
Amy
Cassata
at
acassata@uchicago.edu
Century,
J.,
&
Cassata
A.
(2014).
Conceptual
founda5ons
for
measuring
the
implementa5on
of
educa5onal
innova5ons.
In
L.M.H.
SaneL
&
T.R.
Kratochwill
(Eds.),
Treatment
Integrity:
A
Founda&on
for
Evidence-­‐Based
Prac&ce
in
Applied
Psychology
(pp.
81-­‐108).
Washington,
DC:
American
Psychological
Associa5on.

40. This
work
was
funded
by
the
Na5onal
Science
Founda5on
(#1109595)
and
the
US
Department
of
Educa5on
-­‐
Ins5tute
of
Educa5on
Sciences
(#R305A110621).
Acknowledgements