The document discusses evaluating the implementation of a reading program for 5-7 year old children. Key implementation questions include whether volunteers and coordinators implement the program as designed, whether volunteers receive needed training and support, and whether the discussion group model is meeting resident expectations. Evaluating implementation helps determine what actually happens in the program, its characteristics, who participates, what works and doesn't work, and why the program was or wasn't effective. It allows improvements to be made while the program is developing.

1. Implementation
Evaluation
A focus on the enacted program itself –
operations, activities, functions, performance,
component parts, resources, and context.

2. Why Evaluate Implementation?

3. Why Evaluate Implementation?
• “Verifies what the program is and
whether or not it is delivered as intended
to the targeted recipients” (Scheirer, 1994)

4. Why Evaluate Implementation?
• “Verifies what the program is and
whether or not it is delivered as intended
to the targeted recipients” (Scheirer, 1994)
• Can be a part of a formative or
summative evaluation

5. Strong Start Early
®
Reading Program

6. Primary Implementation
questions:
Strong Start Early
®
Reading Program

7. Do volunteers and site
Primary Implementation coordinators implement the
questions: program as designed?
Strong Start Early
®
Reading Program

8. Do volunteers and site
Primary Implementation coordinators implement the
questions: program as designed?
Do volunteers receive the
training and support they need
to implement the program as
designed?
Strong Start Early
®
Reading Program

9. Do volunteers and site
Primary Implementation coordinators implement the
questions: program as designed?
Do volunteers receive the
training and support they need
to implement the program as
designed?
Do school site co-coordinators
receive the training and support
they need to implement the
program as designed?
Strong Start Early
®
Reading Program

10. Do volunteers and site
Primary Implementation coordinators implement the
questions: program as designed?
Do volunteers receive the
training and support they need
to implement the program as
designed?
Do school site co-coordinators
receive the training and support
they need to implement the
program as designed?
Does the Strong Start® program
Strong Start Early
® respond effectively to the needs
and suggestions of schools and
Reading Program volunteers?

11. CCCC Neighbourhood Links - Theory of Change
To develop a neighbourhood community-
Community
organizing model that is a sustainable and ownership
effective approach that involves residents in of centre
building individual, community, and organizational
capacity.
Residents have necessary b5 Residents and partners are
knowledge, skills & connected and acting
commitment together
Residents and partners a5
have a shared vision for
3e) Positive
change Change in
3d) Awareness
attitude re civic conditions New
of
engagement services/ b7
neighbourhood New programs
& Community committees
issues
New "Small wins"
3c) Leaders Networks New
attain new partnerships
skills
? New
2d) Action participation
3b) Resident Centre-
leaders resident plans created
participating in relationship is
development & stronger ?
planning a4
2c) Centre &
1b) Residents a3, a6 resident
receive referrals
to other b1 Shared
relationship w/
resources & partners is
understanding stronger
services of issues &
priorities Action
3a) Residents 2b) New
Referrals participate in Partnerships
discussion a8
groups
1a) Residents 2a) CCCC
access CCCC b1, b2, b3 knows school
Programs & a1, a3 Discussion Groups & church
services Residents priorities 2
have 1c) Centre
awareness of Residents are
b4 aware of
knows about Partners
CCCC community
services & leadership educated on
opportunities
assets and Networking residents'
programs needs
at CCC a2 priorities &
b8 CCCC
b6
Door 2 Door
Neighbourhood
Links

12. Primary Implementation
CCCC Neighbourhood Links - Theory of Change
To develop a neighbourhood community-
Community
organizing model that is a sustainable and ownership
effective approach that involves residents in
Questions:
of centre
building individual, community, and organizational
capacity.
Residents have necessary b5 Residents and partners are
knowledge, skills & connected and acting
commitment together
Residents and partners a5
have a shared vision for
3e) Positive
change Change in
3d) Awareness
attitude re civic conditions New
of
engagement services/ b7
neighbourhood New programs
& Community committees
issues
New "Small wins"
3c) Leaders Networks New
attain new partnerships
skills
? New
2d) Action participation
3b) Resident Centre-
leaders resident plans created
participating in relationship is
development & stronger ?
planning a4
2c) Centre &
1b) Residents a3, a6 resident
receive referrals
to other b1 Shared
relationship w/
resources & partners is
understanding stronger
services of issues &
priorities Action
3a) Residents 2b) New
Referrals participate in Partnerships
discussion a8
groups
1a) Residents 2a) CCCC
access CCCC b1, b2, b3 knows school
Programs & a1, a3 Discussion Groups & church
services Residents priorities 2
have 1c) Centre
awareness of Residents are
b4 aware of
knows about Partners
CCCC community
services & leadership educated on
opportunities
assets and Networking residents'
programs needs
at CCC a2 priorities &
b8 CCCC
b6
Door 2 Door
Neighbourhood
Links

13. Primary Implementation
CCCC Neighbourhood Links - Theory of Change
To develop a neighbourhood community-
Community
organizing model that is a sustainable and ownership
effective approach that involves residents in
Questions:
of centre
building individual, community, and organizational
capacity.
Residents have necessary b5 Residents and partners are
knowledge, skills & connected and acting
commitment together
Have our outreach
Residents and partners a5
have a shared vision for
3e) Positive
change Change in
3d) Awareness
attitude re civic conditions New
of
services/ b7
activities
neighbourhood engagement New programs
& Community committees
issues
New "Small wins"
3c) Leaders Networks New
attain new partnerships
connected with
skills
? New
2d) Action participation
3b) Resident Centre-
leaders resident plans created
residents as we
participating in relationship is
development & stronger ?
planning a4
2c) Centre &
1b) Residents a3, a6 resident
receive referrals
b1
planned?
to other relationship w/
Shared partners is
resources & understanding
services stronger
of issues &
priorities Action
3a) Residents 2b) New
Referrals participate in Partnerships
discussion a8
groups
1a) Residents 2a) CCCC
access CCCC b1, b2, b3 knows school
Programs & a1, a3 Discussion Groups & church
services Residents priorities 2
have 1c) Centre
awareness of Residents are
b4 aware of
knows about Partners
CCCC community
services & leadership educated on
opportunities
assets and Networking residents'
programs needs
at CCC a2 priorities &
b8 CCCC
b6
Door 2 Door
Neighbourhood
Links

14. Primary Implementation
CCCC Neighbourhood Links - Theory of Change
To develop a neighbourhood community-
Community
organizing model that is a sustainable and ownership
effective approach that involves residents in
Questions:
of centre
building individual, community, and organizational
capacity.
Residents have necessary b5 Residents and partners are
knowledge, skills & connected and acting
commitment together
Have our outreach
Residents and partners a5
have a shared vision for
3e) Positive
change Change in
3d) Awareness
attitude re civic conditions New
of
services/ b7
activities
neighbourhood engagement New programs
& Community committees
issues
New "Small wins"
3c) Leaders Networks New
attain new partnerships
connected with
skills
? New
2d) Action participation
3b) Resident Centre-
leaders resident plans created
residents as we
participating in relationship is
development & stronger ?
planning a4
2c) Centre &
1b) Residents a3, a6 resident
receive referrals
b1
planned?
to other relationship w/
Shared partners is
resources & understanding
services stronger
of issues &
priorities Action
3a) Residents 2b) New
Referrals participate in Partnerships
discussion a8
groups
Is our discussion
1a) Residents 2a) CCCC
access CCCC b1, b2, b3 knows school
Programs & a1, a3 Discussion Groups & church
services Residents priorities 2
have
group model
1c) Centre
awareness of Residents are
b4 aware of
knows about Partners
CCCC community
services & leadership educated on
opportunities
assets and Networking residents'
programs needs
at CCC a2 priorities &
meeting the
b8 CCCC
b6
Door 2 Door
expectations of
residents?
Neighbourhood
Links

15. Primary Implementation
CCCC Neighbourhood Links - Theory of Change
To develop a neighbourhood community-
Community
organizing model that is a sustainable and ownership
effective approach that involves residents in
Questions:
of centre
building individual, community, and organizational
capacity.
Residents have necessary b5 Residents and partners are
knowledge, skills & connected and acting
commitment together
Have our outreach
Residents and partners a5
have a shared vision for
3e) Positive
change Change in
3d) Awareness
attitude re civic conditions New
of
services/ b7
activities
neighbourhood engagement New programs
& Community committees
issues
New "Small wins"
3c) Leaders Networks New
attain new partnerships
connected with
skills
? New
2d) Action participation
3b) Resident Centre-
leaders resident plans created
residents as we
participating in relationship is
development & stronger ?
planning a4
2c) Centre &
1b) Residents a3, a6 resident
receive referrals
b1
planned?
to other relationship w/
Shared partners is
resources & understanding
services stronger
of issues &
priorities Action
3a) Residents 2b) New
Referrals participate in Partnerships
discussion a8
groups
Is our discussion
1a) Residents 2a) CCCC
access CCCC b1, b2, b3 knows school
Programs & a1, a3 Discussion Groups & church
services Residents priorities 2
have
group model
1c) Centre
awareness of Residents are
b4 aware of
knows about Partners
CCCC community
services & leadership educated on
opportunities
assets and Networking residents'
programs needs
at CCC a2 priorities &
meeting the
b8 CCCC
b6
Door 2 Door
expectations of
residents?
Neighbourhood
Is the discussion
group model Links
providing us with

16. Why Evaluate Implementation?

17. Why Evaluate Implementation?
• Helps us learn...

18. Why Evaluate Implementation?
• Helps us learn...
– What actually happens in the program?

19. Why Evaluate Implementation?
• Helps us learn...
– What actually happens in the program?
– What are its key characteristics?

20. Why Evaluate Implementation?
• Helps us learn...
– What actually happens in the program?
– What are its key characteristics?
– Who is participating?

21. Why Evaluate Implementation?
• Helps us learn...
– What actually happens in the program?
– What are its key characteristics?
– Who is participating?
– What is working and not working?

22. Why Evaluate Implementation?
• Helps us learn...
– What actually happens in the program?
– What are its key characteristics?
– Who is participating?
– What is working and not working?
– WHY the program was or was not effective.

23. Why Evaluate Implementation?
• Helps us learn...
– What actually happens in the program?
– What are its key characteristics?
– Who is participating?
– What is working and not working?
– WHY the program was or was not effective.
• Implementation problems are common
cause of program failure.

24. Why Evaluate Implementation?
• Helps us learn...
– What actually happens in the program?
– What are its key characteristics?
– Who is participating?
– What is working and not working?
– WHY the program was or was not effective.
• Implementation problems are common
cause of program failure.
• Allows us to use information to improve
interventions while in dynamic stage of
growth.

25. http://www.nursefamilypartnership.org/assets/PDF/Communities/Implementation_Logic_Model

26. http://www.nursefamilypartnership.org/assets/PDF/Communities/Implementation_Logic_Model

27. How goals will
Goals/Expected Outcome be attained Implementation
Outcomes Indicators (Implementation benchmarks
mechanisms)
Volunteer “coaches” Volunteers work with one
Increased pre-reading child at a time for 30 mins
Score on reading work with children in
skills in 5-7 year-old once a week and follow
assessments one-to-one sessions prescribed curriculum and
children
for 10-weeks. activities

28. How goals will
Goals/Expected Outcome be attained Implementation
Outcomes Indicators (Implementation benchmarks
mechanisms)
Volunteer “coaches” Volunteers work with one
Increased pre-reading child at a time for 30 mins
Score on reading work with children in
skills in 5-7 year-old once a week and follow
assessments one-to-one sessions prescribed curriculum and
children
for 10-weeks. activities

29. How goals will
Goals/Expected Outcome be attained Implementation
Outcomes Indicators (Implementation benchmarks
mechanisms)
Volunteer “coaches” Volunteers work with one
Increased pre-reading child at a time for 30 mins
Score on reading work with children in
skills in 5-7 year-old once a week and follow
assessments one-to-one sessions prescribed curriculum and
children
for 10-weeks. activities

30. How goals will
Goals/Expected Outcome be attained Implementation
Outcomes Indicators (Implementation benchmarks
mechanisms)
Volunteer “coaches” Volunteers work with one
Increased pre-reading child at a time for 30 mins
Score on reading work with children in
skills in 5-7 year-old once a week and follow
assessments one-to-one sessions prescribed curriculum and
children
for 10-weeks. activities

31. How goals will
Goals/Expected Outcome be attained Implementation
Outcomes Indicators (Implementation benchmarks
mechanisms)
Volunteer “coaches” Volunteers work with one
Increased pre-reading child at a time for 30 mins
Score on reading work with children in
skills in 5-7 year-old once a week and follow
assessments one-to-one sessions prescribed curriculum and
children
for 10-weeks. activities

32. Why Evaluate Implementation?

33. Why Evaluate Implementation?
“Unless one knows that a program is operating
according to design, there may be little reason to
expect it to produce the desired
outcomes.” (Patton, 2008)

34. Why Evaluate Implementation?
“Unless one knows that a program is operating
according to design, there may be little reason to
expect it to produce the desired
outcomes.” (Patton, 2008)
“Evaluate only proud programs” (Campbell, 1983)

35. Adaptation vs. Fidelity

36. Adaptation vs. Fidelity
• Fidelity
– Compare the actual with the ideal and establish the degree to which
the program is operating as desired.
– Assess adherence to the core blueprint specifications of how a
program is to be implemented.

37. Adaptation vs. Fidelity
• Fidelity
– Compare the actual with the ideal and establish the degree to which
the program is operating as desired.
– Assess adherence to the core blueprint specifications of how a
program is to be implemented.
• Adaptation
– Original ideas are changed in the face of what is actually possible.
– Successful adaptation of a model to local conditions.

38. Adaptation vs. Fidelity
• Fidelity
– Compare the actual with the ideal and establish the degree to which
the program is operating as desired.
– Assess adherence to the core blueprint specifications of how a
program is to be implemented.
• Adaptation
– Original ideas are changed in the face of what is actually possible.
– Successful adaptation of a model to local conditions.
• How different can an actual program be from its ideal and still
be said to have been implemented?

39. Adaptation vs. Fidelity
• Fidelity
– Compare the actual with the ideal and establish the degree to which
the program is operating as desired.
– Assess adherence to the core blueprint specifications of how a
program is to be implemented.
• “the prime focus should be on finding the right
mix of fidelity and adaptation” (Durlak, 2008)
• Adaptation
– Original ideas are changed in the face of what is actually possible.
– Successful adaptation of a model to local conditions.
• How different can an actual program be from its ideal and still
be said to have been implemented?

40. Effort, Input, Access

41. Effort, Input, Access
• Effort & Input Questions:
– Staff, training, qualifications, ratio, resources, etc
– Involves an inventory of program operations

42. Effort, Input, Access
• Effort & Input Questions:
– Staff, training, qualifications, ratio, resources, etc
– Involves an inventory of program operations
• Access Questions:
– Do targeted constituents receive services?
– What gets in the way?

43. Component Evaluation

44. Component Evaluation
• A formal assessment of distinct parts of a
program

45. Component Evaluation
• A formal assessment of distinct parts of a
program
• Shifts focus of analysis from the program to the
component

46. Component Evaluation
• A formal assessment of distinct parts of a
program
• Shifts focus of analysis from the program to the
component
• How is this applicable to your project?

47. “Without information about actual
program operations and causal
mechanisms, decision makers are
limited in interpreting performance
data for either program improvement
or summative judgment.”
Patton, (2008, pp. 330-331)