Download as KEY, PPTX
Uploaded byScot Evans
Implementation Evaluation
The document discusses evaluating the implementation of a reading program for 5-7 year old children. Key implementation questions include whether volunteers and coordinators implement the program as designed, whether volunteers receive needed training and support, and whether the discussion group model is meeting resident expectations. Evaluating implementation helps determine what actually happens in the program, its characteristics, who participates, what works and doesn't work, and why the program was or wasn't effective. It allows improvements to be made while the program is developing.
More Related Content
![Steering Committee03 10final[2]](https://cdn.slidesharecdn.com/ss_thumbnails/steeringcommittee0310final2-090316200550-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Implementation Evaluation
![[Challenge:Future] Development based on youths initiatives](https://cdn.slidesharecdn.com/ss_thumbnails/challengefuture-development-based-on-youths-initiatives4980-130223025004-phpapp01-thumbnail.jpg?width=640&height=640&fit=bounds)
Implementation Evaluation
- 1. Implementation Evaluation A focus on the enacted program itself – operations, activities, functions, performance, component parts, resources, and context.
- 2.
- 3. Why Evaluate Implementation? •“Verifies what the program is and whether or not it is delivered as intended to the targeted recipients” (Scheirer, 1994)
- 4. Why Evaluate Implementation? •“Verifies what the program is and whether or not it is delivered as intended to the targeted recipients” (Scheirer, 1994) • Can be a part of a formative or summative evaluation
- 5. Strong Start Early ® Reading Program
- 6. Primary Implementation questions: Strong Start Early ® Reading Program
- 7. Do volunteers andsite Primary Implementation coordinators implement the questions: program as designed? Strong Start Early ® Reading Program
- 8. Do volunteers andsite Primary Implementation coordinators implement the questions: program as designed? Do volunteers receive the training and support they need to implement the program as designed? Strong Start Early ® Reading Program
- 9. Do volunteers andsite Primary Implementation coordinators implement the questions: program as designed? Do volunteers receive the training and support they need to implement the program as designed? Do school site co-coordinators receive the training and support they need to implement the program as designed? Strong Start Early ® Reading Program
- 10. Do volunteers andsite Primary Implementation coordinators implement the questions: program as designed? Do volunteers receive the training and support they need to implement the program as designed? Do school site co-coordinators receive the training and support they need to implement the program as designed? Does the Strong Start® program Strong Start Early ® respond effectively to the needs and suggestions of schools and Reading Program volunteers?
- 11. CCCC Neighbourhood Links- Theory of Change To develop a neighbourhood community- Community organizing model that is a sustainable and ownership effective approach that involves residents in of centre building individual, community, and organizational capacity. Residents have necessary b5 Residents and partners are knowledge, skills & connected and acting commitment together Residents and partners a5 have a shared vision for 3e) Positive change Change in 3d) Awareness attitude re civic conditions New of engagement services/ b7 neighbourhood New programs & Community committees issues New "Small wins" 3c) Leaders Networks New attain new partnerships skills ? New 2d) Action participation 3b) Resident Centre- leaders resident plans created participating in relationship is development & stronger ? planning a4 2c) Centre & 1b) Residents a3, a6 resident receive referrals to other b1 Shared relationship w/ resources & partners is understanding stronger services of issues & priorities Action 3a) Residents 2b) New Referrals participate in Partnerships discussion a8 groups 1a) Residents 2a) CCCC access CCCC b1, b2, b3 knows school Programs & a1, a3 Discussion Groups & church services Residents priorities 2 have 1c) Centre awareness of Residents are b4 aware of knows about Partners CCCC community services & leadership educated on opportunities assets and Networking residents' programs needs at CCC a2 priorities & b8 CCCC b6 Door 2 Door Neighbourhood Links
- 12. Primary Implementation CCCC Neighbourhood Links - Theory of Change To develop a neighbourhood community- Community organizing model that is a sustainable and ownership effective approach that involves residents in Questions: of centre building individual, community, and organizational capacity. Residents have necessary b5 Residents and partners are knowledge, skills & connected and acting commitment together Residents and partners a5 have a shared vision for 3e) Positive change Change in 3d) Awareness attitude re civic conditions New of engagement services/ b7 neighbourhood New programs & Community committees issues New "Small wins" 3c) Leaders Networks New attain new partnerships skills ? New 2d) Action participation 3b) Resident Centre- leaders resident plans created participating in relationship is development & stronger ? planning a4 2c) Centre & 1b) Residents a3, a6 resident receive referrals to other b1 Shared relationship w/ resources & partners is understanding stronger services of issues & priorities Action 3a) Residents 2b) New Referrals participate in Partnerships discussion a8 groups 1a) Residents 2a) CCCC access CCCC b1, b2, b3 knows school Programs & a1, a3 Discussion Groups & church services Residents priorities 2 have 1c) Centre awareness of Residents are b4 aware of knows about Partners CCCC community services & leadership educated on opportunities assets and Networking residents' programs needs at CCC a2 priorities & b8 CCCC b6 Door 2 Door Neighbourhood Links
- 13. Primary Implementation CCCC Neighbourhood Links - Theory of Change To develop a neighbourhood community- Community organizing model that is a sustainable and ownership effective approach that involves residents in Questions: of centre building individual, community, and organizational capacity. Residents have necessary b5 Residents and partners are knowledge, skills & connected and acting commitment together Have our outreach Residents and partners a5 have a shared vision for 3e) Positive change Change in 3d) Awareness attitude re civic conditions New of services/ b7 activities neighbourhood engagement New programs & Community committees issues New "Small wins" 3c) Leaders Networks New attain new partnerships connected with skills ? New 2d) Action participation 3b) Resident Centre- leaders resident plans created residents as we participating in relationship is development & stronger ? planning a4 2c) Centre & 1b) Residents a3, a6 resident receive referrals b1 planned? to other relationship w/ Shared partners is resources & understanding services stronger of issues & priorities Action 3a) Residents 2b) New Referrals participate in Partnerships discussion a8 groups 1a) Residents 2a) CCCC access CCCC b1, b2, b3 knows school Programs & a1, a3 Discussion Groups & church services Residents priorities 2 have 1c) Centre awareness of Residents are b4 aware of knows about Partners CCCC community services & leadership educated on opportunities assets and Networking residents' programs needs at CCC a2 priorities & b8 CCCC b6 Door 2 Door Neighbourhood Links
- 14. Primary Implementation CCCC Neighbourhood Links - Theory of Change To develop a neighbourhood community- Community organizing model that is a sustainable and ownership effective approach that involves residents in Questions: of centre building individual, community, and organizational capacity. Residents have necessary b5 Residents and partners are knowledge, skills & connected and acting commitment together Have our outreach Residents and partners a5 have a shared vision for 3e) Positive change Change in 3d) Awareness attitude re civic conditions New of services/ b7 activities neighbourhood engagement New programs & Community committees issues New "Small wins" 3c) Leaders Networks New attain new partnerships connected with skills ? New 2d) Action participation 3b) Resident Centre- leaders resident plans created residents as we participating in relationship is development & stronger ? planning a4 2c) Centre & 1b) Residents a3, a6 resident receive referrals b1 planned? to other relationship w/ Shared partners is resources & understanding services stronger of issues & priorities Action 3a) Residents 2b) New Referrals participate in Partnerships discussion a8 groups Is our discussion 1a) Residents 2a) CCCC access CCCC b1, b2, b3 knows school Programs & a1, a3 Discussion Groups & church services Residents priorities 2 have group model 1c) Centre awareness of Residents are b4 aware of knows about Partners CCCC community services & leadership educated on opportunities assets and Networking residents' programs needs at CCC a2 priorities & meeting the b8 CCCC b6 Door 2 Door expectations of residents? Neighbourhood Links
- 15. Primary Implementation CCCC Neighbourhood Links - Theory of Change To develop a neighbourhood community- Community organizing model that is a sustainable and ownership effective approach that involves residents in Questions: of centre building individual, community, and organizational capacity. Residents have necessary b5 Residents and partners are knowledge, skills & connected and acting commitment together Have our outreach Residents and partners a5 have a shared vision for 3e) Positive change Change in 3d) Awareness attitude re civic conditions New of services/ b7 activities neighbourhood engagement New programs & Community committees issues New "Small wins" 3c) Leaders Networks New attain new partnerships connected with skills ? New 2d) Action participation 3b) Resident Centre- leaders resident plans created residents as we participating in relationship is development & stronger ? planning a4 2c) Centre & 1b) Residents a3, a6 resident receive referrals b1 planned? to other relationship w/ Shared partners is resources & understanding services stronger of issues & priorities Action 3a) Residents 2b) New Referrals participate in Partnerships discussion a8 groups Is our discussion 1a) Residents 2a) CCCC access CCCC b1, b2, b3 knows school Programs & a1, a3 Discussion Groups & church services Residents priorities 2 have group model 1c) Centre awareness of Residents are b4 aware of knows about Partners CCCC community services & leadership educated on opportunities assets and Networking residents' programs needs at CCC a2 priorities & meeting the b8 CCCC b6 Door 2 Door expectations of residents? Neighbourhood Is the discussion group model Links providing us with
- 16.
- 17. Why Evaluate Implementation? • Helps us learn...
- 18. Why Evaluate Implementation? • Helps us learn... – What actually happens in the program?
- 19. Why Evaluate Implementation? • Helps us learn... – What actually happens in the program? – What are its key characteristics?
- 20. Why Evaluate Implementation? • Helps us learn... – What actually happens in the program? – What are its key characteristics? – Who is participating?
- 21. Why Evaluate Implementation? • Helps us learn... – What actually happens in the program? – What are its key characteristics? – Who is participating? – What is working and not working?
- 22. Why Evaluate Implementation? • Helps us learn... – What actually happens in the program? – What are its key characteristics? – Who is participating? – What is working and not working? – WHY the program was or was not effective.
- 23. Why Evaluate Implementation? • Helps us learn... – What actually happens in the program? – What are its key characteristics? – Who is participating? – What is working and not working? – WHY the program was or was not effective. • Implementation problems are common cause of program failure.
- 24. Why Evaluate Implementation? • Helps us learn... – What actually happens in the program? – What are its key characteristics? – Who is participating? – What is working and not working? – WHY the program was or was not effective. • Implementation problems are common cause of program failure. • Allows us to use information to improve interventions while in dynamic stage of growth.
- 25.
- 26.
- 27. How goals will Goals/Expected Outcome be attained Implementation Outcomes Indicators (Implementation benchmarks mechanisms) Volunteer “coaches” Volunteers work with one Increased pre-reading child at a time for 30 mins Score on reading work with children in skills in 5-7 year-old once a week and follow assessments one-to-one sessions prescribed curriculum and children for 10-weeks. activities
- 28. How goals will Goals/Expected Outcome be attained Implementation Outcomes Indicators (Implementation benchmarks mechanisms) Volunteer “coaches” Volunteers work with one Increased pre-reading child at a time for 30 mins Score on reading work with children in skills in 5-7 year-old once a week and follow assessments one-to-one sessions prescribed curriculum and children for 10-weeks. activities
- 29. How goals will Goals/Expected Outcome be attained Implementation Outcomes Indicators (Implementation benchmarks mechanisms) Volunteer “coaches” Volunteers work with one Increased pre-reading child at a time for 30 mins Score on reading work with children in skills in 5-7 year-old once a week and follow assessments one-to-one sessions prescribed curriculum and children for 10-weeks. activities
- 30. How goals will Goals/Expected Outcome be attained Implementation Outcomes Indicators (Implementation benchmarks mechanisms) Volunteer “coaches” Volunteers work with one Increased pre-reading child at a time for 30 mins Score on reading work with children in skills in 5-7 year-old once a week and follow assessments one-to-one sessions prescribed curriculum and children for 10-weeks. activities
- 31. How goals will Goals/Expected Outcome be attained Implementation Outcomes Indicators (Implementation benchmarks mechanisms) Volunteer “coaches” Volunteers work with one Increased pre-reading child at a time for 30 mins Score on reading work with children in skills in 5-7 year-old once a week and follow assessments one-to-one sessions prescribed curriculum and children for 10-weeks. activities
- 32.
- 33. Why Evaluate Implementation? “Unlessone knows that a program is operating according to design, there may be little reason to expect it to produce the desired outcomes.” (Patton, 2008)
- 34. Why Evaluate Implementation? “Unlessone knows that a program is operating according to design, there may be little reason to expect it to produce the desired outcomes.” (Patton, 2008) “Evaluate only proud programs” (Campbell, 1983)
- 35.
- 36. Adaptation vs. Fidelity •Fidelity – Compare the actual with the ideal and establish the degree to which the program is operating as desired. – Assess adherence to the core blueprint specifications of how a program is to be implemented.
- 37. Adaptation vs. Fidelity •Fidelity – Compare the actual with the ideal and establish the degree to which the program is operating as desired. – Assess adherence to the core blueprint specifications of how a program is to be implemented. • Adaptation – Original ideas are changed in the face of what is actually possible. – Successful adaptation of a model to local conditions.
- 38. Adaptation vs. Fidelity •Fidelity – Compare the actual with the ideal and establish the degree to which the program is operating as desired. – Assess adherence to the core blueprint specifications of how a program is to be implemented. • Adaptation – Original ideas are changed in the face of what is actually possible. – Successful adaptation of a model to local conditions. • How different can an actual program be from its ideal and still be said to have been implemented?
- 39. Adaptation vs. Fidelity • Fidelity – Compare the actual with the ideal and establish the degree to which the program is operating as desired. – Assess adherence to the core blueprint specifications of how a program is to be implemented. • “the prime focus should be on finding the right mix of fidelity and adaptation” (Durlak, 2008) • Adaptation – Original ideas are changed in the face of what is actually possible. – Successful adaptation of a model to local conditions. • How different can an actual program be from its ideal and still be said to have been implemented?
- 40.
- 41. Effort, Input, Access •Effort & Input Questions: – Staff, training, qualifications, ratio, resources, etc – Involves an inventory of program operations
- 42. Effort, Input, Access •Effort & Input Questions: – Staff, training, qualifications, ratio, resources, etc – Involves an inventory of program operations • Access Questions: – Do targeted constituents receive services? – What gets in the way?
- 43.
- 44. Component Evaluation • Aformal assessment of distinct parts of a program
- 45. Component Evaluation • Aformal assessment of distinct parts of a program • Shifts focus of analysis from the program to the component
- 46. Component Evaluation • Aformal assessment of distinct parts of a program • Shifts focus of analysis from the program to the component • How is this applicable to your project?
- 47. “Without information aboutactual program operations and causal mechanisms, decision makers are limited in interpreting performance data for either program improvement or summative judgment.” Patton, (2008, pp. 330-331)