The document discusses program evaluation methods emphasizing the importance of setting SMART goals and determining the right metrics to assess effectiveness, efficiency, and overall success. It outlines various evaluation processes including the ADDIE model and Kirkpatrick's four levels, alongside techniques for formative and summative evaluations. Additionally, it covers the significance of reliability, validity, and risk analysis in evaluating organizational programs and their outcomes.

1. Program Evaluation George Vukotich, Ph.D. Copyright © 2004 Intergovernmental Executive Development Program

2. Program Evaluation Measuring Success It all begins with setting goals and knowing where you are going.

3. SMART GOALS Specific Measurable Agreed On Realistic Timed

4. Relevant Measurement Activity vs. Productivity Effectiveness vs. Efficiency Direct vs. Indirect Are we measuring the right thing.

5. Price / Quality / Innovation What is your organization known for. What is your approach to providing services.

6. What to Evaluate e.g. Evaluating IEDP The course material The instructors The facility Quality of interaction with others What was learned

7. Evaluation Process ADDIE Model Analyze Design Develop Implement Evaluate

8. Evaluate to What Standard Benchmarking Based on pre-set goals Compare to other similar projects Similar industries, organizations, products Past / Historical standards

9. How to evaluate Forms Interviews Focus Groups Self Analysis Grades Supervisors Review Work Productivity

10. Degree of Success To what extent does something need to be effective to.

11. Formative vs. Summative Conducting the evaluation during the process and making adjustment along the way. Waiting until the process is complete and evaluating the results.

12. Test – Retest Methodology

13. Sampling

14. Data – Not just getting it, but using it.

15. Reliability and Validity VALIDITY: Is what we are evaluating the right thing to evaluate to help us be successful. RELIABILITY: Will we get consistent measures over time.

16. Kirkpatrick’s 4 - Levels Level 1 – Reaction Level 2 – Learning Level 3 – Behavior Level 4 -- Results

17. Balanced Scorecard

18. Cost / Benefit Analysis Employee job satisfaction Students completing a program Reduction in processing errors Customer service ratings

19. Unintended Consequences What is the cost of making an error in our evaluation. Risk Analysis

20. ♦ Turnover ♦ Time to Hire, Cost to Hire ♦ Reduced Processing Costs ♦ Improved Safety ♦ Reduced Insurance Costs C O R R E L A T I O N S Return on Investment (ROI)

21. Theory into practice How can your team use “program evaluation” to help you be more successful.