Ins and Outs of Program Evaluation


Published on

Published in: Self Improvement
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Ins and Outs of Program Evaluation

  1. 1. Program Evaluation<br />A Practical/Painless Approach<br />NLA/NEMA Conference – Oct. 14, 2010<br />
  2. 2. About the presenter<br />Kathryn Brockmeier<br />Research Analyst/ Special Projects Associate, Nebraska Library Commission<br />Grant writer<br />Evaluator<br /><br />402-471-4002<br />
  3. 3. About you<br />Library staff<br />Trustee<br />Friends/Foundation board member<br />
  4. 4. About this presentation<br />Introduction to program evaluation: designs, methods, and techniques of analysis<br />Real-world examples<br />NLA/NLC & IMLS Internship program<br />NLC/IMLS Scholarship program<br />NLC One Book for Nebraska Kids & Teens<br />Q&A<br />
  5. 5. What is a program?<br />Stems from mission and goals<br />Goal becomes program<br />A collection of organizational resources geared to accomplish a certain major goal or set of goals<br />
  6. 6. What is program evaluation?<br />Collecting information<br />Making decisions<br />“What do we need to know?”<br />For…<br />Needs assessment, return on investment study, grant application/report<br />
  7. 7. Benefits of evaluation<br />Investigate what works/doesn’t work<br />Share results with stakeholders<br />Demonstrate effectiveness<br />Improve staff’s involvement<br />Model the program<br />
  8. 8. Types of program evaluationto meet program objectives<br />Implementation evaluation (process evaluation)<br />Program implementation objectives<br />Outcome evaluation<br />Participant (stakeholder and individual) outcome objectives<br />
  9. 9. Steps in the evaluation process<br />#1: Assemble an evaluation team<br />#2: Prepare for the evaluation<br />#3: Develop an evaluation plan<br />#4: Collect evaluation information<br />#5: Analyze your evaluation information<br />#6: Prepare the evaluation report<br />
  10. 10. 1. Who will conduct the evaluation?<br />Outside evaluator supported by in-house staff<br />In-house evaluator supported by program staff and outside consultant<br />In-house evaluator supported by program staff<br />
  11. 11. 2. How to prepare for the evaluation<br />a. Decide what to evaluate<br />b. Build a logic model of your program<br />c. State your program implementation and participant outcome objectives in measurable terms<br />d. Identify the context for your evaluation<br />
  12. 12. 2a. Decide what to evaluate<br />Entire program, program component, service, activity<br />Depends on financial and staff resources<br />Grant-driven or funding-driven<br />At start-up of program<br />To check effectiveness<br />
  13. 13. 2b. Logic models<br />Assumptions about your target audience<br />Program interventions<br />Implementation objectives<br />Immediate outcomes<br />Immediate participant outcome objectives<br />Intermediate outcomes<br />Long-term outcomes<br />
  14. 14. 2c. SMART objectives<br />Specific<br />Measurable<br />Achievable<br />Relevant<br />Time-framed<br />By the end of ___________________ (enter timeframe), increase _______________ (what will change?) by _______ (enter a percentage) % or more for ___________________ (who will experience the change?) by _________________ (how will the change occur?).<br /><br />
  15. 15. 2d. Evaluation context<br />Assumptions about your target audience<br />Program interventions<br />Implementation objectives<br />Immediate outcomes<br />Immediate participant outcome objectives<br />Intermediate outcomes<br />Long-term outcomes<br />
  16. 16. 3. Develop the plan<br />Discuss the evaluation framework<br />Evaluating implementation objectives<br />Types of info needed, Sources of info, Selection of info sources, Methods for collecting info, Confidentiality, Timeframe, Analysis methods<br />Evaluating participant outcome objectives<br />Do program participants demonstrate changes in knowledge, attitudes, behaviors, or awareness?<br />Were the changes the result of the program’s interventions?<br />Evaluation practices and procedures<br />
  17. 17. 4. “Data” collection<br />When and where info will be collected<br />Who are the respondents?<br />Methods<br />Survey/questionnaire<br />Interview<br />Focus group<br />Existing records<br />Observation<br />
  18. 18. 5. Analyze the information<br />Descriptive statistics<br />Frequency distributions<br />Summary statistics—measures of central tendency, range<br />Cross-tabulation<br />Filters<br />Content analysis<br />Case studies<br />
  19. 19. 6. Write the report<br />Audience<br />Format<br />Writing tips<br />Oral presentations<br />Visual aids<br />Considerations<br />
  20. 20. Resources<br />The Program Manager’s Guide to Evaluation, from the Administration for Children and Families<br /> <br />Basic Guide to Program Evaluation, from the Free Management Library<br /> <br />