Street Jibe Evaluation


Published on

Presentation by Dr. Uzo Anucha

Published in: Economy & Finance, Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Street Jibe Evaluation

  1. 1. A Conversation about Program Evaluation: Why, How and When? Uzo Anucha Applied Social Welfare Research and Evaluation Group School of Social Work York University
  2. 2. <ul><li>Program evaluation means taking a systematic approach to asking and answering questions about a program. </li></ul><ul><li>Program evaluation is not an assessment of individual staff performance. The purpose is to gain an overall understanding of the functioning of a program. </li></ul><ul><li>Program evaluation is not an audit – evaluation does not focus on compliance with laws and regulations. </li></ul><ul><li>Program evaluation is not research . It is a pragmatic way to learn about a program. </li></ul>What is Program Evaluation?
  3. 3. <ul><li>Program evaluation is not one method . It can involve a range of techniques for gathering information to answer questions about a program. </li></ul><ul><li>Most programs already collect a lot of information that can be used for evaluation. Data collection for program evaluation can be incorporated in the ongoing record keeping of the program. </li></ul>What is Program Evaluation?
  4. 4. Definition of Program Evaluation <ul><li>“ Program evaluation is a collection of methods, skills and sensitivities necessary to determine whether a human service is needed and likely to be used , whether it is sufficiently intensive to meet the unmet needs identified, whether the service is offered as planned , and whether the human service actually does help people in need at reasonable cost without undesirable side effects ” (Posavac & Carey, 2003. p.2) </li></ul>
  5. 5. <ul><li>Verify that resources are devoted to meeting unmet needs </li></ul><ul><li>Verify that planned programs do provide services </li></ul><ul><li>Examine the results </li></ul><ul><li>Determine which services produce the best results </li></ul><ul><li>Select the programs that offer the most needed types of services </li></ul>Why Evaluate?
  6. 6. <ul><li>Provide information needed to maintain and improve quality </li></ul><ul><li>Watch for unplanned side effects </li></ul><ul><li>Create program documentation </li></ul><ul><li>Help to better allocate program resources </li></ul><ul><li>Assist staff in program development and improvement </li></ul>Why Evaluate?
  7. 7. Evaluation can…. <ul><li>Increase our knowledge base </li></ul><ul><li>Guide decision making </li></ul><ul><ul><li>Policymakers </li></ul></ul><ul><ul><li>Administrators </li></ul></ul><ul><ul><li>Practitioners </li></ul></ul><ul><ul><li>Funders </li></ul></ul><ul><ul><li>General public </li></ul></ul><ul><ul><li>Clients </li></ul></ul><ul><li>Demonstrate accountability </li></ul><ul><li>Assure that client objectives are being achieved </li></ul>
  8. 8. Who is an evaluation for?
  9. 9. <ul><li>What do they want to know? </li></ul><ul><li>What do we want to tell them about the program? </li></ul><ul><li>How can they contribute to the evaluation? </li></ul><ul><li>Program participants? </li></ul><ul><li>Family members and caregivers? </li></ul><ul><li>Program staff? </li></ul><ul><li>Volunteers? </li></ul><ul><li>Partner agencies and professionals? </li></ul><ul><li>Referral sources? </li></ul><ul><li>Funders? </li></ul><ul><li>Others? </li></ul>Who is an evaluation for?
  10. 10. <ul><li>Every evaluation happens in a political context. Find out what it is. </li></ul><ul><li>Clarify your role in the evaluation. Let people know what you can and can not do. </li></ul><ul><li>Be fair and impartial. </li></ul><ul><li>Consider what is a reasonable and feasible evaluation for the particular program. </li></ul>Being Smart About Evaluation
  11. 11. Challenging Attitudes toward Program Evaluation……. <ul><li>Expectations of slam-bang effects </li></ul><ul><li>Assessing program quality is unprofessional </li></ul><ul><li>Evaluation might inhibit innovation </li></ul><ul><li>Program will be terminated </li></ul><ul><li>Information will be misused </li></ul><ul><li>Qualitative understanding might be lost </li></ul><ul><li>Evaluation drains resources </li></ul><ul><li>Loss of program control </li></ul><ul><li>Evaluation has little impact </li></ul>
  12. 12. Types of evaluations <ul><li>Needs assessment </li></ul><ul><li>Evaluability assessment </li></ul><ul><li>Process evaluation </li></ul><ul><li>Outcome evaluation </li></ul><ul><li>Efficiency evaluation (cost evaluation) </li></ul>
  13. 13. Needs Assessment/Community Capacity Mapping: <ul><li>Prerequisite to program planning and development: </li></ul><ul><ul><li>What is the community profile? </li></ul></ul><ul><ul><li>What are the particular unmet needs of a target population? </li></ul></ul><ul><ul><li>What forms of service are likely to be attractive to the population? </li></ul></ul><ul><ul><li>Are existing services known or acceptable to potential clients? </li></ul></ul><ul><ul><li>What barriers prevent clients from assessing existing services? </li></ul></ul>
  14. 14. Evaluability Assessment <ul><li>Prerequisite to formal evaluation </li></ul><ul><ul><li>Are program goals articulated and measurable? </li></ul></ul><ul><ul><li>Is the program model definable (flow diagram)? </li></ul></ul><ul><ul><li>Are the goals and activities logically linked? </li></ul></ul><ul><ul><li>Is there sufficient rigour and resources to merit evaluation? </li></ul></ul>
  15. 15. Process Evaluation: <ul><li>Verify program implementation </li></ul><ul><ul><li>Is the program attracting a sufficient number of clients? </li></ul></ul><ul><ul><li>Are clients representative of the target population? </li></ul></ul><ul><ul><li>How much does the staff actually contact the client? </li></ul></ul><ul><ul><li>Does the workload of staff match that planned? </li></ul></ul><ul><ul><li>Are there differences in effort among staff? </li></ul></ul>
  16. 16. Outcome Evaluation <ul><li>Describe program effects </li></ul><ul><ul><li>Is the desired outcome observed? </li></ul></ul><ul><ul><li>Are program participants better off than non-participants? </li></ul></ul><ul><ul><li>Is there evidence that the program caused the observed changes? </li></ul></ul><ul><ul><li>Is there support for the theoretical foundations underpinning the program? </li></ul></ul><ul><ul><li>Is there evidence that the program could be implemented successfully elsewhere? </li></ul></ul>
  17. 17. Evaluation of Efficiency <ul><li>Effectiveness relative to cost: </li></ul><ul><ul><li>Are funds spent for intended purposes? </li></ul></ul><ul><ul><li>Are program outcomes achieved at a reasonable cost? </li></ul></ul><ul><ul><li>Can dollar values be assigned to the outcomes? </li></ul></ul><ul><ul><li>Is the outcome achieved greater than other programs of similar costs? </li></ul></ul>
  18. 18. Process Evaluation….
  19. 19. Process Evaluation <ul><li>Sometimes referred to as “formative evaluation” </li></ul><ul><li>Looks at the approach to client service to day operations </li></ul><ul><li>Two major elements: </li></ul><ul><ul><li>1) how a program’s services are delivered to clients (what worker’s do including frequency and intensity; client characteristics; satisfaction </li></ul></ul><ul><ul><li>2) administrative mechanisms to support these services (qualifications; structures; hours; support services; supervision; training) </li></ul></ul>
  20. 20. Process Evaluation <ul><li>Can occur concurrently with outcome evaluation </li></ul><ul><li>  Need to establish common program language </li></ul><ul><li>  Purpose of process evaluation: improve service; generate knowledge; estimate cost efficiency </li></ul><ul><li>  May be essential component of organizational accreditation </li></ul>
  21. 21. Steps in Process Evaluation <ul><li>Deciding what questions to ask </li></ul><ul><ul><li>background; client profile; staff profile; nature, amount and duration of service provided; nature of interventions; admin. supports; satisfaction of key stakeholders; efficiency? </li></ul></ul><ul><li>  Developing data collection instruments </li></ul><ul><ul><li>ease of use; consistency with program operation and objectives; user input) </li></ul></ul>
  22. 22. Steps in Process Evaluation <ul><li>Developing a data collection monitoring system (unit of analysis; number of units to include e.g. sampling; when and how to collect data </li></ul><ul><li>Scoring and analyzing data (categorize by client groups, interventions, program; display graphically). </li></ul><ul><li>Developing a feedback system (clients, workers, supervisors, administrators). </li></ul><ul><li>Disseminating and communicating results </li></ul>
  23. 23. Sources of Process Evaluation Data <ul><li>Funder/agency/program documents (eg. Model; rationale;funding agreement) </li></ul><ul><li>  Key informant interviews with service delivery and admin. personnel or key collateral agencies (e.g. Referral source) </li></ul><ul><li>Service utilization statistics </li></ul><ul><li>  Management Information Systems (M.I.S.) </li></ul><ul><li>  Surveys/interviews with consumers (e.g. client satisfaction </li></ul>
  24. 24. Outcome Evaluation….
  25. 25. Outcome Evaluation <ul><li>Outcomes are benefits or changes for individuals or populations during or after participating in program activities. Outcomes may relate to behavior, skills, knowledge, attitudes, values, condition, or other attributes. </li></ul><ul><li>They are what participants know, think, or can do; or how they behave; or what their condition is, that is different following the program. </li></ul><ul><li>Outcome evaluation helps us to demonstrate the nature of change that took place </li></ul>
  26. 26. Outcome Evaluation <ul><li>Outcome evaluation tests hypotheses about how we believe that clients will change after a period of time in our program. </li></ul><ul><li>Evaluation findings are specific to a specific group of clients experiencing the specific condition of one specific program over a specific time frame at a specific time. </li></ul>
  27. 27. For example: <ul><li>A program to counsel families on financial management, outputs--what the service produces--include the number of financial planning sessions and the number of families seen. The desired outcomes--the changes sought in participants' behavior or status- -can include their developing and living within a budget, making monthly additions to a savings account, and having increased financial stability. </li></ul>
  28. 28. Uses of Outcome Evaluation <ul><li>Improving program services to clients </li></ul><ul><li>Generating knowledge for the profession </li></ul><ul><li>Estimating costs </li></ul><ul><li>Demonstrate nature of change...evaluation of program objectives e.g. what we expect clients to achieve </li></ul><ul><li>Guide major program decisions and program activities </li></ul>
  29. 29. Program-Level Evaluations <ul><li>Program level evaluations vary on a continuum and are fundamentally made up of three levels </li></ul><ul><ul><li>Exploratory </li></ul></ul><ul><ul><li>Descriptive </li></ul></ul><ul><ul><li>Explanatory </li></ul></ul>
  30. 30. The Continuum…… <ul><li>Qualitative ------------Quantitative </li></ul><ul><li>Exploratory----Descriptive----Explanatory </li></ul>
  31. 31. Exploratory Outcome Evaluation Designs <ul><li>Questions here include: </li></ul><ul><ul><li>Did the participants meet a criterion (e.g. Treated vs. Untreated)? </li></ul></ul><ul><ul><li>Did the participants improve (e.g. appropriate direction)? </li></ul></ul><ul><ul><li>Did the participants improve enough (e.g. statistical vs. meaningful difference)? </li></ul></ul><ul><ul><li>Is there a relation between change and service intensity and participant characteristics? </li></ul></ul>
  32. 32. Exploratory Designs <ul><li>One group post test only </li></ul><ul><li>Multi-group post test only </li></ul><ul><li>Longitudinal case study </li></ul><ul><li>Longitudinal survey </li></ul>
  33. 33. Strengths of Exploratory Designs <ul><li>Less intrusive and inexpensive </li></ul><ul><li>Assess the usefulness and feasibility of further evaluations </li></ul><ul><li>Can correlate improvement with other variables. </li></ul>
  34. 34. Descriptive Designs <ul><li>To show that something causes something else, it is necessary to demonstrate: </li></ul><ul><ul><li>That the cause precedes the supposed effects in time e.g. that an intervention precedes the change </li></ul></ul><ul><ul><li>That the cause covaries with the effect – the change covaries with the intervention – the more the intervention, the more the change. </li></ul></ul><ul><ul><li>That no viable explanation of the effect can be found except for the assumed cause e.g. there can be no other explanation for the change except the intervention. </li></ul></ul><ul><li>Both 1 and 2 can be achieved with exploratory designs…but not 3. </li></ul>
  35. 35. Descriptive Designs <ul><li>Randomized one-group posttest only </li></ul><ul><li>Randomized cross-sectional and longitudinal survey </li></ul><ul><li>One-group pretest-posttest </li></ul><ul><li>Comparison group posttest only </li></ul><ul><li>Comparison group pretest-posttest </li></ul><ul><li>Interrupted time series  </li></ul>
  36. 36. Explanatory Designs <ul><li>Defining characteristic is observation of people randomly assigned to either a program or control condition . </li></ul><ul><li>Considered much better at addressing threats to internal validity </li></ul><ul><li>Program group vs. Control group: if groups are formed randomly there is no reason to believe they differ in rate of maturation; no self selection into groups; groups did not begin at different levels </li></ul>
  37. 37. Explanatory Designs <ul><li>Classical experimental </li></ul><ul><li>Solomon four group </li></ul><ul><li>Randomized posttest only control group </li></ul>
  38. 38. Explanatory Designs <ul><li>Strengths/Limitations: </li></ul><ul><ul><li>counter threats to internal validity </li></ul></ul><ul><ul><li>allow interpretations of causation </li></ul></ul><ul><ul><li>expensive and difficult to implement </li></ul></ul><ul><ul><li>frequently resistance from practitioners who already know what is best </li></ul></ul><ul><li>  Suggested Times to Use: </li></ul><ul><ul><li>when new program is introduced </li></ul></ul><ul><ul><li>when stakes are high </li></ul></ul><ul><ul><li>when there is controversy over efficacy </li></ul></ul><ul><ul><li>when policy change is desired </li></ul></ul><ul><ul><li>when program demand is high </li></ul></ul>
  39. 39. Internal Validity (causality) <ul><li>History </li></ul><ul><li>Maturation </li></ul><ul><li>Testing </li></ul><ul><li>Instrumentation error </li></ul><ul><li>Statistical regression </li></ul><ul><li>Differential selection </li></ul><ul><li>Mortality </li></ul><ul><li>Reactive effects </li></ul><ul><li>Interaction effects </li></ul><ul><li>Relations between experimental and control groups (e.g. rivalry) </li></ul>
  40. 40. External Validity (generalizability) <ul><li>Pretest-treatment interaction </li></ul><ul><li>Selection-treatment interaction </li></ul><ul><li>specificity of variables and settings </li></ul><ul><li>reactive effects </li></ul><ul><li>multiple treatment interference </li></ul><ul><li>researcher bias </li></ul>
  41. 41. Steps in Outcome Evaluation <ul><li>Step 1 : Operationalizing program objectives </li></ul><ul><li>Step 2 : Selecting the measurements and stating the outcomes </li></ul><ul><ul><li>psychometrics, administration, measurement burden </li></ul></ul><ul><li>Step 3 : Designing a monitoring system </li></ul><ul><ul><li>sampling (# clients to include - min. 30 per group of interest; sampling strategies; missing data) </li></ul></ul>
  42. 42. Steps in Outcome Evaluation <ul><li>Step 3 (cont’d): Designing a monitoring system: deciding when & how data will be collected will depend on question we are trying to answer: </li></ul><ul><ul><li>What degree is program achieving its objectives e.g. how much change? </li></ul></ul><ul><ul><li>Differences between program participants and non-participants </li></ul></ul><ul><ul><li>Question of causality </li></ul></ul><ul><ul><li>Longevity of client changes </li></ul></ul><ul><li>Deciding how data will be collected: </li></ul><ul><ul><li>By telephone, mail or in person. </li></ul></ul>
  43. 43. Steps in Outcome Evaluation <ul><li>Step 4: Analysing and displaying data </li></ul><ul><ul><li>Can also present outcome data according to subgroups by using demographics </li></ul></ul><ul><ul><li>most useful when when data can be aggregated and summarized to provide an overview on client outcomes. </li></ul></ul><ul><li>Step 5 : Developing a feedback system </li></ul><ul><li>Step 6 : Disseminating results </li></ul>
  44. 44. Ready, Set, Go? Some things to consider…..
  45. 45. Important to consider… <ul><li>Internal or external evaluators? </li></ul><ul><li>Scope of evaluation? </li></ul><ul><ul><li>Boundary </li></ul></ul><ul><ul><li>Size </li></ul></ul><ul><ul><li>Duration </li></ul></ul><ul><ul><li>Complexity </li></ul></ul><ul><ul><li>Clarity and time span of program objectives </li></ul></ul><ul><ul><li>Innovativeness </li></ul></ul>
  46. 46. Sources of Data for Evaluation <ul><li>Intended beneficiaries of the program </li></ul><ul><ul><li>Program participants </li></ul></ul><ul><ul><li>Artifacts </li></ul></ul><ul><ul><li>Community indexes </li></ul></ul><ul><li>Providers of service </li></ul><ul><ul><li>Program staff </li></ul></ul><ul><ul><li>Program records </li></ul></ul><ul><li>Observers </li></ul><ul><ul><li>Expert observers </li></ul></ul><ul><ul><li>Trained observers </li></ul></ul><ul><ul><li>Significant others </li></ul></ul><ul><ul><li>Evaluation staff </li></ul></ul>
  47. 47. Good Assessment Procedures <ul><li>Multiple Sources: triangulation and corroborating evidence  </li></ul><ul><li>Multiple Variables: focus on single variable can corrupt evaluation; different variables affected by different sources of error  </li></ul><ul><li>Non-Reactive Measures: measures which do not themselves effect respondents  </li></ul><ul><li>Important Variables: politically, conceptually and methodologically important variables </li></ul>
  48. 48. Good Assessment Procedures <ul><li>Valid Measures: instrument measures what it is supposed to measure (face, criterion, construct); more focused on objective behaviour...more likely to be valid </li></ul><ul><li>Reliable Measures: consistent measure of construct (stable e.g test-retest; recognizable e.g. inter-rater; homogeneity e.g. split-half) </li></ul><ul><li>Sensitivity to Change: able to detect small changes; pre-test scores should be scrutinized </li></ul>
  49. 49. Good Assessment Procedures <ul><li>Cost-effectiveness: length and ease and cost of production and distribution  </li></ul><ul><li>Grounded in existing research and Experiential relevance: use within literature; published psychometric data and population norms; pre-tested with relevant population </li></ul>
  50. 50. Ideal Program Evaluation Characteristics… <ul><li>Counter threats to internal/external validity </li></ul><ul><ul><li>established time ordering (intervention precedes effect) </li></ul></ul><ul><ul><li>intervention is manipulated (admin. to at least one group) </li></ul></ul><ul><ul><li>relations between intervention and program must be established </li></ul></ul><ul><ul><li>design must control for rival hypotheses </li></ul></ul><ul><ul><li>one control group must be used </li></ul></ul><ul><ul><li>random assignment </li></ul></ul>
  51. 51. Planning an Evaluation <ul><li>Identify the program and stakeholders </li></ul><ul><li>Identify information needs of evaluation </li></ul><ul><li>Examine the literature </li></ul><ul><li>Determine the methodology (sample, design, data collection, procedures, analysis, time lines, budget) </li></ul><ul><li>Prepare a written proposal </li></ul>
  52. 52. Preparing for an Evaluation <ul><li>Obtain a complete program description (newsletters, annual reports, etc). </li></ul><ul><li>Identify & meet with stakeholders – program director, staff, funders/program sponsors and clients/program participants. </li></ul><ul><li>Identify the information needs of evaluation: </li></ul><ul><ul><li>Who wants an evaluation? </li></ul></ul><ul><ul><li>What should evaluation focus on? </li></ul></ul><ul><ul><li>Why is an evaluation needed? </li></ul></ul><ul><ul><li>When is an evaluation wanted? </li></ul></ul><ul><ul><li>What resources are needed? </li></ul></ul><ul><ul><li>What is the evaluability of the program? </li></ul></ul>
  53. 53. Things to Consider….. <ul><li>Planning an evaluation follows similar steps to the conduct of more basic research with some additional considerations </li></ul><ul><li>More effort needs to be expended in engaging and negotiating with stakeholder groups (e.g. interviews, or research study steering committee) </li></ul><ul><li>There needs to be a keener awareness of the social/political context of the evaluation (e.g. differing and competing interests) </li></ul>
  54. 54. Things to Consider….. <ul><li>Greater effort needs to be expended in becoming familiar with and articulating program evaluation criteria including goals, objectives and implementation of model and theory </li></ul><ul><li>The choice of measurement needs to be grounded in a detailed understanding of program goals, objectives and service delivery in addition to the qualities of ideal assessment measures (e.g. reliability and validity) </li></ul><ul><li>All stages of planning must take into consideration practical limitations (eg. time, budget, resources) </li></ul>
  55. 55. Program Evaluation Exercise….. <ul><li>Consider a social service setting with which you are familiar and illustrate how program evaluation activities could be applied to it. </li></ul><ul><ul><li>What needs to be done if you were to contemplate evaluating this program? </li></ul></ul><ul><ul><li>Who are the different stakeholders who should be involved in an evaluation and how? </li></ul></ul><ul><ul><li>What are some evaluation questions that could be asked and what methods could one use? </li></ul></ul><ul><ul><li>What are some criteria of program success that can be easily measured but miss the central point of the program (measurable but irrelevant)? </li></ul></ul><ul><ul><li>What are some measurable and relevant criteria? </li></ul></ul><ul><ul><li>What else needs to be considered? </li></ul></ul>