June 20 2010 bsi christie


Published on

Published in: Education, Business, Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • 06/19/10
  • 06/19/10
  • 06/19/10
  • 06/19/10
  • 06/19/10 A logic model can be used to help clarify the activities of a program, how those activities lead to certain outcomes and how those outcomes link to the ultimate goal of a program. Kellog defines a logic model as: a picture of how your organization does its work – the theory and assumptions underlying the program. A program logic model links outcomes (both short- and long-term) with program activities/processes and the theoretical assumptions/principles of the program.
  • 06/19/10
  • 06/19/10
  • 06/19/10 YOUR PLANNED WORK describes what resources you think you need to implement your program and what you intend to do. YOUR INTENDED RESULTS include all of the program’s desired results (outputs, out- comes, and impact).
  • 06/19/10
  • 06/19/10
  • 06/19/10
  • 06/19/10
  • 06/19/10
  • 06/19/10 From Kellogg, Ch. 4, p. 36-37 The logic model can be divided into three categories: Context, implementation, and outcomes. Context: influences and resources Implementation: activities and outputs Outcomes: short-term, intermediate, long/term “ Context is how the program functions within the economic, social, and political environment of its community.” “ Implementation assesses the extent to which activities were executed as planned.” “ Outcomes determine the extent to which progress is being made toward the desired changes in individuals, organizations, communities, or systems.”
  • 06/19/10
  • e
  • intro
  • edits
  • intro
  • Balance communications with resources available
  • June 20 2010 bsi christie

    1. 1. June 20, 2010 Christina A. Christie, Ph.D. University of California, Los Angeles Michael Harnar, MA Claremont Graduate University
    2. 2. <ul><li>Understand and consider the various ways evaluation can influence decisions and activities </li></ul><ul><li>Examine the use of logic models in the evaluation process </li></ul><ul><li>Identify strategies for developing logic models and promoting evaluation use </li></ul>
    3. 3. <ul><li>Evaluation refers to the process of determining the merit, worth, or value of something, or the product of that process. (Scriven, 1991, p. 139) </li></ul><ul><li>Program evaluation is the use of social research methods to systematically investigate the effectiveness of social intervention programs. (Rossi, Lipsey, Freeman, 2004, p. 28) </li></ul><ul><li>Program evaluation is the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming. (Patton, 1997, p. 23) </li></ul>
    4. 4. <ul><li>Intended for: </li></ul><ul><ul><li>Program decision making </li></ul></ul><ul><ul><li>Rendering judgments </li></ul></ul><ul><li>Stakeholders set the agenda </li></ul><ul><li>Primary audience for the study: </li></ul><ul><ul><li>Program staff </li></ul></ul><ul><li>Findings are: </li></ul><ul><ul><li>Program & context specific </li></ul></ul><ul><ul><li>Shared on an ongoing basis </li></ul></ul><ul><li>Intended for: </li></ul><ul><ul><li>Adding to the existing knowledge base </li></ul></ul><ul><li>Researcher sets the agenda </li></ul><ul><li>Primary audience for the study: </li></ul><ul><ul><li>Scientific community & the public </li></ul></ul><ul><li>Findings are: </li></ul><ul><ul><li>Intended to be broadly applicable or generalizable </li></ul></ul><ul><ul><li>Shared at the end of the study </li></ul></ul>Research
    5. 5. <ul><li>Evaluation is carried out for different reasons. The unique purpose of each evaluation impacts its design. Some of the more general reasons for evaluation include: </li></ul><ul><li>To gain insight about a program and its operations – To learn how a program has been operating and how it is evolving, and to establish what works and what doesn’t </li></ul><ul><li>To improve practice – to modify or adapt practice to enhance the success of activities </li></ul><ul><li>To assess effects – to see how well a program is meeting it’s objectives and goals, how it benefits the community, and what evidence there is for its effectiveness </li></ul><ul><li>To build capacity - increase funding, enhance skills, strengthen accountability </li></ul>
    6. 6. <ul><li>Some more specific reasons for conducting evaluation: </li></ul><ul><li>A program evaluation can find out “what works” and “what does not work.” </li></ul><ul><li>A program evaluation can showcase the effectiveness of a program to the campus community and to funders. </li></ul><ul><li>A program evaluation can improve faculty practice, administrators’ leadership, and promote organizational development . </li></ul><ul><li>A program evaluation can increase a program or colleges’ capacity to conduct a critical self assessment and plan for the future. </li></ul><ul><li>A program evaluation can build knowledge for more effective teaching and learning. </li></ul>
    7. 7. <ul><li>The purpose and design of a program evaluation can answer different questions and provide a broad range of information. Some examples include: </li></ul><ul><li>Learning whether proposed program materials are suitable </li></ul><ul><li>Learning whether program plans are feasible </li></ul><ul><li>Providing an early warning system for problem identification </li></ul><ul><li>Learning whether programs are producing the desired results </li></ul><ul><li>Learning whether programs have unexpected benefits or </li></ul><ul><li>problems </li></ul><ul><li>Enabling managers to improve program processes </li></ul><ul><li>Monitoring progress toward the program’s goals </li></ul><ul><li>Producing data on which to base future programs </li></ul><ul><li>Demonstrating the effectiveness of the program </li></ul><ul><li>Identifying the most effective parts of the program for refinement </li></ul><ul><li>Gathering valuable information that can be shared </li></ul>
    8. 8. Evaluation, like anything else, can be done well and produce benefits or it can be done poorly adding little value, sometimes having negative impact . The following table contrasts characteristics of effective and ineffective evaluation. Evaluation is… Evaluation is not… Done with you Done TO you Able to provide rich information Simply program monitoring Intended to be used Intended to sit on a shelf or to check a box For the program stakeholders For the evaluator or only for management Systematic Haphazard FUN! Scary (Really it isn’t!– you’ll see  )
    9. 9. <ul><li>There are many ways to approach an evaluation of a program </li></ul><ul><ul><li>And so there are “theories” </li></ul></ul><ul><ul><li>(also called “models” or “approaches” or “frameworks” ) </li></ul></ul><ul><ul><li>about how best to conduct an evaluation study </li></ul></ul><ul><li>There are many evaluation theories and there is no one “right” theory or way of doing an evaluation </li></ul>
    10. 10. <ul><li>An evaluation theory offer evaluators a conceptual framework that helps to organize the procedures </li></ul><ul><li>Theories are distinguished by what is believed to be the primary purpose of evaluation </li></ul>
    11. 11. <ul><li>The theoretical framework that guides this module focuses on practices that increase the likelihood that the information generated from the evaluation is used and on those who will use the information </li></ul><ul><li>Specifically, this module will highlight the Centers for Disease Control (CDC) Evaluation Framework. The CDC framework merges several approaches designed to promote use and includes steps that are common to most evaluation models </li></ul>
    12. 12. www.cdc.gov/eval
    13. 13. The six steps of the CDC Evaluation Model are representative of components of most program evaluation models. There are six basic steps to program evaluation, all of which are related. The first steps provide a foundation for the later ones.
    14. 14. <ul><li>Engage stakeholders </li></ul><ul><li>Describe program </li></ul><ul><li>Evaluation purpose & questions (focus the evaluation) </li></ul><ul><li>Design & data collection </li></ul><ul><li>Analyze & draw evaluative conclusions </li></ul><ul><li>Communicate progress & findings </li></ul>
    15. 15. <ul><li>Most program efforts involve partners, all of whom are stakeholders in the effort. </li></ul><ul><li>Stakeholders must be involved in the evaluation; without their involvement, an evaluation may lack important information or be less useful. </li></ul>
    16. 16. <ul><li>Engagement of Stakeholders is essential to successful evaluation. </li></ul><ul><li>Engagement assures that evaluation is useful and credible. It clarifies roles and responsibilities and avoids real or perceived conflicts of interest. </li></ul><ul><li>Successful engagement further enhances cultural competence and increases protection for human subjects who may be involved. </li></ul>
    17. 17. <ul><li>Stakeholders… </li></ul><ul><li>Bring a variety of valuable & informative perspectives </li></ul><ul><li>Add credibility to the evaluation </li></ul><ul><li>May have resources to help </li></ul><ul><li>May be critical to implementing or advocating for action based on findings </li></ul><ul><li>Also engaging stakeholders can help… </li></ul><ul><li>Everyone learn more about evaluation and the program </li></ul><ul><li>Build trust and understanding among program constituents </li></ul>
    18. 18. <ul><li>There are strategies common to successful stakeholder engagement. These include but are not limited to: </li></ul><ul><li>Consulting program insiders (e.g., leaders, staff, participants, community members, and program funding sources) and outsiders (e.g., skeptics). </li></ul><ul><li>Taking special effort to promote the inclusion of less powerful groups or individuals. </li></ul><ul><li>Effective engagement requires that evaluators foster input, participation, and power-sharing among those who have an investment in the evaluation and the findings. </li></ul>
    19. 19. <ul><li>People who have a vested interest in evaluation findings can be categorized into three levels: </li></ul><ul><ul><li>Primary stakeholders are direct users of evaluation finding and may use them to alter a program’s course </li></ul></ul><ul><ul><li>Secondary stakeholders will likely be affected be affected by programmatic changes that result from evaluation. </li></ul></ul><ul><ul><li>Tertiary stakeholders may have interest in the evaluation findings, but will not be directly impacted </li></ul></ul>
    20. 20. <ul><li>Who are typical stakeholders in program evaluation? </li></ul><ul><li>Those involved in program operations (e.g., sponsors, collaborators, partners, funding officials, administrators, managers, and staff). </li></ul><ul><li>• Those served or affected by the program (e.g., students, faculty, neighborhood organizations, other academic institutions, elected officials, advocacy groups, professional associations, skeptics). </li></ul><ul><li>• Those who will be the primary consumers of information produced by the evaluation </li></ul>
    21. 21. Who is… Affected by the program? Involved in program operations? Intended users of evaluation findings? Who do we need to… Enhance credibility? Implement program changes? Advocate for changes? Fund, authorize, or expand the program? To identify stakeholders, we should ask the following questions:
    22. 22. <ul><li>Engage stakeholders </li></ul><ul><li>Describe program </li></ul><ul><li>Evaluation purpose & questions (focus the evaluation) </li></ul><ul><li>Design & data collection </li></ul><ul><li>Analyze & draw evaluative conclusions </li></ul><ul><li>Communicate progress & findings </li></ul>
    23. 23. <ul><li>The second step in prevention program evaluation is to Describe the Program in clear measurable terms. A strong prevention program description is important because it can: </li></ul><ul><li>Establish clarity & consensus about program activities & intended effects </li></ul><ul><li>Identify “holes” or problems in the program early on </li></ul><ul><li>Document agreement about how smaller components of the program fit into the larger picture </li></ul><ul><li>Articulate & document common vision </li></ul>
    24. 24. <ul><li>The program description details the mission and objectives of the program. </li></ul><ul><li>Descriptions should help evaluators understand the program goals and strategies. </li></ul><ul><li>Stakeholders should review and agree with the program description. </li></ul><ul><li>Program descriptions will vary for each evaluation. </li></ul>
    25. 25. <ul><li>An effective Program Description should include a strong statement of need that describes the problem that the program addresses. This includes: </li></ul><ul><ul><li>Expected effects are what the program must do to be successful. </li></ul></ul><ul><ul><li>Program activities are what the program does to effect change. </li></ul></ul><ul><ul><li>Resources include the time, talent, technology, equipment, information, money, and other assets available to conduct program activities. </li></ul></ul><ul><ul><li>The program’s stage of development reflects its maturity. </li></ul></ul><ul><ul><li>The context should describe the setting within which the program operates. </li></ul></ul><ul><ul><li>Some programs have used a logic model as a planning tool. </li></ul></ul>
    26. 26. <ul><li>Why does your program exist? What is the need? </li></ul><ul><li>Who is engaged in and affected by the program? </li></ul><ul><li>What do you want to change as a result of your efforts? </li></ul><ul><li>What activities do you perform as part of your program? </li></ul><ul><li>Who develops & performs these activities? </li></ul><ul><li>Who funds these activities? </li></ul><ul><li>What is the context in which your program operates? </li></ul>Program Theories can help with this… To develop a program description ask the following questions:
    27. 27. Program Theory <ul><li>A systematic configuration of stakeholders’ prescriptive assumptions (what actions must be taken) and descriptive assumptions (what causal processes are expected to happen) underlying programs , whether explicit or implicit assumptions. </li></ul><ul><li>Chen, p. 136, Evaluation Roots , 2004 </li></ul>
    28. 28. Why Develop a Program Theory <ul><li>Theory-based evaluations help to get at the why and how of program success or failure. </li></ul><ul><li>Weiss, p. 158, Evaluation Roots , 2004 </li></ul><ul><li>Why do people expect the program to work? </li></ul><ul><li>How do they expect to accomplish its ends? </li></ul><ul><li>What do they expect to happen and through what sequence of microsteps? </li></ul>
    29. 29. Theory of Change <ul><li>Identifies the causal processes generated by a program and through which a given type of social change is expected to occur </li></ul><ul><ul><li>Goals and outcomes (needs and measurable results) </li></ul></ul><ul><ul><li>Determinants (leverage to meet a need) </li></ul></ul><ul><ul><li>Intervention or treatment (agent of change) </li></ul></ul>
    30. 30. Theory of Action <ul><li>Maps out a systematic plan or specific pathway in a theory of change to deal with the causal processes that are expected to happen to attain program goals </li></ul><ul><ul><li>Arranging staff </li></ul></ul><ul><ul><li>Resources </li></ul></ul><ul><ul><li>Settings </li></ul></ul><ul><ul><li>Support organizations </li></ul></ul>
    31. 31. Logic Model <ul><li>Systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan, and the changes or results you hope to achieve. </li></ul><ul><li>The W.K.Kellogg Foundation Logic Model Development Guide </li></ul><ul><li>Graphical depiction of a program theory </li></ul>
    32. 32. Definition of a Logic Model The program logic model is defined as a picture of how your organization does its work – the theory and assumptions underlying the program. A program logic model links outcomes (both short- and long-term) with program activities/processes and the theoretical assumptions/principles of the program. - The W.K.Kellogg Foundation Logic Model Development Guide
    33. 33. <ul><li>Logic Models are common tools used to describe programs. </li></ul><ul><li>Logic models provide a “Road Map” of a program. </li></ul><ul><ul><li>Drawing a picture of expected program achievements and how the achievements will be realized. </li></ul></ul><ul><ul><li>Creating a visual of relationships hypothesized to exist between the program activities and the intended program effects </li></ul></ul><ul><li>Logic Models describe expectations/intentions of a program. </li></ul>
    34. 34. Why Use a Logic Model <ul><li>Identify what a program does and intends to do </li></ul><ul><li>Indicate if there is a disjuncture between what a program intends to do and what a program is actually doing </li></ul><ul><li>Point out strengths and weaknesses in a program </li></ul><ul><li>Allow stakeholders to tryout possible scenarios </li></ul><ul><li>The visual representation allows flexibility and provides the opportunity to adjust approaches and change course as plans develop </li></ul>
    35. 35. Components of a Logic Model <ul><li>Two components: </li></ul><ul><li>Your Planned Work: Resources (inputs) and Activities </li></ul><ul><li>Your Intended Results: Outputs, Outcomes, and Impact </li></ul>
    36. 36. External Factors/Context: Description of environment in which program takes place Assumptions: The underlying assumptions that influence the program’s design, implementation or goals Intermediate Long term Short term Resources/Inputs Resources needed to achieve program’s objectives Activities What the program does with resources to meet objectives Outputs Direct products of program activities Outcomes Changes that result from the program’s activities and outputs
    37. 37. <ul><li>The next slide is a logic model of an education program designed to prevent HIV infection in Native Americans. </li></ul><ul><li>While this program does not explicitly address mental health outcomes, the evaluation approach is consistent with mental health prevention program evaluation. </li></ul>
    38. 39. <ul><li>To create a logic model & graphic use the following steps: </li></ul><ul><li>Move from right to left answering the following questions: </li></ul><ul><ul><li>What do I want to accomplish with this program? </li></ul></ul><ul><ul><li>What changes do I expect to see from this program? </li></ul></ul><ul><ul><li>In whom or what will these changes occur? </li></ul></ul><ul><ul><li>What do we do to affect change? </li></ul></ul><ul><li>Then revise, refine, and more precisely describe and visually depict the relationships among components </li></ul><ul><li>Connect components with arrows…to show flow </li></ul><ul><li>Describe the context in which your program resides & operates </li></ul><ul><li>Revise and refine again until its “just right” – remember this is an iterative process </li></ul>
    39. 40. Activities Outputs Inputs Short-term Outcomes Intermediate Long-term Assumptions External Factors/Context
    40. 41. Program Activities Short-term Outcomes Intermediate Long-term PSA “ Tough Classes” Target Audience: 8th - 10th grade students nationwide Assumptions: (1) students have access to computers and the internet (to view PSA and related materials), and (2) attitudes are the primary determinants of course-taking behaviors  knowledge of classes that prepare for college students view taking tough classes as cool, rebellious  taking of college prep courses  college eligibility college application and acceptance  preparedness  knowledge of resources related to college access  utilization of resources (including campaign website)  college achievement
    41. 42. Sample Logic Model Framework source: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html
    42. 43. Example Logic Model 1
    43. 44. Example Logic Model 2
    44. 45. Using a Logic Model to Inform the Evaluation <ul><li>Provides a framework for the evaluation </li></ul><ul><li>Helps prioritize activities </li></ul><ul><li>Keeps focus on stakeholders’ questions </li></ul><ul><li>- The W.K.Kellogg Foundation Logic Model Development Guide </li></ul>
    45. 46. What Type of Question Will be Answered? Formative Evaluation (Improve) Summative Evaluation (Prove) Periodic reports Share quickly Demonstrate results to stakeholders Monitor Progress Mid-course corrections Intermediate outcomes and impact Determine value and worth based on results Helps to bring suggestions for improvement to the attention of the staff Describes quality and effectiveness by documenting impact
    46. 47. What is Going to be Evaluated?
    47. 48. Focus Area Indicators How to Evaluate Influential Factors Resources Activities Outputs Outcomes & Impacts
    48. 49. <ul><li>Engage stakeholders </li></ul><ul><li>Describe program </li></ul><ul><li>Evaluation purpose & questions (focus the evaluation) </li></ul><ul><li>Design & data collection </li></ul><ul><li>Analyze & draw evaluative conclusions </li></ul><ul><li>Communicate progress & findings </li></ul>
    49. 50. <ul><li>There are many reasons for doing an evaluation. Sometimes the reason is explicit & transparent. Sometimes it isn’t. </li></ul><ul><li>It is very important that the evaluation's purpose is clear. </li></ul><ul><li>This step in the evaluation process addresses the question: </li></ul><ul><li>Why is this evaluation being conducted? </li></ul><ul><li>For example: </li></ul><ul><ul><li>Will this evaluation be used to make improvements to the prevention program? </li></ul></ul><ul><ul><li>Or will the evaluation be used to make a decision about the program’s expansion or elimination? </li></ul></ul><ul><ul><li>Or will it be used for both? </li></ul></ul><ul><li>Engaging stakeholders in focusing the prevention program evaluation purpose and questions to be addressed as part of the study will help to ensure your study will be relevant, meaningful, and culturally responsive. </li></ul>
    50. 51. <ul><li>There are different types of evaluation questions: </li></ul><ul><li>Formative and Summative questions refer to the purpose of the study </li></ul><ul><li>Process and Outcome questions refer to the phase of the program being studied </li></ul>
    51. 52. <ul><li>Formative questions are intended to generate information for program improvement. They are designed to assist in program development. </li></ul><ul><li>Summative questions are intended to generate information for decisions about program adoption, continuation, or expansion. They are d esigned to assist with decisions about whether to continue or end a program, extend it or cut it back. </li></ul>
    52. 53. <ul><li>OUTCOME </li></ul><ul><li>Measures effects, results, impact on participants </li></ul><ul><ul><li>These can be intended or unintended </li></ul></ul><ul><li>Asks the questions : What was the impact? and Compared to what? </li></ul><ul><li>How does this look in the short-term and long-term? </li></ul><ul><li>PROCESS </li></ul><ul><li>Examines what is going on with this program </li></ul><ul><li>Describes what the program is doing, by whom and for whom </li></ul><ul><li>Asks the questions : What produced the outcomes and Why? </li></ul><ul><li>How did/does it work? </li></ul><ul><li>How was it implemented? </li></ul>Process questions ask about the operation of the program. Outcome questions ask about how the program is affecting recipients.
    53. 54. <ul><li>As stated earlier, effective evaluation is respectful of, involves, values and is valuable to stakeholders. In this step, it is necessary to define a purpose for the evaluation and to ensure that the questions being addressed by the study are important to stakeholders. </li></ul><ul><li>What types of things do stakeholders typically want to know about? </li></ul><ul><ul><li>Whether program is implemented as intended? </li></ul></ul><ul><ul><li>How is this program effecting the recipients? </li></ul></ul><ul><ul><li>Are the intended changes occurring? </li></ul></ul><ul><ul><li>If the “bang is worth the buck”? </li></ul></ul>
    54. 55. <ul><li>An evaluation requires the right number of questions. Too few makes an evaluation vulnerable to being weak, not worth it. Too many questions can overwhelm evaluation resources and confuse the final results. </li></ul><ul><li>An evaluation requires one clear purpose statement, and a set of 3-12 clearly worded evaluation questions </li></ul><ul><li>FOR EXAMPLE: The purpose of this evaluation is to explore the reasons why participants successfully complete a Summer Bridge Program. </li></ul><ul><li>How many & for what reasons do participants attend a Summer Bridge Program? </li></ul><ul><li>What factors contribute to participants completing the Summer Bridge Program? </li></ul><ul><li>What factors are in place in participants’ lives at Program completion that lead to continued success? </li></ul>
    55. 56. <ul><li>Encourage open sharing of interests and potential questions. </li></ul><ul><li>Sort through and prioritize questions with stakeholders. </li></ul><ul><li>Try to clarify questions so that they are measureable and meaningful. </li></ul><ul><li>Address questions that stakeholders don’t value. </li></ul><ul><li>Immediately “write-off” questions that seem unanswerable. </li></ul>When narrowing the list of potential evaluation questions to be addressed in a study, the evaluator… shouldn’t:
    56. 57. <ul><li>Listen to stakeholders’ needs and interests. </li></ul><ul><li>Help stakeholders to articulate the questions to be answered. </li></ul><ul><li>Foster stakeholders’ “ownership” of the evaluation. </li></ul><ul><li>Monopolize the evaluation. </li></ul><ul><li>Decide on the evaluation questions without stakeholder input. </li></ul><ul><li>Treat this program like all other programs. </li></ul>When narrowing the list of potential evaluation questions to be addressed in a study, the evaluator… shouldn’t:
    57. 58. <ul><li>As evaluators and stakeholders gain experience they are able to ask more specific questions that better meet their needs. Some examples follow: </li></ul><ul><ul><li>How would a sound answer to this question help the program? </li></ul></ul><ul><ul><li>What is the stage of development of this program? </li></ul></ul><ul><ul><li>How intensive is this program? What is the “dosage”? </li></ul></ul><ul><ul><li>Is this a “need to know” or “nice to answer” question? </li></ul></ul><ul><ul><li>When do we need an answer to this question for it to be useful? </li></ul></ul><ul><ul><li>What level of resources are available to do this evaluation? Will answers to this question be credible? </li></ul></ul>
    58. 59. <ul><li>Engage stakeholders </li></ul><ul><li>Describe program </li></ul><ul><li>Evaluation purpose & questions (focus the evaluation) </li></ul><ul><li>Design & data collection </li></ul><ul><li>Analyze & draw evaluative conclusions </li></ul><ul><li>Communicate progress & findings </li></ul>
    59. 60. <ul><li>The fourth step in program evaluation is to determine how to produce answers to the evaluation questions. </li></ul><ul><li>Two major steps to produce these answers: </li></ul><ul><ul><li>Decide on an evaluation design </li></ul></ul><ul><ul><li>Decide on data collection methods </li></ul></ul><ul><li>Evaluation questions should drive design and data collection methods - not the other way around. </li></ul>
    60. 61. Is random assignment used? Randomized or true experiment yes Is there a control group or multiple measures? no Quasi-experiment Non-experiment yes no Obtained from the Research Methods Knowledge Base: http://www.socialresearchmethods.net/kb/destypes.php Evaluation designs can be divided into 3 categories or types: Randomized or true experimental design, Quasi-Experimental design and non-experimental design. Decisions about whether or not to use a “control group” and how individual’s are assigned to intervention and control groups determines the category or type of evaluation design.
    61. 62. <ul><li>Experimental </li></ul><ul><li>When answering the question: Did my program (x) cause this specific impact on participants (y)? </li></ul><ul><li>Only possible when random assignment is feasible </li></ul><ul><li>Random assignment is often not possible in local level prevention program evaluation & thus this is not a commonly used design in this context </li></ul><ul><li>Non-experimental </li></ul><ul><li>When answering questions that are observational or , descriptive in nature. For example: </li></ul><ul><li>How is our program being implemented? </li></ul><ul><li>This is one of the most common forms of designs used in local level evaluation studies. However, it isn't a particularly good choice for addressing cause-effect questions that examine the relationship between program and its outcomes. </li></ul><ul><li>Quasi-experimental </li></ul><ul><li>When answering the question: How does my program look when compared to…? </li></ul><ul><li>These designs are used when an implementing an experimental design is not possible </li></ul>
    62. 63. <ul><li>Experimental designs randomly assign persons to either an intervention or a non-intervention group. </li></ul><ul><li>Random assignment helps to ensure that the two groups similar in their composition, with the exception that one group receives the intervention and the other receives either “business as usual” or no program at all. </li></ul><ul><li>When the intervention is complete, differences between the two groups in the outcome of interest are measured. If we find a difference, it was likely caused by the intervention. </li></ul><ul><li>Randomly assigning participants to either a Summer Bridge program or a general campus orientation program, to determine if participation in the Summer Bridge program caused an increase in retention and success is an example of study using an experimental design. </li></ul>
    63. 64. <ul><li>Quasi-experimental designs are similar to experimental designs, except that persons are not randomly assigned to a group. </li></ul><ul><li>This type of method is used when randomization is not feasible. </li></ul><ul><li>Studying whether after program participation, student engagement is significantly higher for those who participated in the program when compared to persons who did not participate in the program (and persons were not assigning randomly to the program) is an example of a quasi-experimental design. </li></ul>
    64. 65. <ul><li>Non-experimental designs tend to be descriptive and attempt to understand differences, similarities, and processes within a group. </li></ul><ul><li>Reporting the number of students who transfer to the UC and CSU is example of an application of a non-experimental design. </li></ul>
    65. 66. <ul><li>Experimental </li></ul><ul><li>Post test only with control group </li></ul><ul><li>Pretest-posttest with control group </li></ul><ul><li>Random assignment to groups is required condition when implementing an experimental design </li></ul><ul><li>Non-experimental </li></ul><ul><li>Case study </li></ul><ul><li>Post test only </li></ul><ul><li>Retrospective pretest </li></ul><ul><li>Cross-sectional surveys </li></ul><ul><li>Quasi-experimental </li></ul><ul><li>Pretest-posttest </li></ul><ul><li>Time series </li></ul><ul><li>Regression discontinuity </li></ul>
    66. 67. <ul><li>The choice of design has implications for what will be used as evidence. Evidence comprises the various forms of data that we use to support our evaluation claims. </li></ul><ul><li>The design will determine how evidence will be gathered and what kind of claims can be made. </li></ul><ul><li>The design also determines how data sources will be selected, what data collection instruments will be used, who will collect the data, and what data management systems will be needed. </li></ul>
    67. 68. <ul><li>The other major consideration in answering evaluation questions is data collection </li></ul><ul><li>There are many ways to collect data, for example: </li></ul><ul><ul><li>Surveys </li></ul></ul><ul><ul><li>Interviews </li></ul></ul><ul><ul><li>Focus groups </li></ul></ul><ul><ul><li>Observations </li></ul></ul><ul><ul><li>Document review </li></ul></ul><ul><ul><li>Tests (assessments) </li></ul></ul>
    68. 69. <ul><li>There are other important considerations for data collection that will impact the quality of data, use of evaluation resources & the timeliness of evaluation information. Some considerations include: </li></ul><ul><ul><li>Who will collect the data? </li></ul></ul><ul><ul><li>To what extent do data collectors require training? </li></ul></ul><ul><ul><li>Pilot testing instruments, checking reliability & validity </li></ul></ul><ul><ul><li>When does the data need to be collected? </li></ul></ul><ul><ul><li>Resources available?– Staff, money, technology </li></ul></ul>
    69. 70. <ul><li>Use of existing data may reduce or eliminate the need to collect new data for evaluation. </li></ul><ul><li>Explore existing data sources, for example… </li></ul><ul><ul><li>Administrative data (Cal Pass) </li></ul></ul><ul><ul><li>Campus data </li></ul></ul><ul><ul><li>Etc. </li></ul></ul><ul><li>However, it is important not to tailor evaluation questions towards data just because it is readily available. Evaluation questions must drive data collection. </li></ul>
    70. 71. <ul><li>Document your data collection strategy. An example follows: </li></ul>Evaluation Questions Data Collection Method Source Data Type <ul><ul><li>To what extent did the intended target audience receive the message? </li></ul></ul>Survey Random sample of households in county X Quantitative For those who didn’t receive messages, why? Survey Random sample of households in county X Qualitative
    71. 72. <ul><li>Care must be taken to design an evaluation that generates credible information that stakeholders perceive as culturally relevant, trustworthy & valuable </li></ul><ul><li>Evaluators should strive to collect information that will convey a well-rounded picture of the program & be seen as credible by the evaluation’s users </li></ul><ul><li>When stakeholders find evaluation data and designs to be credible, they are more likely to accept the findings & to act on its recommendations </li></ul>
    72. 73. <ul><li>Credibility can be improved by using multiple procedures (i.e., data collection strategies) and by involving stakeholders in determining what data should be gathered. </li></ul><ul><li>The following factors affect people’s perceptions of the credibility of your evaluation evidence: </li></ul><ul><li>indicators, sources, quality, quantity, and logistics. </li></ul>
    73. 74. <ul><li>Indicators are aspects of the program that can be examined </li></ul><ul><li>to address the questions of the evaluation. </li></ul><ul><li>Examples of indicators that can be defined and tracked include the program’s capacity to deliver services, the participation rate, levels of client satisfaction, the efficiency of resource use, and the amount of intervention exposure. </li></ul><ul><li>Other measures of program effects, such as changes in participant behavior, community norms, policies or practices, and the settings or environment around the program, can also be tracked. </li></ul>
    74. 75. <ul><li>Sources of evidence are persons, documents, or observations. </li></ul><ul><li>More than one source might be used to gather </li></ul><ul><li>evidence. </li></ul><ul><li>Use of multiple sources provides different perspectives. </li></ul>
    75. 76. <ul><li>Quality refers to the correctness and integrity of the information. </li></ul><ul><li>Quality data are representative of what they intend to measure and are informative for their intended use. </li></ul><ul><li>Instrument design, data-collection procedures, training of data collectors, source selection, coding, data management, and routine error-checking all influence the quality of your data. </li></ul>
    76. 77. <ul><li>Quantity refers to the amount of evidence gathered. The amount of information needed should be estimated in advance. </li></ul><ul><li>All evidence collected should have a clear and anticipated use, with only minimal burden placed on respondents. </li></ul><ul><li>Another example of quantity would involve determining how many persons must provide information to adequately address the evaluation question. The burden on persons to provide information should always approach the minimum needed. </li></ul>
    77. 78. <ul><li>Logistics encompass the methods, timing, and physical </li></ul><ul><li>infrastructure for gathering and handling evidence. </li></ul><ul><li>Each technique selected for gathering evidence must be suited to the source(s), analysis plan, and strategy for communicating findings. </li></ul><ul><li>Cultural issues should influence decisions about acceptable ways of asking questions and collecting information. </li></ul><ul><li>Procedures for gathering evidence should be sensitive to cultural conditions in each setting and must ensure that the privacy and confidentiality of the information and sources are protected. </li></ul>
    78. 79. <ul><li>Engage stakeholders </li></ul><ul><li>Describe program </li></ul><ul><li>Evaluation purpose & questions (focus the evaluation) </li></ul><ul><li>Design & data collection </li></ul><ul><li>Analyze & draw evaluative conclusions </li></ul><ul><li>Communicate progress & findings </li></ul>
    79. 80. <ul><li>While designing the evaluation, decide upon what data will be analyzed </li></ul><ul><li>Early consideration of data analysis will help to assure the “right” data are collected </li></ul><ul><li>Early discussions of data analysis can stimulate conversations with stakeholders about “indicators” that will be helpful to have available when answering questions (criteria) & what constitutes “good” performance on these “indicators” (standards) </li></ul>
    80. 81. <ul><li>Once data have been collected and analyzed, stakeholders must decide what to do with this information. That is, state findings. Common next steps include: </li></ul><ul><ul><li>Examine against agreed upon standards </li></ul></ul><ul><ul><li>Think about what this examination means: </li></ul></ul><ul><ul><ul><li>With respect to practical significance (interpret) </li></ul></ul></ul><ul><ul><ul><li>Assign claims of merit, worth, or significance (judge) </li></ul></ul></ul><ul><ul><li>Recommend actions that should be considered based upon findings </li></ul></ul>
    81. 82. <ul><li>Consider the following when developing conclusions from analysis of evaluative data. </li></ul><ul><li>Use culturally and methodologically appropriate methods of analysis and synthesis to summarize findings; </li></ul><ul><li>Interpret the significance of results for deciding what the findings mean; </li></ul><ul><li>Make judgments according to clearly stated values that classify a result (e.g., as positive or negative and high or low); </li></ul><ul><li>Consider alternative ways to compare results (e.g., compared with program objectives, a comparison group, national norms, past performance, or needs); </li></ul><ul><li>Generate alternative explanations for findings and indicate why these explanations should be discounted; </li></ul><ul><li>Recommend actions or decisions that are consistent with the conclusions; </li></ul><ul><li>Limit conclusions to situations, time periods, persons, contexts, and purposes for which the findings are applicable. </li></ul>
    82. 83. <ul><li>Engage stakeholders </li></ul><ul><li>Describe program </li></ul><ul><li>Evaluation purpose & questions (focus the evaluation) </li></ul><ul><li>Design & Data collection </li></ul><ul><li>Analyze & Draw Evaluative Conclusions </li></ul><ul><li>Communicate progress & findings </li></ul>
    83. 84. <ul><li>Disseminating information about the progress and findings of the evaluation is the final step in the process; however, communication is an important function throughout the evaluation process from beginning to end. </li></ul><ul><li>Effective communication ensures that evaluators and program staff benefit from participants’ perspectives on program and evaluation goals and priorities and maximizes the cultural relevance, value, and use of the evaluation. </li></ul><ul><li>As with data analysis, communications about the evaluation should not be an afterthought. Communicating throughout the evaluation is important to facilitating use of findings </li></ul><ul><li>Aspects of communications to consider include: </li></ul><ul><ul><li>Purpose of communication </li></ul></ul><ul><ul><li>Intended audiences </li></ul></ul><ul><ul><li>Format </li></ul></ul><ul><ul><li>Frequency & timing </li></ul></ul><ul><ul><li>Deadlines </li></ul></ul>
    84. 85. <ul><li>Communication has multiple purposes that include: </li></ul><ul><li>Promoting collaborative decision making about evaluation design and activities </li></ul><ul><li>Informing those directly involved about specific upcoming evaluation activities </li></ul><ul><li>Keeping those directly and indirectly involved informed about evaluation progress </li></ul><ul><li>Presenting & sometimes collaboratively interpreting initial/interim findings </li></ul><ul><li>Presenting & sometimes collaboratively interpreting final findings </li></ul>
    85. 86. <ul><li>Consider each type of evaluation stakeholder & their cultural values when designing and delivering project communication. </li></ul><ul><li>Tailor your communications to the cultural norms and needs of the particular audience for which they are intended. </li></ul>
    86. 87. <ul><li>Consider the culture & characteristics of audiences when designing and delivering communication: </li></ul><ul><ul><li>Accessibility- Be sure that your intended audience has access to the communication – e.g., if posting communications on a webpage be sure your audience has regular access to a computer with internet access </li></ul></ul><ul><ul><li>Reading ability- Gear written communications to the average reading level for the stakeholder audience for which you are targeting </li></ul></ul><ul><ul><li>Familiarity with the program and or the evaluation- Be sure to explain your program & its evaluation in communications with audiences that may not be familiar with your work </li></ul></ul><ul><ul><li>Role in decision making- Consider whether this audience will be using the information in the communication for decision-making & tailor your communication to these needs </li></ul></ul><ul><ul><li>Familiarity with research and evaluation methods- Consider whether your audience will want a description of your methods & if not, perhaps include as an appendix </li></ul></ul><ul><ul><li>Experience using evaluation findings- Consider engaging in strategies that help to promote use & the extent to which your intended audience may be (un)familiar with thinking about and using evaluation findings in decision making </li></ul></ul>
    87. 88. <ul><li>Verbal Presentations </li></ul><ul><li>Written reports </li></ul><ul><li>Executive Summaries </li></ul><ul><li>Websites </li></ul><ul><li>Poster Sessions </li></ul><ul><li>Newsletters </li></ul><ul><li>Memos </li></ul><ul><li>Postcards </li></ul><ul><li>Email </li></ul><ul><li>Personal discussions </li></ul><ul><li>Working sessions </li></ul>The audience and purpose of the communication activities will also determine their formats. Communication formats can fall into two categories – formal and informal. Informal:
    88. 89. Most Interactive with Audience Least Interactive with Audience <ul><li>Working Sessions </li></ul><ul><li>Impromptu or planned meetings with individuals </li></ul><ul><li>Verbal presentations </li></ul><ul><li>Videotape or computer-generated presentations </li></ul><ul><li>Posters </li></ul><ul><li>Internet communications </li></ul><ul><li>Memos & postcards </li></ul><ul><li>Comprehensive written reports </li></ul><ul><li>Executive summaries </li></ul><ul><li>Newsletters, bulletins, brochures </li></ul><ul><li>Newsmedia communications </li></ul>The level of interactivity with the audience will be impacted by the format of communication.
    89. 90. <ul><li>Document your plan for communications </li></ul>As with evaluation design and data collection it is helpful to document the evaluation communication plan, detailing the purpose, formats, timelines and other critical information. Audience: Parents & youth at the pilot sites Purpose Possible formats Timing/Dates Notes ✓ Include in decision making about evaluation design/activities Personal discussions/meetings Early April Pilot parent survey ✓ Inform about specific upcoming evaluation activities Article included in site newsletter Early September Keep informed about progress of the evaluation
    90. 91. <ul><li>Evaluation is a systematic process of inquiry </li></ul><ul><li>It is imperative to have a clear plan & direction for the evaluation </li></ul><ul><li>Evaluation must provide accurate information </li></ul><ul><li>Evaluation must provide usable information </li></ul><ul><li>The inclusion of stakeholders throughout the evaluation process is critical to planning and conducting a successful evaluation that is culturally responsive and relevant </li></ul><ul><li>Use is necessary for evaluation to be a tool for organizational, policy and social change </li></ul><ul><li>Useful evaluation processes and results inform decisions, clarify options, identify strengths and weaknesses, and provide information on program improvements, policies, and key contextual factors affecting the program and its participants </li></ul>
    91. 92. <ul><li>Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR 1999; 48 (No. RR-11). </li></ul><ul><li>Fitzpatrick, J.L., Sanders, J.R., & Worthen, B.R. (2004). Program Evaluation: Alternative Approaches and Practical Guidelines (3 rd Ed.). Boston: Pearson. </li></ul><ul><li>Patton, M. (1997). Utilization-focused evaluation . Thousand Oaks, CA: Sage. </li></ul><ul><li>Russ-Eft,D. & Preskill, H. (2001). Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance, & Change. New York: Basic Books. </li></ul><ul><li>Scriven, M. (1991). Evaluation thesaurus . Thousand Oaks, CA: Sage </li></ul><ul><li>SPAN USA, Inc. (2001). Suicide Prevention: Prevention Effectiveness and Evaluation. SPAN USA, Washington, DC. </li></ul><ul><li>Rossi, P., Lipsey, M., Freeman H. (2004) . Evaluation: A systematic approach . Thousand Oaks, CA: Sage. </li></ul><ul><li>Trochim, M.K. (2006). Research Methods Knowledge Base . http://www.socialresearchmethods.net/kb/index.php </li></ul><ul><li>U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005. </li></ul><ul><li>The author would like to thank Leslie Fierro for sharing her work and ideas in developing this presentation. </li></ul>