Evaluation refers to the process of determining the merit, worth, or value of something, or the product of that process. (Scriven, 1991, p. 139)
Program evaluation is the use of social research methods to systematically investigate the effectiveness of social intervention programs. (Rossi, Lipsey, Freeman, 2004, p. 28)
Program evaluation is the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming. (Patton, 1997, p. 23)
The purpose and design of a program evaluation can answer different questions and provide a broad range of information. Some examples include:
Learning whether proposed program materials are suitable
Learning whether program plans are feasible
Providing an early warning system for problem identification
Learning whether programs are producing the desired results
Learning whether programs have unexpected benefits or
Enabling managers to improve program processes
Monitoring progress toward the program’s goals
Producing data on which to base future programs
Demonstrating the effectiveness of the program
Identifying the most effective parts of the program for refinement
Gathering valuable information that can be shared
Evaluation, like anything else, can be done well and produce benefits or it can be done poorly adding little value, sometimes having negative impact . The following table contrasts characteristics of effective and ineffective evaluation. Evaluation is… Evaluation is not… Done with you Done TO you Able to provide rich information Simply program monitoring Intended to be used Intended to sit on a shelf or to check a box For the program stakeholders For the evaluator or only for management Systematic Haphazard FUN! Scary (Really it isn’t!– you’ll see )
The theoretical framework that guides this module focuses on practices that increase the likelihood that the information generated from the evaluation is used and on those who will use the information
Specifically, this module will highlight the Centers for Disease Control (CDC) Evaluation Framework. The CDC framework merges several approaches designed to promote use and includes steps that are common to most evaluation models
The six steps of the CDC Evaluation Model are representative of components of most program evaluation models. There are six basic steps to program evaluation, all of which are related. The first steps provide a foundation for the later ones.
Who are typical stakeholders in program evaluation?
Those involved in program operations (e.g., sponsors, collaborators, partners, funding officials, administrators, managers, and staff).
• Those served or affected by the program (e.g., students, faculty, neighborhood organizations, other academic institutions, elected officials, advocacy groups, professional associations, skeptics).
• Those who will be the primary consumers of information produced by the evaluation
Who is… Affected by the program? Involved in program operations? Intended users of evaluation findings? Who do we need to… Enhance credibility? Implement program changes? Advocate for changes? Fund, authorize, or expand the program? To identify stakeholders, we should ask the following questions:
A systematic configuration of stakeholders’ prescriptive assumptions (what actions must be taken) and descriptive assumptions (what causal processes are expected to happen) underlying programs , whether explicit or implicit assumptions.
Systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan, and the changes or results you hope to achieve.
The W.K.Kellogg Foundation Logic Model Development Guide
Graphical depiction of a program theory
Definition of a Logic Model The program logic model is defined as a picture of how your organization does its work – the theory and assumptions underlying the program. A program logic model links outcomes (both short- and long-term) with program activities/processes and the theoretical assumptions/principles of the program. - The W.K.Kellogg Foundation Logic Model Development Guide
Your Planned Work: Resources (inputs) and Activities
Your Intended Results: Outputs, Outcomes, and Impact
External Factors/Context: Description of environment in which program takes place Assumptions: The underlying assumptions that influence the program’s design, implementation or goals Intermediate Long term Short term Resources/Inputs Resources needed to achieve program’s objectives Activities What the program does with resources to meet objectives Outputs Direct products of program activities Outcomes Changes that result from the program’s activities and outputs
Program Activities Short-term Outcomes Intermediate Long-term PSA “ Tough Classes” Target Audience: 8th - 10th grade students nationwide Assumptions: (1) students have access to computers and the internet (to view PSA and related materials), and (2) attitudes are the primary determinants of course-taking behaviors knowledge of classes that prepare for college students view taking tough classes as cool, rebellious taking of college prep courses college eligibility college application and acceptance preparedness knowledge of resources related to college access utilization of resources (including campaign website) college achievement
Sample Logic Model Framework source: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html
- The W.K.Kellogg Foundation Logic Model Development Guide
What Type of Question Will be Answered? Formative Evaluation (Improve) Summative Evaluation (Prove) Periodic reports Share quickly Demonstrate results to stakeholders Monitor Progress Mid-course corrections Intermediate outcomes and impact Determine value and worth based on results Helps to bring suggestions for improvement to the attention of the staff Describes quality and effectiveness by documenting impact
There are many reasons for doing an evaluation. Sometimes the reason is explicit & transparent. Sometimes it isn’t.
It is very important that the evaluation's purpose is clear.
This step in the evaluation process addresses the question:
Why is this evaluation being conducted?
Will this evaluation be used to make improvements to the prevention program?
Or will the evaluation be used to make a decision about the program’s expansion or elimination?
Or will it be used for both?
Engaging stakeholders in focusing the prevention program evaluation purpose and questions to be addressed as part of the study will help to ensure your study will be relevant, meaningful, and culturally responsive.
Formative questions are intended to generate information for program improvement. They are designed to assist in program development.
Summative questions are intended to generate information for decisions about program adoption, continuation, or expansion. They are d esigned to assist with decisions about whether to continue or end a program, extend it or cut it back.
As stated earlier, effective evaluation is respectful of, involves, values and is valuable to stakeholders. In this step, it is necessary to define a purpose for the evaluation and to ensure that the questions being addressed by the study are important to stakeholders.
What types of things do stakeholders typically want to know about?
An evaluation requires the right number of questions. Too few makes an evaluation vulnerable to being weak, not worth it. Too many questions can overwhelm evaluation resources and confuse the final results.
An evaluation requires one clear purpose statement, and a set of 3-12 clearly worded evaluation questions
FOR EXAMPLE: The purpose of this evaluation is to explore the reasons why participants successfully complete a Summer Bridge Program.
How many & for what reasons do participants attend a Summer Bridge Program?
What factors contribute to participants completing the Summer Bridge Program?
What factors are in place in participants’ lives at Program completion that lead to continued success?
The fourth step in program evaluation is to determine how to produce answers to the evaluation questions.
Two major steps to produce these answers:
Decide on an evaluation design
Decide on data collection methods
Evaluation questions should drive design and data collection methods - not the other way around.
Is random assignment used? Randomized or true experiment yes Is there a control group or multiple measures? no Quasi-experiment Non-experiment yes no Obtained from the Research Methods Knowledge Base: http://www.socialresearchmethods.net/kb/destypes.php Evaluation designs can be divided into 3 categories or types: Randomized or true experimental design, Quasi-Experimental design and non-experimental design. Decisions about whether or not to use a “control group” and how individual’s are assigned to intervention and control groups determines the category or type of evaluation design.
When answering the question: Did my program (x) cause this specific impact on participants (y)?
Only possible when random assignment is feasible
Random assignment is often not possible in local level prevention program evaluation & thus this is not a commonly used design in this context
When answering questions that are observational or , descriptive in nature. For example:
How is our program being implemented?
This is one of the most common forms of designs used in local level evaluation studies. However, it isn't a particularly good choice for addressing cause-effect questions that examine the relationship between program and its outcomes.
When answering the question: How does my program look when compared to…?
These designs are used when an implementing an experimental design is not possible
Experimental designs randomly assign persons to either an intervention or a non-intervention group.
Random assignment helps to ensure that the two groups similar in their composition, with the exception that one group receives the intervention and the other receives either “business as usual” or no program at all.
When the intervention is complete, differences between the two groups in the outcome of interest are measured. If we find a difference, it was likely caused by the intervention.
Randomly assigning participants to either a Summer Bridge program or a general campus orientation program, to determine if participation in the Summer Bridge program caused an increase in retention and success is an example of study using an experimental design.
Quasi-experimental designs are similar to experimental designs, except that persons are not randomly assigned to a group.
This type of method is used when randomization is not feasible.
Studying whether after program participation, student engagement is significantly higher for those who participated in the program when compared to persons who did not participate in the program (and persons were not assigning randomly to the program) is an example of a quasi-experimental design.
Indicators are aspects of the program that can be examined
to address the questions of the evaluation.
Examples of indicators that can be defined and tracked include the program’s capacity to deliver services, the participation rate, levels of client satisfaction, the efficiency of resource use, and the amount of intervention exposure.
Other measures of program effects, such as changes in participant behavior, community norms, policies or practices, and the settings or environment around the program, can also be tracked.
Quantity refers to the amount of evidence gathered. The amount of information needed should be estimated in advance.
All evidence collected should have a clear and anticipated use, with only minimal burden placed on respondents.
Another example of quantity would involve determining how many persons must provide information to adequately address the evaluation question. The burden on persons to provide information should always approach the minimum needed.
While designing the evaluation, decide upon what data will be analyzed
Early consideration of data analysis will help to assure the “right” data are collected
Early discussions of data analysis can stimulate conversations with stakeholders about “indicators” that will be helpful to have available when answering questions (criteria) & what constitutes “good” performance on these “indicators” (standards)
Disseminating information about the progress and findings of the evaluation is the final step in the process; however, communication is an important function throughout the evaluation process from beginning to end.
Effective communication ensures that evaluators and program staff benefit from participants’ perspectives on program and evaluation goals and priorities and maximizes the cultural relevance, value, and use of the evaluation.
As with data analysis, communications about the evaluation should not be an afterthought. Communicating throughout the evaluation is important to facilitating use of findings
Consider the culture & characteristics of audiences when designing and delivering communication:
Accessibility- Be sure that your intended audience has access to the communication – e.g., if posting communications on a webpage be sure your audience has regular access to a computer with internet access
Reading ability- Gear written communications to the average reading level for the stakeholder audience for which you are targeting
Familiarity with the program and or the evaluation- Be sure to explain your program & its evaluation in communications with audiences that may not be familiar with your work
Role in decision making- Consider whether this audience will be using the information in the communication for decision-making & tailor your communication to these needs
Familiarity with research and evaluation methods- Consider whether your audience will want a description of your methods & if not, perhaps include as an appendix
Experience using evaluation findings- Consider engaging in strategies that help to promote use & the extent to which your intended audience may be (un)familiar with thinking about and using evaluation findings in decision making
As with evaluation design and data collection it is helpful to document the evaluation communication plan, detailing the purpose, formats, timelines and other critical information. Audience: Parents & youth at the pilot sites Purpose Possible formats Timing/Dates Notes ✓ Include in decision making about evaluation design/activities Personal discussions/meetings Early April Pilot parent survey ✓ Inform about specific upcoming evaluation activities Article included in site newsletter Early September Keep informed about progress of the evaluation
It is imperative to have a clear plan & direction for the evaluation
Evaluation must provide accurate information
Evaluation must provide usable information
The inclusion of stakeholders throughout the evaluation process is critical to planning and conducting a successful evaluation that is culturally responsive and relevant
Use is necessary for evaluation to be a tool for organizational, policy and social change
Useful evaluation processes and results inform decisions, clarify options, identify strengths and weaknesses, and provide information on program improvements, policies, and key contextual factors affecting the program and its participants
Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR 1999; 48 (No. RR-11).
Fitzpatrick, J.L., Sanders, J.R., & Worthen, B.R. (2004). Program Evaluation: Alternative Approaches and Practical Guidelines (3 rd Ed.). Boston: Pearson.
Patton, M. (1997). Utilization-focused evaluation . Thousand Oaks, CA: Sage.
Russ-Eft,D. & Preskill, H. (2001). Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance, & Change. New York: Basic Books.
Scriven, M. (1991). Evaluation thesaurus . Thousand Oaks, CA: Sage
SPAN USA, Inc. (2001). Suicide Prevention: Prevention Effectiveness and Evaluation. SPAN USA, Washington, DC.
Rossi, P., Lipsey, M., Freeman H. (2004) . Evaluation: A systematic approach . Thousand Oaks, CA: Sage.
Trochim, M.K. (2006). Research Methods Knowledge Base . http://www.socialresearchmethods.net/kb/index.php
U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005.
The author would like to thank Leslie Fierro for sharing her work and ideas in developing this presentation.