The nature of program evaluation


Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

The nature of program evaluation

  1. 1. The Nature of Program Evaluation Carlo Magno, PhD Counseling and Educational Psychology Department
  2. 2. Answer the following questions: • Why is program evaluation needed? • What are the roles of a professional program evaluator?
  3. 3. Program evaluation is needed because… • Policy makers need good information about the relative effectiveness of the program. – Which programs are working well? – Which poorly? – What are the program’s relative cost and benefits? – Which parts of the program are working? – What can be done with those parts that are not working well? – Have al parts of the program been thought through carefully at the planning stage? – What is the theory or logic model for the program effectiveness? – What adaptations would make the program more effective?
  4. 4. Program Evaluation • Systematic investigation of the merit, worth or significance of an object (Scriven, 1999), hence assigning “value” to a program’s efforts means addressing those three inter-related domains: – Merit (or quality) – Worth (or value, i.e., cost-effectiveness) – Significance (or importance) • The identification, clarification, and application of defensible criteria to determine an object’s value in relation to those criteria (Fitzpatrick, WEorthen, & Sanders, 2004).
  5. 5. Prerequisite to evaluation • Need a program: - an organized action – Direct service interventions – Community mobilization efforts – Research initiatives – Surveillance systems – Policy development activities – Outbreak investigations – Laboratory diagnostics – Communication campaigns – Infrastructure building projects – Training and education services – Administrative systems
  6. 6. Inquiry and Judgment in Evaluation • (1) Determining standards for judging quality and deciding whether those standards should be relative or absolute. • (2) Collecting relevant information • (3) Applying the standards to determine value, quality, utility, effectiveness, or significance.
  7. 7. Evidence of value and judgement: • What will be evaluated? (i.e., what is "the program" and in what context does it exist?) • What aspects of the program will be considered when judging program performance? • What standards (i.e., type or level of performance) must be reached for the program to be considered successful? • What evidence will be used to indicate how the program has performed? • What conclusions regarding program performance are justified by comparing the available evidence to the selected standards? • How will the lessons learned from the inquiry be used to improve public health effectiveness?
  8. 8. Difference between Research and Evaluation • Purpose • Approaches • Who sets the agenda? • Generalizability of results • Criteria and standards • Preparation
  9. 9. Difference in Purpose • Research – Add knowledge in a field, contribute to theory – Seeks conclusion • Evaluation – Help those who hold a stake in whatever is being evaluated – Leads to judgments
  10. 10. Difference in Approaches • Research – Quest for laws – Explore and establish causal relationships • Evaluation – Describing a phenomenon may use causal relationships – Causal relationships will depend on the needs of the stakeholders
  11. 11. Difference on who sets the agenda • Research – The hypothesis investigated is chosen by the researcher and the appropriate steps in developing the theory. • Evaluation – Questions to be answered comes form many sources (stakeholders). – Consults with stakeholders to determine the focus of the study.
  12. 12. Difference in generalizability of results • Research – Methods are designed to maximize generalizability to many different settings • Evaluation – Specific to the context which evaluation object rests.
  13. 13. Difference in Criteria and standards • Research – Internal validity (causality), – external validity (generalizability) • Evaluation – Accuracy (corresponding to reality) – Utility (results serve practical information) – Feasibility (realistic, prudent, diplomatic, frugal) – Propriety (done legally and ethiocally)
  14. 14. Difference in Preparation • Research – In depth training on a single discipline in their field of inquiry. • Evaluation – Responds to the needs of clients and stakeholders with many information needs and operating in many different settings. – Interdisciplinary: Sensitive to a wide range of phenomenon that they must attend to. – Familiar with a wide variety of methods – Establish personal working relationships with clients (interpersonal and communication skills)
  15. 15. Competencies needed by professional Evaluators (Sanders, 1999) • Ability to describe the object and context of an evaluation • Conceptualize appropriate purposes and framework for evaluation • Identify and select appropriate evaluation questions, information needs, and sources of information • Select mans for collecting and analyzing information • Determine the value of the object of an evaluation • Communicate plans and results effectively to audiences • Manage the evaluation • Maintain ethical standards • Adjust to external factors influencing the evaluation • Evaluate the evaluation
  16. 16. Purposes of Evaluation • Talmage (1982) – Render judgment in the worth of the program – Assist decision makers responsible for deciding policy – Serve a political function • Rallis and Rossman (2000) – Learning, helping practitioners and others better understand and interpret their observations •
  17. 17. Purposes of Evaluation • Weiss (1988) and Henry (2000) – Bring about social betterment • Mark, Henry, and Julnes (1999) – Betterment – alleviation of social problems, meeting of human needs • Chelimsky (1997) – takes a global perspective: new technologies, demographic imbalance, environmental protection, sustainable development, terrorism, human rights
  18. 18. Purposes of Evaluation • House and Howe (1999) – Foster deliberate democracy-work to help less powerful stakeholders gain a voice and to stimulate dialogue among stakeholders in a democratic fashion. • Mark, Henry, and Julnes (1999) – Assessment of merit and worth – Oversight and compliance – Program and organizational improvement – Knowledge development
  19. 19. Roles of the Professional Evaluator • Rallis and Rossman (2000) – Critical friend: “someone the emperor knows and can listen to. She is more friend than judge, although she is not afraid to offer judgment” (p. 83) • Schwant (2001) – Helping practitioners develop critical judgment
  20. 20. Roles of the Professional Evaluator • Patton (1996) – Facilitator – Collaborator – Teacher management consultant – OD specialist – Social-change agent • Preskilll and Torres (1999) – Bring about organizational learning and instilling a learning environment
  21. 21. Roles of the Professional Evaluator • Mertens (1999), Chelimsky (1998), and Greene (1997) – Including the stakeholders as part of the evaluation process • House and Howe (1999) – Stimulating dialogue among various groups
  22. 22. Roles of the Professional Evaluator • Bickman (2001) and Chen (1990) – Take part in program planning – Help articulate program theories or logic model • Wholey (1996) – Help policy makers and managers select the performance dimension to be measured as well as the tools to use in measuring those dimensions
  23. 23. Roles of the Professional Evaluator • Lipsey (2000) – Provides expertise to track things down, systematically observe and measure them, and compare, analyze, and interpret with a good faith attempt at aobjectivity.
  24. 24. Roles of the Professional Evaluator • Fitzpatrick, Worthen, and Sanders (2004) – Negotiating with stakeholders group to define the purpose of evaluation – Developing contracts – Hiring and overseeing staff – Managing budgets – Identifying disenfranchised or underrepresented groups – Working with advisory panels – Collecting and analyzing and interpreting qualitative and quantitative information – Communicating frequently with various stakeholders to seek input into the evaluation and to report results – Writing reports – Considering effective ways to disseminate information – Meeting with the press and other representatives to report on progress and results – Recruiting others to evaluate the evaluation
  25. 25. Examples of evaluation use in Education • To empower teachers to have more say about how school budget are allocated • To judge the quality of the school curricula in specific content areas • To accredit schools that meet minimum accreditation standards • To determine the value of a middle school’s block scheduling • To satisfy an external funding agency’s demands for reports on effectiveness of school programs it supports • To assist parents and students in selecting schools in a district with school choice • To help teachers improve their reading program to encourage more voluntary reading
  26. 26. Examples of evaluation use in other public and Nonprofit sectors • To decide whether to implement an urban development program • To establish the value of a job-training program • To decide whether to modify a low-cost housing project’s rental policies • To improve a recruitment program for blood donors • To determine the impact of a prison’s early release program in recidivism • To gauge community reaction to proposed fire-burning restrictions to improve air quality • To determine the cost-benefit contribution of a new sports stadium for a metropolitan area
  27. 27. Examples of evaluation use in Business and industry • To improve a commercial product • To judge the effectiveness of a corporate training program on teamwork • To determine the effect of a new flextime policy on productivity, recruitment, and retention • To identify the contributions of specific programs to corporate profits • To determine the public’s perception of a corporation’s environmental image • To recommend ways to improve retention among younger employees • To study the quality of performance-appraisal dfeedback
  28. 28. Formative and Summative Evaluation • Formative – provide information for program improvement. Judgment of a part of a program. • Summative – concerned with providing information to serve decisions or assist in making judgments about program adoption, continuation or expansion.