Program evaluation is needed to provide policymakers information on program effectiveness and areas for improvement. The roles of a professional evaluator include collecting relevant information, applying standards to make judgments on a program's merit, worth and significance, and communicating evaluation results to stakeholders. Examples of evaluation uses span various sectors such as education, nonprofit, business and industry to improve programs and make decisions about program adoption and continuation.

1. The Nature of Program
Evaluation
Carlo Magno, PhD
Counseling and Educational
Psychology Department

2. Answer the following questions:
• Why is program evaluation needed?
• What are the roles of a professional
program evaluator?

3. Program evaluation is needed
because…
• Policy makers need good information about the relative
effectiveness of the program.
– Which programs are working well?
– Which poorly?
– What are the program’s relative cost and benefits?
– Which parts of the program are working?
– What can be done with those parts that are not working well?
– Have al parts of the program been thought through carefully at
the planning stage?
– What is the theory or logic model for the program effectiveness?
– What adaptations would make the program more effective?

4. Program Evaluation
• Systematic investigation of the merit, worth or
significance of an object (Scriven, 1999), hence
assigning “value” to a program’s efforts means
addressing those three inter-related domains:
– Merit (or quality)
– Worth (or value, i.e., cost-effectiveness)
– Significance (or importance)
• The identification, clarification, and application of
defensible criteria to determine an object’s value
in relation to those criteria (Fitzpatrick,
WEorthen, & Sanders, 2004).

5. Prerequisite to evaluation
• Need a program: - an organized action
– Direct service interventions
– Community mobilization efforts
– Research initiatives
– Surveillance systems
– Policy development activities
– Outbreak investigations
– Laboratory diagnostics
– Communication campaigns
– Infrastructure building projects
– Training and education services
– Administrative systems

6. Inquiry and Judgment in Evaluation
• (1) Determining standards for judging
quality and deciding whether those
standards should be relative or absolute.
• (2) Collecting relevant information
• (3) Applying the standards to determine
value, quality, utility, effectiveness, or
significance.

7. Evidence of value and judgement:
• What will be evaluated? (i.e., what is "the program" and
in what context does it exist?)
• What aspects of the program will be considered when
judging program performance?
• What standards (i.e., type or level of performance) must
be reached for the program to be considered
successful?
• What evidence will be used to indicate how the program
has performed?
• What conclusions regarding program performance are
justified by comparing the available evidence to the
selected standards?
• How will the lessons learned from the inquiry be used to
improve public health effectiveness?

8. Difference between Research and
Evaluation
• Purpose
• Approaches
• Who sets the agenda?
• Generalizability of results
• Criteria and standards
• Preparation

9. Difference in Purpose
• Research
– Add knowledge in a field, contribute to theory
– Seeks conclusion
• Evaluation
– Help those who hold a stake in whatever is
being evaluated
– Leads to judgments

10. Difference in Approaches
• Research
– Quest for laws
– Explore and establish causal relationships
• Evaluation
– Describing a phenomenon may use causal
relationships
– Causal relationships will depend on the needs
of the stakeholders

11. Difference on who sets the agenda
• Research
– The hypothesis investigated is chosen by the
researcher and the appropriate steps in
developing the theory.
• Evaluation
– Questions to be answered comes form many
sources (stakeholders).
– Consults with stakeholders to determine the
focus of the study.

12. Difference in generalizability of
results
• Research
– Methods are designed to maximize
generalizability to many different settings
• Evaluation
– Specific to the context which evaluation object
rests.

13. Difference in Criteria and standards
• Research
– Internal validity (causality),
– external validity (generalizability)
• Evaluation
– Accuracy (corresponding to reality)
– Utility (results serve practical information)
– Feasibility (realistic, prudent, diplomatic, frugal)
– Propriety (done legally and ethiocally)

14. Difference in Preparation
• Research
– In depth training on a single discipline in their field of
inquiry.
• Evaluation
– Responds to the needs of clients and stakeholders
with many information needs and operating in many
different settings.
– Interdisciplinary: Sensitive to a wide range of
phenomenon that they must attend to.
– Familiar with a wide variety of methods
– Establish personal working relationships with clients
(interpersonal and communication skills)

15. Competencies needed by professional
Evaluators (Sanders, 1999)
• Ability to describe the object and context of an evaluation
• Conceptualize appropriate purposes and framework for
evaluation
• Identify and select appropriate evaluation questions,
information needs, and sources of information
• Select mans for collecting and analyzing information
• Determine the value of the object of an evaluation
• Communicate plans and results effectively to audiences
• Manage the evaluation
• Maintain ethical standards
• Adjust to external factors influencing the evaluation
• Evaluate the evaluation

16. Purposes of Evaluation
• Talmage (1982)
– Render judgment in the worth of the program
– Assist decision makers responsible for deciding policy
– Serve a political function
• Rallis and Rossman (2000)
– Learning, helping practitioners and others better
understand and interpret their observations
•

17. Purposes of Evaluation
• Weiss (1988) and Henry (2000)
– Bring about social betterment
• Mark, Henry, and Julnes (1999)
– Betterment – alleviation of social problems, meeting
of human needs
• Chelimsky (1997) – takes a global perspective:
new technologies, demographic imbalance,
environmental protection, sustainable
development, terrorism, human rights

18. Purposes of Evaluation
• House and Howe (1999)
– Foster deliberate democracy-work to help less
powerful stakeholders gain a voice and to stimulate
dialogue among stakeholders in a democratic fashion.
• Mark, Henry, and Julnes (1999)
– Assessment of merit and worth
– Oversight and compliance
– Program and organizational improvement
– Knowledge development

19. Roles of the Professional Evaluator
• Rallis and Rossman (2000)
– Critical friend: “someone the emperor knows
and can listen to. She is more friend than
judge, although she is not afraid to offer
judgment” (p. 83)
• Schwant (2001)
– Helping practitioners develop critical judgment

20. Roles of the Professional Evaluator
• Patton (1996)
– Facilitator
– Collaborator
– Teacher management consultant
– OD specialist
– Social-change agent
• Preskilll and Torres (1999)
– Bring about organizational learning and
instilling a learning environment

21. Roles of the Professional Evaluator
• Mertens (1999), Chelimsky (1998), and
Greene (1997)
– Including the stakeholders as part of the
evaluation process
• House and Howe (1999)
– Stimulating dialogue among various groups

22. Roles of the Professional Evaluator
• Bickman (2001) and Chen (1990)
– Take part in program planning
– Help articulate program theories or logic
model
• Wholey (1996)
– Help policy makers and managers select the
performance dimension to be measured as
well as the tools to use in measuring those
dimensions

23. Roles of the Professional Evaluator
• Lipsey (2000)
– Provides expertise to track things down,
systematically observe and measure them,
and compare, analyze, and interpret with a
good faith attempt at aobjectivity.

24. Roles of the Professional Evaluator
• Fitzpatrick, Worthen, and Sanders (2004)
– Negotiating with stakeholders group to define the purpose of
evaluation
– Developing contracts
– Hiring and overseeing staff
– Managing budgets
– Identifying disenfranchised or underrepresented groups
– Working with advisory panels
– Collecting and analyzing and interpreting qualitative and
quantitative information
– Communicating frequently with various stakeholders to seek
input into the evaluation and to report results
– Writing reports
– Considering effective ways to disseminate information
– Meeting with the press and other representatives to report on
progress and results
– Recruiting others to evaluate the evaluation

25. Examples of evaluation use in
Education
• To empower teachers to have more say about how
school budget are allocated
• To judge the quality of the school curricula in specific
content areas
• To accredit schools that meet minimum accreditation
standards
• To determine the value of a middle school’s block
scheduling
• To satisfy an external funding agency’s demands for
reports on effectiveness of school programs it supports
• To assist parents and students in selecting schools in a
district with school choice
• To help teachers improve their reading program to
encourage more voluntary reading

26. Examples of evaluation use in other
public and Nonprofit sectors
• To decide whether to implement an urban development
program
• To establish the value of a job-training program
• To decide whether to modify a low-cost housing project’s
rental policies
• To improve a recruitment program for blood donors
• To determine the impact of a prison’s early release
program in recidivism
• To gauge community reaction to proposed fire-burning
restrictions to improve air quality
• To determine the cost-benefit contribution of a new
sports stadium for a metropolitan area

27. Examples of evaluation use in
Business and industry
• To improve a commercial product
• To judge the effectiveness of a corporate training
program on teamwork
• To determine the effect of a new flextime policy on
productivity, recruitment, and retention
• To identify the contributions of specific programs to
corporate profits
• To determine the public’s perception of a corporation’s
environmental image
• To recommend ways to improve retention among
younger employees
• To study the quality of performance-appraisal dfeedback

28. Formative and Summative
Evaluation
• Formative – provide information for
program improvement. Judgment of a part
of a program.
• Summative – concerned with providing
information to serve decisions or assist in
making judgments about program
adoption, continuation or expansion.