The document discusses various evaluation approaches, including expertise-oriented, consumer-oriented, and program-oriented evaluations. Expertise-oriented evaluations rely on professional judgment and can be formal or informal, while consumer-oriented evaluations help the public assess the quality of products. Program-oriented evaluations focus on determining the extent to which specified objectives are achieved, utilizing various methodologies such as objectives-oriented, theory-driven, and goal-free evaluations.

1. Expertise, Consumer-Oriented,
and Program-Oriented
Evaluation Approaches
John H. Curry, Ph.D.

2. Expertise-Oriented Approach
•The oldest type of formal evaluation
•Relies on professional expertise to
judge the quality of an institution,
program, product, or activity.

3. Types of Expertise-Oriented
Evaluations
•Formal review system
•Informal review system
•Ad hoc panel review
•Ad hoc individual review

4. Formal Review Systems
•Mostly used in accreditation
•Examine existing structure of organization
•Examine published standards
•Follow a specified schedule
•Increasing emphasis on outcomes
•Use the opinions of multiple experts
•Status of those being evaluated is
affected by the results

5. Informal Review Systems
•Used primarily for evaluations that
lack published standards or follow a
specified review schedule.
•Use multiple reviewers
•Status of those being reviewed is
affected by results
•Examples: Peer review of articles,
theses or dissertation committees

6. Ad Hoc Panel Reviews
•Occur at irregular intervals when
circumstances demand
•Reviews not related to
institutionalized evaluation or
standards
•Usually one-shot evaluations
prompted by a particular, time-bound
need for evaluative information

7. Examples of Ad Hoc Panels
•Panels to develop standards
•Funding agency review panels
•Blue ribbon panels

8. Ad Hoc Individual Reviews
•Review of any entity by an individual
selected for his/her expertise
•Usually to judge value or make
recommendations
•Example: Employment of a
consultant to review an educational,
social, or commercial program

9. Consumer-Oriented Evaluation
•Much like expertise-oriented
•Helps inform decisions on what to
purchase or trade
•Judge the quality of something, establish
value, merit or worth of a product
•Audience is broader: The purchasing
public, and they are not known to the
evaluator.

10. Consumer-Oriented, cont.
•Developed by Scriven
•Much like needs assessments
•Functional analysis of product
•IMPORTANT to:
•IDENTIFY the criteria correctly
•DEVELOP standards to judge those criteria
•COLLECT data
•SYNTHESIZE information to make a final
judgment

11. Sample Consumer-Oriented
Criteria
•Need
•Market
•Performance (Field Trials)
•Performance (Consumer)
•Performance (Critical Comparisons)
•Performance (Long-Term)
•Performance (Side Effects)
•Performance (Process)
•Performance (Causation)
•Performance (Statistical Significance)
•Performance (Educational Significance)
•Cost-effectiveness
•Extended Support

12. Consumer-Oriented Examples
•Consumer Reports
•US News and World Report

13. Program-Oriented Approaches
•Focus on learning some key features
of the program, and then serve to
help the evaluator decide which
questions should be asked.
•Most common type: The objectives-
oriented evaluation

14. Objectives-Oriented
Evaluation
•Distinguishing feature: some activity
is specified, and then the evaluation
tries to determine the extent to which
those objectives are achieved.

15. Tylerian Evaluation
•Steps:
•Establish broad goals or objectives
•Classify the goals or objectives
•Define the objectives in behavioral terms
•Find situations in which achievement of objectives can be shown
•Develop or select measurement techniques
•Collect performance data
•Compare performance data with behaviorally stated objectives
•Generally discredited today as
oversimplifying a complex system

16. Provus Discrepancy Evaluation
•Steps:
•Agree on standards (objectives)
•Determine whether a discrepancy
exists between performance and
standards
•Using that information, decide
whether to improve, maintain, or
terminate the program or some
aspect of it

17. Theory-driven evaluation
•Program theory is “the construction
of a plausible and sensible model for
how a program is supposed to work.”
•Programs fail to achieve goals for
two different reasons:
•The program isn’t delivered as planned and, therefore, isn’t
ready to be tested (implementation failure)
•The program is delivered as planned, and the results
indicate that the program theory was incorrect (theory
failure)

18. Theory-driven evaluation, cont.
•Steps:
•Engage relevant stakeholders
•Develop a first draft of program theory
•Present a draft to stake holders for
discussion, reaction, and input
•Conduct a plausibility check
•Communicate findings to key stakeholders
•Probe arrows for model specificity
•Finalize program impact theory

19. Goal-Free Evaluation
•The evaluator purposely avoids becoming aware
of the program goals
•Predetermined goals are not permitted to
narrow the focus of the evaluation study
•Goal-free evaluation focuses on the actual
outcomes rather than intended program
outcomes
•The goal-free evaluator has minimal contact
with the program manager and staff
•Goal-free evaluation increases the likelihood
that unanticipated side effects will be noted

20. Reference
•Fitzpatrick, J., Sanders, J., &
Worthen, B. (2011). Program
evaluation: Alternative approaches
and practical guidelines (4th ed.).
Upper Saddle River, N.J.: Pearson
Education.