Meta Evaluation and Evaluation Dissemination(Manual 3)

META EVALUATION
AND EVALUATION
DISSEMINATION
Dawit Wolde ( MSc, Lecturer).ICME-JU
College of Health Sciences of Jimma University
E-mail:dave86520@gmail.com or
dawit818@yahoo.com
Cell phone:(+251)-922489558/967657712
P.O.Box:378,Jimma University Jimma Ethiopia

Presentation objectives
At the end of the presentations participants will able to:
o Define Meta Evaluation and recognize historical basis for Meta
Evaluation
o Differentiate b/n Ethical rules, Codes, Standards, Principles and
Theories
o Be familiar with standards for Evaluation and guiding principles
for Evaluators
o Identify types and role of Meta Evaluators
o Discuss on Purpose of Evaluation report and important factors in
planning Evaluation report
o Recognize techniques of Evaluation information dissemination
o Understand key components of Evaluation written report
o Explain Evaluation information use
4/17/2016 Meta Evaluation and Evaluation dissemination 2

Presentation outline
• Historical origin and basic concepts of Meta
Evaluation
• Standards of Evaluation
• Guiding principles of Evaluators
• Types and roles of Meta Evaluators
• Meaning and Rationale for Evaluation
dissemination
• Consideration in report preparation
• Channels and formats of dissemination
• Structure of Evaluation report
• Evaluation use and factors affecting it

Training methods
• Interactive lectures,
• Group discussion(exercises),
• Plenary presentations
 Allocated time:20 hours

Basic steps in Program M&E
1.Engage
stakeholders
2.Describe the
program
3.Focus the
evaluation
design
4.Gather
credible
evidence
5.Justify
conclusions
6.Ensure use
and share
lesson learned
Source:CDC’s framework for program evaluation in public health,1999.
Evaluation
standards:
o Utility
o Feasibility
o Propriety
o Accuracy
Evaluation
standards require
gathering of
credible/correct
information(Accur
acy) and use of
information(Utility)
****MM Evaluation
And to ensure perception
of credibility by
evaluation users ,use of
multiple source of data is
one strategy
This will
enhances
….

Basic concepts of Meta Evaluation
• Meta Evaluation is Evaluation of an evaluation to determine the
quality and/or value of an evaluation(Scriven,1991).
• It is a systematic and formal evaluation of evaluations, evaluation
systems or use of Specific evaluation tools in order to guide
planning / management of evaluations within Organizations
(Scriven, 2009).
• It is the process of delineating, obtaining, and applying
descriptive information and judgmental information about the
utility, feasibility, propriety, and accuracy of an evaluation and its
systematic nature, competent conduct, integrity/honesty,
respectfulness, and social responsibility to guide the evaluation
and/or report its strengths and weaknesses‛(Stufflebeam, 2000)

Basic concepts of Meta Evaluation…
Purpose:
• To keep quality of the evaluation and help
evaluation live up to its potential so that:
• Formative Meta Evaluation(Proactive) can
improve an evaluation study before it is
irretrievably too late.
• And Summative Meta Evaluation(Retroactive)
can add credibility to final results.

Evolution of Meta Evaluation:
• In an informal sense, Meta evaluation has been around as long as
evaluation.
• However formally Meta evaluation introduced in 1960’s where:
• Evaluators began to discuss formal Meta evaluation procedures
and criteria
• Writers began to suggest what constituted good and bad
evaluations and unpublished checklist of evaluation standards
began to be exchanged informally among evaluators.
• Several evaluators published their proposed guidelines, or Meta
evaluation‛ criteria, for use in judging evaluation plans or
reports.
• Attempt made among authors to make evaluation criteria’s
useful to evaluation consumers with the objective of avoiding
number of unhelpful and wasteful evaluations.

• However until 1970, there has been a debate among evaluators
and consumers on set of evaluation criteria leading to
disagreement among them in choosing one best criteria among the
list.
• Late 1970- effort made to develop a comprehensive set of
standards explicitly tailored for use in Educational evaluations and
containing general agreed on standards for quality evaluation.
• Developments of these standards began in 1975, under the
direction of Daniel stufflebeam and under authorization of
authorization a Joint Committee on Standards for Educational
Evaluation.
• The result of the Joint Committee work was named as:
Standards for Evaluations of Educational programs, Projects
and Materials.
• The standards include 30 standards under four major standards:
Utility, Feasibility, Propriety and Accuracy.

Benefits of the standards:
1. A common language to facilitate communication and
collaboration in evaluation
2. A set of general rules for dealing with a variety of
specific evaluation problems
3. A conceptual framework by which to study the often-
confusing world of evaluation
4. A set of working definition to guide research and
development on the evaluation process
5. A public statement of the state of the art in evaluation
6. A basis for self-regulation and accountability by
professional evaluators and
7. An aid to developing public credibility for the
evaluation field.

EXERCISE 1
Be in your group and :
• Differentiate between ethical rules, codes, standards,
principles and theory?
• Discuss on :Why Evaluation standards are required in
addition to Fundamental Ethical principles??(30 minutes)

1. Ethical rules: are specific statements about ethical
behavior(in our situation related to professional setting or
professional practices).These rules prescribe behavior in a
relatively detailed fashion. For example: Rules about
obtaining informed consent from evaluation or research
participants are frequently specific, although they may
vary in the amount of detail.
2. Ethical codes: are compilations of ethical rules. For
example: The ethical code for educational researchers has
six sections: responsibilities to the field, research
populations, intellectual ownership, editing and revewing
research,and students and student researchers(American
Educational Research Association,1992).

3.Standard-synonyms with a rule but also can suggest model
behavior. The evaluation standards focus on overall quality of
evaluations, not on specific rules. Use of the word ethics is
assiduously avoided in these standards, but embedded within
them are guideline that might be considered ethical in nature.
A typical example for this is Propriety evaluation standard.
4.Ethical principles: are broader than rules or codes and often
serve as the foundation on which rules and codes are built.
Principles such as “don’t harm” and Golden rule provide
guidance for many ethical decisions and are helpful when the
rules or codes conflict or don’t provide specific guidelines for
our ethical concerns. The Fundamental ethical principles
includes:Beneficence,Nonmaleficence,Authonomy,Justice
and Fidelity.

5.Ethical theory: refer to efforts to explain how people go
about making ethical decisions. Ethical theory is the science
of ethical decision making. Ethical theories are general ways
of determing what behavior is considered right or wrong.
Classified in to five based on the criteria used to decide
whether the behavior is right or wrong as:
• Consequences
• Duty
• Rights
• Social justice and
• Ethics of care

Rules Statements of specific do’s and most often don’ts
Example: Provide mean and median scores on all standardized tests used in
an evaluation, Take both oral and written consent from research participants.
Codes Compilations of rules, usually adopted and endorsed by a professional
organizations.
Example: Ethical codes of American Educational Research Association and
American Psychological Association
Standards Similar to rules but often suggest ideal behavior.
Example: Make sure that all stakeholders understand the technical terms used
in an evaluation report.
Principles Broader than rules or codes; provide guidance when rules conflict or when
rules are not specific to the context.
Example: Evaluate programs as you would want your program to be
evaluated.
Theories Justification or criteria for ethical decisions; the science and rationale for
making ethical descions.
Example: The consequences of an action are the determinants of what
constitutes ethical or unethical behavior.

Relationship b/n ethical theories and principles
Theory/Criteria Related principles
Consequences(utilitarianism):What
are the consequences of my choice?
What would happen, for example ,if
every evaluator made the same
decision?
Autonomy,Beneficence,Nonmaleficence
Duty(deontological):What duties and
obligations do I have as an evaluator?
Autonomy,Beneficence,Fidelity,Nonmalefi
cence
Rights: What right do my clients
have? What rights do I have?
Autonomy,Justice,Nonmaleficence
Social Justice: What would be just or
fair in this situation?
Justice,Beneficence,Nonmaleficence
Caring(Ethics of care):What would
be the caring response or course of
action?
Beneficence,Nonmaleficence

STANDARDS OF EVALUATION
Utility
Feasibility
Propriety
Accuracy

Standards for program Evaluation
• The joint committee standards comprised a set of 30
evaluation standards under four core standards.
• These core standards were :
( 1) Utility,
(2) Feasibility,
(3) Propriety, and
(4) Accuracy.

Standards for program Evaluation…
Utility standards:
• The utility standard is intended to ensure that an
evaluation will serve the intended information
needs of its intended users.
• Intended users can be specific individuals,
organizations or any entity that receives and uses
evaluation findings.
• Under these standards there are seven (7) sub-
standards.

U1:Stakeholders Identification
 Persons involved in or affected by the evaluation should be
identified, so that their needs can be addressed.
U2:Evaluator credibility
• The person conducting the evaluation should be both
trustworthy and competent to perform the evaluation, so
that the evaluation findings achieve maximum credibility
and acceptance.
U3:Information scope and selection
• Information collected should be broadly selected to address
pertinent questions about the program and be responsive to
the needs and interests of clients and other specified
stakeholders.

U4:Values identification:
• The perspectives, procedures, and rationale used to
interpret the findings should be carefully described,
so that the bases for value judgment are clear.
U5:Report clarity
• Evaluation reports should clearly describe the
program being evaluated, including its context, and
the purpose, procedures, and findings of the
evaluation, so that essential information is provided
and easily understood.

U6:Report timeliness and Dissemination
• Significant interim findings and evaluation reports
should be distributed to intended users, so that they
can be used in a timely fashion.
U7:Evaluation Impact
• Evaluation should be planned, conducted and
reported in ways that encourage follow-through by
stakeholders, so that the likelihood that the
evaluation will be used is increased.

Feasibility Standards:
• The feasibility standards are intended to ensure that
an evaluation will be realistic, prudent, diplomatic,
and frugal.
• It comprises of three (3) standards.
F1:Practical Procedures:
• The evaluation procedures should be practical; to
keep disruption to a minimum while needed
information is obtained.

F2:Political Viability
• The evaluation should be planned and conducted with
anticipation of the different positions of various interest
groups, so that their cooperation may be obtained and so
that possible attempts by any of these groups to curtail
evaluation operations or to bias or misapply the results can
be averted or counteracted.
F3:Cost Effectiveness:
• The evaluation should be efficient and produce information
of sufficient value, so that the resources expended can be
justified.

Propriety Standards:
• The propriety standards are intended to ensure that an
evaluation will be conducted legally, ethically, and with due
regard for the welfare of those involved in the evaluation as
well as those affected by its results.
• This standard comprises of 8 sub -standards.
P1:Service Orientation:
• Evaluation should be designed to assist organizations to
address and effectively serve the needs of the full range of
targeted participants.

P2:Formal Agreement:
• Obligations of the formal parties to an evaluation (what is to
be done, how, by whom, when) should be agreed to in
writing, so that these parties are obligated to adhere to all
conditions of the agreement or formally to renegotiate it.
P3:Rights of Human Subjects:
• Evaluations should be designed and conducted to respect
and protect the rights and welfare of human subjects.
P4:Human Interactions:
• Evaluators should respect human dignity and worth in their
interaction with other persons associated with an
evaluation, so that participants are not threatened or
harmed.

P5:Complete and Fair Assessment
• The evaluation should be complete and fair in its
examination and recording of strengths and weaknesses of
the program being evaluated, so that the strengths can be
built up on and problem areas addressed.
P6:Disclosure of Findings:
• The formal parties to an evaluation should ensure that the
full set of evaluation findings along with pertinent
limitations are made accessible to the persons affected by the
evaluation and to any others with expressed legal rights to
receive the results.

P7:Conflict of Interest:
• Conflict of interest should be dealt openly and
honestly, so that it does not compromise the
evaluation process and results.
P8:Fiscal Responsibility:
• The evaluator’s allocation and expenditures of
resources should reflect sound accountability
procedures and otherwise be prudent and ethically
responsible, so that expenditures are accounted for
and appropriate.

Accuracy Standards:
• The accuracy standards are intended to ensure that an
evaluation will reveal and convey technically adequate
information about the features that determine worth or merit
of the program being evaluated.
• It comprise of a total of 12 sub -standards.
A1:Program Documentation:
• The program being evaluated should be described and
documented clearly and accurately, so that the program is
clearly identified.

A2:Context Analysis:
• The context in which the program exists should be examined
in enough detail so that its likely influences on the program
can be identified.
A3:Described Purposes and Procedures:
• The purpose and procedures of the evaluation should be
monitored and described in enough detail so that they can be
identified and assessed.
A4:Defensible Information Sources:
• The sources of information used in a program evaluation
should be described in enough detail so that the adequacy of
the information can be assessed.

A5:Valid Information
• The information-gathering procedures should be chosen or
developed and then implemented so that they will ensure
that the interpretation arrived at is valid for the intended use.
A6:Reliable Information:
• The information-gathering procedures should be chosen or
developed and then implemented so that they will ensure
that the information obtained is sufficiently reliable for the
intended use.

A7:Systematic Information:
• The Information collected, Processed and reported in an
evaluation should be systematically reviewed and any errors
found should be corrected.
A8:Analysis of Quantitative Information
• Quantitative information in an evaluation should be
appropriately and systematically analyzed, so that evaluation
questions are effectively answered.
A9:Analysis of Qualitative Information
• Qualitative information in an evaluation should be
appropriately and systematically analyzed, so that evaluation
questions are effectively answered.

A10:Justified Conclusions:
• The conclusions reached in an evaluation should be
explicitly justified, so that the stakeholders can
assess them.
A11:Impartial Reporting:
• Reporting procedures should guard against
distortion caused by personal feelings and biases of
any party to the evaluation, so that the evaluation
reports fairly reflect the evaluation findings.

A12:Metaevaluation:
• The evaluation itself should be formatively and
summatively evaluated against this and other
pertinent standards, so that its conduct is
appropriately guided and, on completion,
stakeholders can closely examine its strengths and
weaknesses.

Principles
Evaluator Autonomy Nonmalificence Beneficence Justice Fidelity
Political
viability(F2)
Practical procedures(F1) Stakeholder
identification(U1)
Report
dissemination(U6)
Evaluators
Credibility(U2)
Disclosure of
findings(P6)
Reliable measurement(A6) Information scope
and selection(U3)
Formal
obligations(P2)
Valid Measurement(A5) Report Clarity(U5) Report
timeliness(U6)
Conflict of interest(P7) Systematic
information(A7)
Practical
Procedures(F1)
Disclosure of findings(P6) Analysis of QUAN
information(A8)
Rights of Humans(P3) Complete and fair
assessment(P5)
Analysis of QUAL
information(A9)
Defensible
information
sources(A4)
Impartial
reporting(A11)
Human interactions(P4)
Justifiable conclusions(A10)
Political Viability(F2)
Stakeholders Information scope
and selection(U3)
Values
identification(U4)
Evaluators
Credibility(U2)
Report timeliness and
dissemination(U6)
Rights of Human
subjects(P3)
Formal
Obligations(P2)
Balanced
reporting(P6)

Evaluators role and Ethical Dilemma's
1. Evaluators as consultant or administrator(Vignette 1.1 and
1.2)
2. Evaluator as data collector/researcher(Vignette 1.3 and
1.4)
3. Evaluator as reporter(Vignette 1.5 and 1.6)
4. Evaluator as member of a profession(Vignette 1.7 and 1.8)
5. Evaluator as member of a society(Vignette 1.9 and 1.10)

EXERCISE 2:APPLICATION OF
EVALUATION STANDARDS
Be in your previous group and conduct Meta Evaluation on the
Evaluation reports provided to you(30 minutes).
Group I:PMTCT program
Group II: Nutrition Program/ICCM Program
Group III: Malaria program

Guiding Principles for Evaluators
• The AEA principles are lifestyle expectations for
professional evaluators rather than a set of
standards to be applied to any one specific study.
• It promotes a lifestyle of systematic inquiry,
professional development, honesty, respect, and
concern for society.
• These guiding principles permeate the day-to-day
activities of the evaluator over an entire career.

Guiding Principles for Evaluators…
• The AEA principles include:
1. Systematic inquiry-Evaluators conduct systematic, data-based
inquiries about whatever is being evaluated.
2. Competence-Evaluators provide competent performance to
stakeholders.
3. Integrity/Honesty-Evaluators ensure the honesty and integrity
of the entire evaluation process.
4. Respect for people-Evaluators respect the security, dignity, and
self -worth of the respondents, program participants, clients,
and other stakeholders with whom they interact.
5. Responsibilities for General and Public Welfare-Evaluators
articulate and take into account the diversity of interest and
values that may be related to the general and public welfare.

Relationship b/n AEA guiding principles for Evaluators
and Kitchener’s principles
AEA guiding principles Kitchener’s principles
Systematic inquiry: Evaluators conduct systematic,
data-based inquiries about whatever is being
evaluated.
Beneficence,Nonmaleficence
Competence: Evaluators provide competent
performance to stakeholders.
Non-maleficence
Integrity/honesty: Evaluators ensures the honesty
and integrity of the entire evaluation process.
Fidelity
Respect for people: Evaluators respect the
security,dignity,and self-worth of the respondents,
program participants, clients and other stakeholders
with whom they interact.
Autonomy, Beneficence
Responsibilities for general and public
welfare:Evaluators articulate and take into account
the diversity of interests and values that may be
related to the general and public welfare.
Justice

EXERCISE 3:WHO SHOULD
CONDUCT META
EVALUATION?
Discuss on and present outcome of your discussion to
general class(30 minutes).

Types of Meta Evaluators
1. Meta Evaluation conducted by the Original evaluator:
sometimes evaluations were evaluated by the evaluator himself
/herself (original evaluators). The problem behind this is the
biases that can accrue from evaluating one’s own works and
recommended to have another evaluator review their own
works.
2. Meta Evaluation conducted by Evaluation consumer-Often the
evaluation sponsor, client, or other stakeholders are left to judge
the adequacy of an evaluation plan or report without assistance
from a professional evaluator. But the success of this approach
depends heavily on the technical competence of the consumer.
Recommended to consult expertise.
3. Meta Evaluation conducted by Competent Evaluators: is the
best arrangement in that it has both the advantage of technical
competence and minimal biases.

DISSEMINATION AND USE
OF EVALUATION FINDINGS
Section II

Basic concepts and terminologies
• Designing, preparing ,and presenting an evaluation reports is
the very essence of evaluation,
• But if that report does not prove useful or influential, then
that essence quickly evaporates into an empty exercise.
• Many program collect valuable and informative data on their
program and services.
• But they may not know how to share the data with the public
and with influential people at the local and state levels to
ensure that the findings will be used and that programmatic
approaches and interventions can be replicated.

Basic concepts and terminologies
What is dissemination?
• Dissemination is making information available and usable to
various audiences through a wide variety of channels or
formats.
• Here:
• A channel refers to a route of communication such as a
news conference or posters.
• And a format refers to the actual layout for
communicating the information.
• Information can be communicated through both oral
and written formats.

EXERCISE 4:REASONS TO
DISSEMINATE
Be in your previous group and discuss on the rationale for
disseminating monitoring and evaluation information.
Then present outcome of your discussion to larger
group(20 minutes).

Purpose of Evaluation reports
• The purpose of evaluation report is directly linked to the use
intended for the evaluation.
• For example:
• The overall purpose of formative evaluation is to improve
the program ,and the report should inform program staff
early about how the program is functioning and what
changes must be made to improve it.
• And in summative evaluation the report should provide
information and judgment about the mature programs
value to those who:(1)may wish to adopt it, (2)will
determine resource allocation for its continuation, or
(3)have a right to know about the program for other
reasons.

Purpose of Evaluation reports…
• In general evaluation reports can serve many different purposes
including:
• Demonstrating accountability
• Assisting in making a decision
• Bringing an issue to attention of others
• Helping stakeholders elaborate or refine their opinion of an issue
• Convincing others to take action
• Exploring and investigating issues
• Involving stakeholders in program planning or policy development
• Gaining support for a program
• Promoting understanding of issue
• Changing attitudes
• Changing individual behaviors
• Changing the nature of dialogue or interaction among groups
• Influencing policy
• Introducing those involved to new ways of thinking through
evaluation.

Important factors in planning evaluation
reports
• Identifying the intended audiences for the evaluation report.
Think that:
• Different stakeholders have different information needs?
• Different stakeholders prefer different channel of
communication?
• Understanding what information is demanded by
stakeholders (audience) and for what purposes.
• This help to tailor the report content to the evaluation
audience preferences.
• Tailoring the report format, style, and language to the
preferences of evaluation audiences.
• Timing of the evaluation information dissemination

reports…
• In corresponding Information to Particular Audiences
answer the following questions:
1. Who is your audience? (e.g., directors and staff at the
ministry or regional level, donors, customers etc…)
2. What does your audience need to know or what are their
specific interests? (e.g., improvement in outcomes, such as
increased knowledge, more positive attitudes, or healthy
behavior change)
3. What do you hope to gain by disseminating this
information or these results? (e.g., to justify the existence
of the program or to leverage additional funding)

reports…
4.How will you communicate about the ongoing
program? (e.g., briefings at board meetings, progress
reports with a summary, and verbal presentations)
5.How will you communicate about the program
upon its completion? (e.g., final written report with a
summary, verbal debriefing, videos, and oral
presentation)

EXERCISE 5:CHANNELS
AND TIMING OF
REPORTING
Be in your previous group discuss on:
• The different channels of reporting(Strength and weaknesses)
and
• Timing of reporting of evaluation results(30 minutes).

Channels of reporting
• Written reports
• Photo essays
• Audiotape reports
• Slide-tape presentations
• Film or video tape reports
• Multimedia presentations
• Dialogues/testimonies
• Hearings or mock trials
• Product displays
• Simulations
• Scenarios
• Portrayals
• Case studies
• Graphs and charts
• Test score summaries
• Question/answers
• E-mail reports
N.B:In choosing among channels the recommendation is to discuss with
stakeholders and identify appropriate channel for them.

Timing of Evaluation reporting
• Timing of reporting is crucial for effective use of evaluation
findings by intended users.
• Based on timing of reporting, Evaluation reports can be:
1. Scheduled Interim Reports: Reports can be scheduled at
milestones in either the evaluation or the program or at
regular intervals corresponding to routine meetings of
clients or stakeholders.
2. Unscheduled Interim Reports: used when unexpected
event or results pop up.
3. Final reports: reports prepared after the interim reports of
evaluation. Sometimes preliminary final report released
for review and reaction by stakeholders, and then final
report.

Timing of Evaluation reporting…
For example of interim unscheduled reports:
• In a formative evaluation the evaluator may discover a major
problem or impediment, such as the fact that video monitors
used in an experimental program designed to train federal meat
inspectors are too small for trainees beyond the third row to see
the critical indicators of possible contamination. It would be a
gross disservice to withhold that information until the next
scheduled interim report, which might be weeks away, and then
deliver the not-to surprising message that a majority of the new
generation of meat inspectors didn‘t seem to be learning much
from the experimental program that would serve the cause for
public health.

Key component of a written report
1. Title page
2. Executive Summary(2-6 pages)/ Executive Abstract(1-2 pages)
3. Introduction to the report
 Purpose of the evaluation
 Audiences for the evaluation report
 Limitation of the evaluation and explanation of disclaimers(if any)
 Overview of report contents
4.Focus of the evaluation
 Description of the evaluation object
 Evaluative questions or objectives used to focus the study
 Information needed to complete the evaluation
5.Brief overview of evaluation plan and procedures
6.Presentation of evaluation results
 Summary of evaluation findings
 Interpretation of evaluation findings

Key component of a written report…
7.Conclusion and Recommendations
• Criteria and standards used to judge evaluation object
• Judgment about evaluation object(strength and
weakness)
• Recommendations
8.Minority reports or rejoinders(if any)
9.Appendices
• Description of evaluation plan/ design, instruments,
and data analysis and interpretation
• Detailed tabulations or analysis of quantitative data,
and transcripts or summaries of qualitative data
• Other information, as necessary

Evaluation report outline examples
Example 1:
1. Title Page:
• Title and nature of evaluation
• Title of Program, phase, duration
• Identification of author, date of submission, commissioning
service
2. Table of contents:
• Main headings and sub-headings
• Index of tables of figures and graphs
3.Executive Summary:
• An overview of the entire report in no more than five
pages
• A discussion of the strengths and weakness of the chosen
evaluation design

Evaluation report outline examples…
4.Introduction:
• Description of the Program in terms of needs,
objectives, delivery systems etc.
• The context in which the Program operates
• Purpose of the evaluation in terms of scope and main
evaluation questions.
• Description of other similar studies which have been
done
5.Research methodology:
• Design of research
• Implementation of research and collection of data
• Analysis of data

Evaluation report outline examples
6.Evaluation results:
• Findings
• Conclusions
• Recommendations
7.Annexes:
• Terms of reference of the evaluation
• References and sources
• Names of evaluators and their companies (CV should also be
shown, but summarized and limited to one page per person).
• Methodology applied for the study (phases, methods of data
collection, sampling, etc).
• Logical framework matrices (original and
improved/updated).
• List of persons and organizations consulted, literature and
documentation other than technical annexes (e.g. statistical
analyses)

Example 2:
• The basic elements of a final evaluation report might include the
following:
 ƒTitle page
 ƒExecutive summary
 ƒIntended use and users
 ƒProgram description
 ƒEvaluation focus
 ƒData sources and methods
 ƒResults, conclusions, and interpretation
 ƒUse, dissemination, and sharing plan
 ƒTools for clarity

Example 3
I. Executive Summary
II. Program Description
a. Implementation Process
b. Program Goals and Objectives
III. Evaluation Design and Methodology
IV. Results
a. Data
b. Process Report
b. Outcomes Report
V. Interpretation of Process and Outcomes Report
VI. Conclusions
VII. Recommendations
VIII.Appendices

Example 4
II. Problem Statement
III. Evaluation Design
IV. Evaluation Methodology
V. Results
a. Quantitative
b. Qualitative
VI. Conclusions
VIII.Appendices

Example 5
II. Program Description
III. Project Objectives and Activity Plans
IV. Evaluation Plan and Methodology
V. Results
a. Process and Outcome
b. Successes and Barriers
VI. Conclusions
VIII.Appendices

Key component of a written report…
• The evaluation report should follow a logical
structure.
• What is important is that the structure of the report
meets the needs of the donors of the evaluation as
well as the principal stakeholders.

Basics of Good Evaluation Reporting(Tips)
1. Consider the needs of your target audience(s) even before you
begin your evaluation.
2. Give your audience important details of the evaluation.
• What– Your central evaluation question
• Why– Your purpose for the evaluation
• Who– Your source(s) of information, including sample
or census size and response rate
• How– Your data collection methods
• Where– The locations from which you collected data
• When– The time frame you collected the data

Basics of Good Evaluation Reporting(Tips)…
3.Use caution in reporting findings and drawing conclusions.
• Remember the type of data you collected and how you
collected it helps determine what you can eventually
conclude about it.
• Do not claim that your program caused a specific result
unless you have experimental evidence in support of that
claim.
4.Have others read or listen to your report and give you feedback
before you create the final version.

EXERCISE 6:UTILIZATION
OF EVALUATION AND
FACTORS AFFECTING IT
Be in your group and discuss on culture of information
utilization in your organization and factors affecting it.
Then present outcome of your discussion to larger
group(30 minutes).

The evaluation theory three
Adopted from:Marvin C. Alkin and Christina A. Christie 2004
11/17/2015 Introduction to M&E for Environmental Health 69
Process
Use
Findings
Use

Utilization of Evaluation
• The utility of any evaluation is a prime criterion for judging
its worth.
• Use is one of the factor that distinguishes evaluation from
research.
• Evaluation is intended to have an immediate or at least a
near -term impact, while research is intended to add to
knowledge and theory in a field, but the result it yields may
not be used for some time.
• Evaluation use can be Instrumental utilization, Conceptual
utilization, Enlightenment, persuasive utilization and
Process utilization .
• The first four are related to the use of the findings.

Utilization of Evaluation…
1. Direct(Instrumental) utilization: refers to the use of evaluation
findings to make some immediate decisions or judgments.
Evaluators often tailor their evaluations to produce results that
can have a direct influence in the improvement of the structure,
or on the process of a program. E.g. Decisions for funding/no
funding based on results.
2. Conceptual utilization: refers to use of evaluation findings to
enlighten and inform stakeholders about issues. Even if
evaluation results do not have a direct influence in the re-
shaping of a program, they may still be used to make people
aware with regard to the issues of concern. This can lead to
instrumental use at later time.
3. Enlightenment-when evaluation finding add knowledge to the
field and thus may be used by anyone, not just those involved with
the program or evaluation of the program.

4.Process use:
• Refers to ways in which being engaged in the process of
evaluation can be useful quite apart from the findings that may
emerge from the study
• Participating in an evaluation can cause changes to occur: in one’s
thought about organizational management and programs, in
drawing attention to new issues, in creating dialogues between
different stakeholders.
5.Persuasive Utilization:
• It is the enlistment of evaluation results in an effort to persuade an
audience to either support an agenda or to oppose it.
• In this case unless the 'persuader' is the same person that ran the
evaluation, this form of utilization is not of much interest to
evaluators as they often cannot foresee possible future efforts of
persuasion.

Level of influence of evaluation: Process use

Change Processes forEvaluationInfluenceat the IndividualLevel

Factors affecting Evaluation utilization
Results From the 2006 Sample

Factors affecting Evaluation utilization
Factors affecting evaluation utilization(finding utilization):
Relevance of the evaluation to decision makers and/or other
stakeholders
Involvement of users in the planning and reporting stages of
the evaluation
Reputation or credibility of the evaluator
Quality of the communication of findings: timeliness,
frequency, method.
Development of procedures to assist in use or
recommendations for action.

Factors affecting Evaluation utilization…
• Factors that affect process use:
(a) Facilitation of evaluation processes;
(b) Management support;
(c) Advisory group characteristics;
(d) Frequency, methods, and quality of communications;
and
(e) Organization characteristics.

Results From the 2006 Sample…

Recommended readings
1. Jody Fitzpatrick, James R. Sanders, Blaine R.Worthen.Program
Evaluation: Alternative Approaches and Practical guidelines.
Third edition. Pearson Education,Inc.2004.
2. Joint Committee on Standards for program Evaluation. The
program Evaluation standards. Second edition. Thousand Oaks,
CA: Sage.1994.
3. Kristin Olsen and Sheelagh O‟Reilly. Evaluation
Methodologies፡A brief review of Meta-evaluation, Systematic
Review and Synthesis Evaluation methodologies and their
applicability to complex evaluations within the context of
international development. June 2011.
4. Disseminating Program Achievements and Evaluation Findings
to Garner Support. Evaluation Briefs. No. 9 | February 2009.

Recommended readings…
5.Daniel A. McDonald, Pamela B. C. Kutara, Lucinda S.
Richmond, Sherry C. Betts. Culturally respectful evaluation.
December 2004, Vol. 9, No. 3
ISSN 1540 5273:https://ncsu.edu/ffci/ publications/2004/v9-n3-
2004-december/ ar-1-cuturally.php

THANK YOU….
………A Good Journey
Evaluation=Use+ Methods + Valuing

Meta Evaluation and Evaluation Dissemination(Manual 3)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Meta Evaluation and Evaluation Dissemination(Manual 3)

Similar to Meta Evaluation and Evaluation Dissemination(Manual 3) (20)

Meta Evaluation and Evaluation Dissemination(Manual 3)