Introduction to project evaluations for SLOGA / Trialog

Workshop
Introduction to project evaluations
Inka Píbilová
inka@evaluace.com
10 February 2014

1

Climbing Kilimanjaro (5,895 m) in 1 year

Source: Suzanne Field

2

EC Project cycle management

Evaluation

Implementation
and monitoring

Financing

Programming

Identification

Formulation

http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

3

Review logical framework


4

Use both quantitative and qualitative indicators
Compare using trends (increase), thresholds (min. 30%), targets (strategy by 12/Y1)
Quantitative - SMARTER
Specific / Simple (to understand, collect)
Measurable
Attainable/Available at cceptable costs
Relevant to project / stakeholders
Time-bound
Evaluate/Engaging
Reevaluate/Recordable

Qualitative - SPICED
Subjective
Participatory
Interpreted and communicable
Cross-checked and compared
Empowering
Diverse / disaggregated (by gender)

Min. 30 % of participants initiate a project
aiming to address a local issue.

Reasons why participants have (not)
implemented a project to address a local
issue

http://www.smarttoolkit.net/?q=node/391

5

Monitoring x evaluation x audit
Evaluation
• Assessment of project efficiency, effectiveness, impact, relevance and
sustainability for the purpose of learning and accountability to stakeholders
Monitoring
• Ongoing analysis of project progress towards achieving planned results with
the purpose of improving management decision making
Audit
• Assessment of (i) the legality and regularity of project expenditure and
income i.e. compliance with laws and regulations and with applicable
contractual rules and criteria; (ii) whether project funds have been used
efficiently and economically i.e. in accordance with sound financial
management;; and (iii) whether project funds have been used effectively i.e.
for purposes intended.
• Primarily a financial and financial management focus, with the focus of
effectiveness being on project results.

6

Evaluation principles for a HORRIBLE evaluation

Source: Participants of the workshop

7

Evaluation principles for GREAT evaluations


8

Evaluation standards and principles
• EC http://ec.europa.eu/europeaid/
– Impartiality and independence of the evaluation process from the
programming and implementation functions;
– Credibility of the evaluation, through use of appropriately skilled and
independent experts and the transparency of the evaluation process,
including wide dissemination of results;
– Participation of stakeholders in the evaluation process, to ensure
different perspectives and views are taken into account; and
– Usefulness of the evaluation findings and recommendations, through
timely presentation of relevant, clear and concise information to
decision makers.
• OECD DAC key norms, standards, criteria
http://www.oecd.org/development/evaluation/dcdndep/41612905.pdf
• UNDP Evaluation policy http://web.undp.org/evaluation/policy.htm
• IPDET Handbook on Evaluation Ethics, Politics, Standards, Principles
http://dmeforpeace.org/sites/default/files/M14_NA.pdf
• Patton, Michael Quinn.(2008) Utilization-Focused Evaluation: 4th edition

9


10

Evaluation Terms of Reference - structure
•
•
•
•
•
•
•
•
•

Background (project, context, stakeholders)
Purpose
Use of outputs
Scope and focus
Evaluation criteria and questions
Methodology
Timetable
Budget
Human resources – responsibilities, expertise required

Guidelines:
• UNDP Handbook (p. 194 - 200) ttp://web.undp.org/evaluation/handbook/
• EC/EuropeAid Project Cycle Management Guidelines (p. 126 - 136)
http://ec.europa.eu/europeaid/multimedia/publications/documents/tools/
europeaid_adm_pcm_guidelines_2004_en.pdf
• New Zealand Guide
11
http://dmeforpeace.org/sites/default/files/NZAID_ToR%20Guideline.pdf

Problem Tree
Example: River Pollution


12

Stakeholder Analysis Matrix
Example: River Pollution


13

Influence
/Power

Stakeholder Analysis – Power / Interest / Attitude

KEEP SATISFIED

INFLUENCE & ENGAGE

Attitude

Policy makers

CSOs
MONITOR

Activists

KEEP INFORMED

Interest
14

Evaluation Terms of Reference - example
Background (project, context)
- project name, identification
- history of the project, objectives, results, key activities, progress over
time (add logical framework if you wish – you get more tailored proposals)
- organisational, social and political context in which the evaluation occurs
- main stakeholders involved in the project incl. target groups, beneficiaries,
partners, donors
- focus and scope of the evaluation – which project components,
geographical area, time period, target groups etc.
Our project:

15

The rationale and purpose of the evaluation
- Why the evaluation is being undertaken (what do you want to get out of it)
including for accountability, learning, improvement
- Why now
Use of outputs
- How it will benefit the different stakeholders
- To whom, when and how the findings will be reported (debriefing,
presentation, report, videos, posters – printed or on-line…)
Our project:

16

Methodology - sources of evaluation questions
• OECD/DAC evaluation criteria (similar to the EC/EuropeAid)
– using project logical framework
• Questions, concerns and values of stakeholders
• Previous research / evaluations
• Guidelines / Evaluation Tools such as Kirkpatrick Model
• Experts

17

Source: Road to Results

How to measure learning outcomes?

http://leanlearning.wikispaces.com/learning_analytics

18

OECD/DAC Evaluation criteria


19

OECD/DAC Evaluation Criteria
Relevance – are we doing the right things?
The appropriateness of project objectives to the problems that it was
supposed to address, and to the physical and policy environment within
which it operated. The extent to which the project is suited to the priorities
and policies of the target group, recipient and donor.
It should include and including an assessment of the quality of project
preparation and design – i.e. the logic and completeness of the project
planning process, and the internal logic and coherence of the project design.
Potential evaluation questions:
• To what extent are the objectives of the programme still valid?
• Are the activities and outputs of the programme consistent with the
overall goal and the attainment of its objectives?
• Are the activities and outputs of the programme consistent with the
intended impacts and effects?
20

Source: OCEED/DAC, http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

Effectiveness – are we doing things right?
An assessment of the contribution made by results to achievement of the
Project Purpose, and how Assumptions have affected project achievements.
This should include specific assessment of the benefits accruing to target
groups, including women and men and identified vulnerable groups such as
children, the elderly and disabled.
• To what extent were the objectives achieved / are likely to be achieved?
• What were the major factors influencing the achievement or nonachievement of the objectives?

21


Efficiency – is the project worthwhile?
The fact that the project results have been achieved at reasonable cost, i.e.
how wellinputs/means have been converted into activities, in terms of
quality, quantity and time, and the quality of the results achieved.
The project shou use the least costly resources possible in order to achieve
the desired results.
This generally requires comparing alternative approaches to achieving the
same outputs, to see whether the most efficient process has been adopted.
• Were activities cost-efficient?
• Were objectives achieved on time?
• Was the programme or project implemented in the most efficient way
compared to alternatives?
22


Impact – what changes has the project achieved / contributed to?
The effect of the project on its wider environment, and its contribution to the
wider policy or sector objectives (as summarised in the project’s Overall
Objective).
The positive and negative changes produced by a project, directly or
indirectly, intended or unintended, positive and negative.
This involves the main impacts and effects resulting from the project on the
local social, economic, environmental and other development indicators. It
must also include the positive and negative impact of external factors.
• What has happened as a result of the programme or project?
• What real difference has the activity made to the beneficiaries?
• How many people have been affected?


23

Sustainability – wil the changes (nebefits for target group) last?
An assessment of the likelihood of benefits produced by the project to
continue to flow after external funding has ended, and with particular
reference to factors of ownership by beneficiaries, policy support, economic
and financial factors, socio-cultural aspects, gender equality, appropriate
technology, environmental aspects, and institutional and management
capacity.
Sustainability is concerned with measuring whether the benefits of a project
are likely to continue after donor funding has been withdrawn. Projects need
to be environmentally as well as financially (and socially) sustainable.
• To what extent did the benefits of a programme or project continue after
donor funding ceased?
• What were the major factors which influenced the achievement or nonachievement of sustainability of the programme or project?
24


Evaluation Questions - Descriptive
•
•
•
•

What „is“
Describe project (inputs, activities, outputs) or process
Simple: Who, what, where, when, how, how many …
Often used to gather opinions from target groups.

Examples:
• What is the project objective from the perspectives of different
stakeholders?
• What were the reasons for joining the program?
• How many persons were reached?
• How was the project implemented?
Evaluation design to answer them:
One-shot, before-and-after, time series, (long-term) panel, case studies…

25

Evaluation Questions - Normative
• Compare what „is “ with what „should be“ (target)
Examples:
• Do the project activities address the needs of stakeholders?
• To what extent has the project achieved the result/objective indicators?
• Have min. 5.000 persons been reached through the campaign?
• Has the number of schools involved in Global Learning increased?
(baseline!)
One-shot, before-and-after, time series, (long-term) panel, case studies…


26

Evaluation Questions – Cause-and-Effect
•
•
•
•

Determine what difference the project makes - what change has it brought
Often refer to outcome, impact
Compare indicators before and after, with and without the project (graph)
Careful about attribution x contribution!
– Can you say that the project achieved this or has contributed to this?
– Are there any alternative explanations (external factors) for achievements?

Examples:
• As a result of the training, have teachers incorporated Global Learning in
their lesson plans?
Experimental (control group), quasi
experimental (compare group),
nonexperimental (causual tracing, case
study, story harvesting, outcome
mapping…)

Goal

with
without

27


T=0

T=1

Time

Evaluation Terms of Reference – Evaluation Matrix
• Evaluation criteria and questions

Design, Data
Our project – Question Baseline data Indicators Sources Collection Methods

28

Tips for evaluation questions
• Avoid questions on multiple issues – separate these
– Has the methodology been developed and disseminated to min. 5.000
teachers?
• You can develop subquestions for a particular question on an issue
– What concerns have teachers while introducing Global Learning at school ?
(descriptive)
– Has the project addressed these concerns of the teachers? (normative)
– Has the number of teachers using Global Learning increased as a result of
the project? (cause-and-effect)
• Set a realistic number of questions!


29

Select a few, most important questions…how?
Would the evaluation question…

Q1 Q2 Q3..

•
•
•
•
•
•
•
•

be in line with the evaluation purpose?
be of interest to stakeholders?
reduce present uncertainty?
yield important information?
be of continuing (not fl eeting) interest?
be critical to the evaluation´s scope and comprehensiveness? (or nice to have)
have an impact on the course of events?
be answerable given the fi nancial and human resources, time, methods, and
technology available?
• Be reasonable to ask given the project cycle?
(Questions about impact, for example, are best answered after the project has
been fully operational for a few years)
Source: Road to Results, Kusters et al.: Making evaluations Matter

30

Data Collection Methods - Quantitative x Qualitative x Mixed
Sampling random x non-random (quota, typical case, snow-ball…), sample size!
Participatory – community meetings, mapping, transect walks..

Conversation
with
Concerned
Individuals

Expert (panel)
Key Informant judgement
Interviews Delhi technique

Participant
Observations:
Community
Diaries,
Interviews
Reviews journals, selfreported
of
checklists
Official
FieldRecords
Visits

Panel
Surveys

Focus
Group
Interviews

One-Time
Surveys

Direct
Observations

Informal/Less Structured Methods

Source: Road to Results / IPDET © 2009, adopted

Censuses

Field
Experiments
Surveys

Formal/More Structured Methods
31

Evaluator
Selection

Interviews
Surveys
Focus groups
Case studies

Final
debriefing
of all partners
Draft evaluation
report
commented by
all partners

Final evaluation report

Desk study

Preliminary findings & conclusions

Initial briefing and inception

Terms of Reference – Objectives,
scope, stakeholders, questions,
budget, schedule, outputs, use.

Timeline and resources

Communication with the Project Partners

Inception
phase

Field
research

Reporting
phase

1-3 months

1-3 months

1-2 months

32

Internal evaluators

or

₊ May have a better understanding of
the projetct, context, policies
₊ Develop organisational capacities
₊ Higher ownership of
recommendations by the organisation
₊ Usually cheaper
₋ May not be able to see alternative
perspectives, solutions
₋ Influenced more by the implementing
organisation (want to keep their jobs)
₋ May be less credible to stakeholders
₋ May be time-consuming

external evaluators

₊ May bring a new perspective or
special (technical, evaluation)
expertise
₊ More independent from the
implementer – may facilitate better
between stakeholders (across
hierarchies, in case of mistrust)
₊ Usually perceived as more credible
₋ May not be able to comprehend fully
the project due to time/other
contrains
₋ Usually more expensive

or mixed - participatory
! External evaluation is not necessarily independent! (Who pays the evaluator?
Who checks the quality?)
! Even external evaluation consumes time of the project team!
33
! It depends WHO the evaluator is 

• Timetable
• Budget
• Human resources – responsibilities, expertise required
Our project:

34

Are we ready?
Evaluability
- Are clear why we do the evaluation?
- Do we have an (updated) logical framework?
- Do we have sufficient (baseline, monitoring) data available?
- Do we have accessible reliable information sources?
- Do we have sufficient funds for an internal/external evaluation? Will the
evaluation be cost-effective, will it bring reasonable benefits vs. costs?
- Is it likely that it will be used to improve actions in future? Can stakeholders
influence the evaluation decisions? Will they accept and use the findings? Is
there a strong leadership to put the recommendations in practice?
- Are there no major factors hindering the evaluation? Are staff members or
other stakeholders overloaded due to other priorities? Are there any
tendencies that would affect impartiality?

Source: UNDP Handbook, Road to Results, Making evaluations matter

35

Qualitative Data Analysis – frequency and relations
Evaluation question: Were parents satisfied with the training?
Topics

Quotes

Findings

Parents decide on topics

I think the process of
deciding would be valuable.

There was a strong feeling
that parents should be more
involved in the choice of
topics

Sometimes we just got into
a topic and then it was time
to leave or move to
something else. We need
more time to discuss.

Many participants (38 or 52
interviewed) thought there
should be more time for
discussion.

//// //// //// //// //// ////
//// ////
Cover a couple of topics per
session
//// //// //// ///

Not enough time spent on
each topic
//// //// //// ///

Develop categories, check reliability, analyse, interpret, share draft, report
Source: IPDET © 2009

36

Quantitative Data Analysis
• Clean data
• Use mode (most frequent), median (middle value), mean (average), range
(lowest to highest value), statistical methods (correlations…)
• Produce tables, graphs etc

Source: IPDET © 2009, docs. Google.com

37

Data Analysis and Interpreting

• Needs to be clear before data collection
• Methodology incl. reliability and limits is a part of report

• Categorize and triangulate findings
• Use quotes, examples, graphs
• Distinguiish between findings (evidence) and interpreting (conclusions)

• Do not generalize findings from 3 respondents to the whole sector!
• Have a short summary for those who cannot read the whole report

Source: Road to Results, evaluace.com

38

Reporting: as per the expected use by each stakeholder

Case
study
39

Reporting – example of quantitative data

the Project A
Project A
Project A
Project A
Project A

Source: Inka Pibilova

40

Reporting – example of quantitative data
No. of women reached
Participationosvětových akcích
Účast na in awareness raising

1033

Checked-up inv mobilních…
Vyšetřeny mobile clinics

3244

Odeslané do onkocenter
Sent to oncocentres

476

Checked-up in oncocentre
Vyšetřeny v onkocentrech

303

Rakovina
Diagnosed with cancer

31

Diagnosed Prekanceróza
with pre-cancer

48

Treated with cancer
Léčba rakoviny

Léčba with pre-cancer
Treated prekancerózy
Vyléčeno
Cured

20
2

?

Source: Naviga4: Prevention and early detection of women with cancer, Georgia, MFA CR

41

Reporting – example of qualitative data

Project A

Finding: Most volunteers expressed doubts about
the Programme, which partially impacted external
communication and advocacy. They believed the
donor should better clarify the program objectives
– if the programme is to primarily serve the
communities in the South or the young
professionals from Europe.

“I believe the development sector
needs well trained and well managed
professionals, not volunteers sent
with a weak/unclear mandate to 'do
something'. This may end up doing
more harm than good." Volunteer
42

Reporting –– example of qualitative data
Reporting example of qualitative data

Too expensive
treatment...
Project A

...incorrect
Radiation
inaccessible
Chemistry teacher and mother, 26 years old
Cervical precancerosis
Incorrect treatment and relapse

43

Where to learn more?
• Examples of evaluation reports (see bottom of evaluation section) and
other tools at http://www.evaluace.com/ - or contact inka@evaluace.com
• Road to Results
https://openknowledge.worldbank.org/handle/10986/2699
• EPDET development evaluation training 31 August – 6 September 2014 in
Slovakia – check http://www.dww.cz/index.php?page=epdet
• www.Betterevaluation.org
• EC evaluation guidelines
http://ec.europa.eu/europeaid/evaluation/methodology/tools/too_en.ht
m
• UNDP Handbook (p. 194 - 200) ttp://web.undp.org/evaluation/handbook/
• OECD DAC key norms, standards, criteria
http://www.oecd.org/development/evaluation/dcdndep/41612905.pdf
• RISK, UK: How do we know it is working?
http://www.developmenteducationreview.com/issue11-review1
• The Most Significant Change Guide:
http://www.mande.co.uk/docs/MSCGuide.pdf
44

Introduction to project evaluations for SLOGA / Trialog

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Viewers also liked

Viewers also liked (13)

Similar to Introduction to project evaluations for SLOGA / Trialog

Similar to Introduction to project evaluations for SLOGA / Trialog (20)

More from Inka Pibilova

More from Inka Pibilova (19)

Recently uploaded

Recently uploaded (20)

Introduction to project evaluations for SLOGA / Trialog