This presentation shares and reflects on the practical implications of the design choices made around standards of rigor, inclusiveness and feasibility in the impact evaluation of the IFAD-funded Root & Tuber Improvement and Marketing Program (RTIMP) in Ghana. The approach used in this evaluation was developed with support from IFAD and the BMGF to assess and explain the impact of program/project investments on rural poverty in a collaborative and participatory manner.

1. Balancing rigor,
inclusiveness and feasibility
Learnings from the design of a participatory impact
evaluation of the IFAD-funded RTIMP in Ghana.
Glowen Kyei-Mensah (PDA)
Adinda Van Hemelrijck & Irene Guijt (IFAD & BMGF)
UKES May 2015 London

2. Impact Evaluation under IFAD9
Attribution
of complex
government
programs?
Participation
of people?
Learning
with partners?
80m
people out
of poverty?

3. Improved learning initiative for the piloting of a
Participatory Impact Assessment
& Learning Approach (PIALA)
✔
Almost
✔
$ 90 K $ 260 K
DBRP RTIMP

4. Objectives
Assessing
to what extent
impacts occurred
(or not)
Debating
how
impacts can be
enhanced
Explaining
why
impacts occurred
(or not)
LEARNING
1. Produce rigorous qualitative and quantitative
evidence for global reporting and advocacy
2. Facilitate
inclusive analysis
and reflection for
collaborative learning
3. Generate a
scalable model for
strengthening IFAD’s
self-evaluation system
& Purposes

5. Design challenges
• Serving the different purposes with limited
budgets
• Understanding the typical attributes of IFAD-
funded government programs
• Generating solid conversation about “what
has worked for whom, under which conditions
and why?”

6. PIALA Quality Assurance Framework
1. Focus & frame
the evaluation
2. Describe &
link changes
3. Identify causes
4. Manage quality
5. Synthesise
findings
6. Analyse & debate
contributions
7. Report & share
Rigour Inclusiveness FeasibilityEvaluation Design
http://www.participatorymethods.org/authors/adi
nda-van-hemelrijck-and-glowen-kyei-mensah

7. 1. Focusing & framing the evaluation
• Upfront discussion with sponsors about
design options (scope & scale) and budgets
• Reconstruction of Theory of Change based
on desk review and consultations
• Design workshop with national stakeholders
for identifying
– Impact & contribution claims
– Core assumptions
– Evaluation & learning questions

8. M2b: Training & starter pack for commercial
seed growers to multiply certified R&T seeds
C3a: R&T processors grow and develop into
GPCs that are profitable enterprises
O3: Enhanced R&T
processed volumes of
high quality at scale
O2: Enhanced
R&T productivity
and production
at scale
M2c: Farmer Field Forums (FFF) engage
farmers, extension agents and researchers in
developing, demonstrating and promoting
appropriate R&T production technologies
C2a: Resource-poor R&T farmers & seed
producers gain access to and adopt improved
R&T seed varieties, technologies & inputs to
improve crop husbandry, soil fertility and
pest management practices
C2b: Resource-poor R&T farmers organise
and register as FBOs that can access credit
and bargain better market prices
C1b: Resource-poor R&T processors, farmers
& seed producers commercialize and establish
effective supply chain linkages
C1a: R&T supply chain farmers & processors
are capable of developing and implementing
viable business and marketing plans
C3c: R&T supply chain farmers and
processors gain access to business financing
and market-linking services
M3b: Subsidized upgrading of advanced R&T
processors into Good Practice Centres (GPCs)
that demonstrate and promote good quality
processing & management practices
C3b: R&T supply chain processors gain
access to and adopt standardized processing
technology and good quality
management practices
O1: R&T supply chain
actors effectively solve
their supply & demand
issues and timely obtain
technical support,
resulting in sustainable
and inclusive CCs
linked to old and new
markets
I2: Improved R&T-
based livelihoods for the
rural poor in CC
catchment areas
M2a: R&D for developing bio-agents
M1c: Information, Education &
Communication (IEC) about CC support
services, inputs and technologies
M1a: Training of resource-poor farmers and
processors involved in the R&T supply chains in
business development and marketing
M3c: Co-financing of R&T supply chain farmers
and processors by matching 40% RTIMP funds
with 50% loans from PFIs and 10% self-financing
through the Micro-Enterprise Fund (MEF)
M3a: Training of artisans to produce and maintain
standardized processing equipment
for R&T supply chain processors and GPCs
I1: Rural poor people
in CC catchment areas
have increased access
to food & income to
sustain an active and
healthy life
M1: District Stakeholder
Forums (DSFs) for addressing
supply & demand issues and
technical support needs of R&T
supply chain actors members
M1b: Supply Chain Facilitation (SCF) and
market linking through the Initiative Fund (IF)
EC1
EC3c
EC1
EO2
EO1
EC3b
MEF
GPC
DSF
FFF
Roots & Tubers Improved Marketing Program (30m)

9. Hypothesis:
[Enhanced smallholder R&T production and processing at scale] + [Sustainable and inclusive CC market linkages]
=> Improvement livelihoods and poverty reduction in rural Ghana
Assumption:
· Livelihoods and poverty status in rural Ghana can be improved by commericializing smallholder R&T production and processing businesses combined with the
establishment of competitive market-driven and inclusive CC linkages.
Evaluation/learning question:
· To what extent and for whom does this assumption hold true (or not) and under which conditions? Does it hold true for resource-poor women and youth/young
adults in remote rural areas?What conditions need to be changed to enable women and young adults overcome cultural and socio-economic barriers?
Causal link House-hold
level
Community cluster
level
District level Zonal & national level Sampling approach
I2→I1 HH survey
(with households in the
the supply chain
catchment area)
Review of the 2010 Ghana
Living Standard Survey
report and other relevant
secondary data
Stratified sampling of 30
households from the
community clusters in each
sampled district
O3+O2+O1
→I2
Generic change analysis
(in gender/age-specific focus
groups of community members from
the supply chain catchment area)
Review of RTIMP RIMS
baseline ad other M&E data
Stratefied sampling of
community members from
the community clusters in
each sampled district
O2
O1
I2
I1
O3
Impact claim – Poverty reduction

10. Contribution Claim 3 – Enhanced R&T processing
Hypothesis:
[Access to business financing] + [Demonstration of good practices]
=> Development of profitable processing enterprices by R&T supply chain farmers and processors
=> Enhancement and scaling of smallholder R&T processing, contributing to the growth of the R&T supply chains
Assumptions:
· Resource- poor processors and farmers who are well trained in quality management, business planning and marketing apply
for matching grant funding (MEF) to invest in their businesses. PFIs are prepared to provide credit to well- trained resource- poor processors and farmers up to 50%
of their planned investments.
· GPCs can reach and teach resource- poor processors to develop more profitable agri- processing businesses by demonstrating good quality processing and
management practices, including the use of improved technologies and stnadardized equipments. As a result, resource- poor processors apply to the MEF and invest
in new technologies and equipment that help them to produce greater volumes of higher quality at lower cost.
Evaluation/learning question:
· To what extent and for whom do these assumptions hold true (or not)?
· What conditions need to be in place for GPCs to become profitable and attractive businesses particularly for young adults living in remote areas?What supports or
hinders GPCs to better link the supply chain farmers to old and new markets, and how is this influenced by the DSF?
· Reach and added value of the MEF? Effects of the MEF on growth of the funded agro- processing businesses? Avoidance of elite capture?
Causal link Household
level
Community cluster
level
Disrict level Zonal & national level Sampling
approach
M3b→
C3a+C3b→O3
Livelihood analysis
(in gender/age-specific focus
groups with supply chain farmers
and processors)
KIIs with GPCs Review of RTIMP and REP
M&E data and supervision
reports (incl. the 2014
thematic impact studies on
MEF and GPC); the
comparative case study on
matching grant facilities
Stratified sampling of
supply chain farmers,
seed growers and
processors
Stratified sampling of
GPC- and non-GPC-
participants (incl. MEF
beneficiaries)
M3b+M3c+C1a
→C3c
Constituent feedback
(with mixed groups of (non-)
GPC participants and MEF
beneficiaries )
KIIs with BACs and PFI local
branches
C3c
M3b C3b
M3c
C3aM3a
O3

11. Selection of appropriate methods specific to the
links and questions:
• HH survey – correlation between changes in “access
to food & income” and R&T livelihood changes and
investments
• PRA-based methods – causes of R&T livelihood
changes and investments
• SenseMaker lithe – patterns in experiences of “R&T
livelihood changes” (400)
• Constituent feedback – effects and reach of selected
program mechanisms (DSF, FFF, MEF & GPC)
2. Describing & linking changes

12. 3 zones
8 regions
4 commodity chains
25 random districts
30 community clusters
150 Parti FGDs
(with 1200 ppts, 45% women)
860 HH
Surveys
Parti Sensemaking
(in 23 districts with 640 ppts;
national with 100 ppts)
100 KIIs
with officials, bankers,
researchers, enterprises…

13. 6. Analysing & debating contributions
• Configurational analysis and integrated QUAL-QUANT
synthesis
– Systematic collation of data from the different methods at
district and aggregated levels (with 0-6 rating of strength of
links and evidence for each cluster)
– Analysis of patterns in the evidence resulting from different
with/without configurations across districts for each cluster
• Sensemaking involving stakeholders in a collective
analysis and debate of evidence of impact and
contributions

14. PIALA QAF:
2. Describe &
link changes
3. Identify
the causes
4. Manage
quality
Rigour Inclusiveness Feasibility
Multi-stage sampling
enabling comparative
analysis of with/without
configurations of
program treatment
Nested mixed-methods
• consistent and equal
QUANT & QUAL data
collection relevant to the
links & questions
• complementarity of types
of information
• Triangulation of different
sources & types of data
Instant data entry and
linking enabling on-site
integrated analysis and
sensemaking
Data quality monitoring
and process reflection
every evening while
doing data collation
Field research capacity
(nr of teams,
time in the field,
logistics & mobilization,
supervision & quality
assurance, time for
instant data processing)• Classic HH survey
• PRA-based methods
• Testing of new tools
(SenseMaker & CF) to
overcome respondent &
researcher bias
• District sensemaking
workshops
• 8 regions, 3 zones
• 30 clusters in 25 districts
(propotional to CC size)
• 30 HHs/ cluster (tot 900)
• 40 participants/cluster
(tot 1200, 45% women)
• Process vs data (R + I)
• Participation vs independence (R + I)
• Scope vs depth (F + I)
• Scale vs voice (R + I)
Trade-offs under budget constraints?

15. Final notes
Value for
money?
@ baseline
mid & end?
Replicability?
Harvesting
the best of
all
Involving policy-
makers, donors &
constituents
Answering
the questions
Capturing variability in
program treatment
Capacity
trumps all