Presented by Pascale Schnitzer and Carlo Azzarri, IFPRI at the Africa RISING–CSISA Joint Monitoring and Evaluation Meeting, Addis Ababa, Ethiopia, 11-13 November 2013

1. Evaluation in Africa RISING
Pascale Schnitzer and Carlo Azzarri, IFPRI
Africa RISING–CSISA Joint Monitoring and Evaluation Meeting,
Addis Ababa, Ethiopia, 11-13 November 2013

2. Outline
• Quantitative (experimental and
quasi-experimental)
• Qualitative
• Mix methods

3. Quantitative
Experimental
• RCT
• Choice experiments, auctions, games
Quasi-experimental
• Double-Difference (Diff-in-Diff)
• Matching
• RD
• IV and encouragement design

4. Example: providing fertilizers to farmers
 Intervention: provide fertilizer to farmers in district A
 Program targets all farmers living in district A
 Farmers have to enroll at the local extension office to
receive the fertilizer
 District B does not receive any intervention
 Starts in 2012, ends in 2016, we have data on yields for
farmers in district A and district B for both years
 What is the effect of giving fertilizer on agricultural
yields?

5. Case I: Before & After
Y
(1) Observe only
beneficiaries
(2) Two observations
in time:
yields at T=0
and yields at T=1.
2200
A
α = 200
2000
B
T=2012
IMPACT=A-B= 200?
T=2016
Time

6. Case I: What’s the problem?
Unusual good weather/rain:
o Real Impact=A-C
2200
o A-B is an overestimate
A
Impact?
α = 200
Drought
o Real Impact=A-D
o A-B is an
underestimate
C?
2000
Impact?
B
D?
T=2012
T=2016
Time

7. Case II: Those Enrolled & Not Enrolled
If we have post-treatment data on
o
o
Enrolled: treatment group
Not-enrolled: “comparison” group (counterfactual)
(a) Those that choose NOT to participate
(b) Those ineligible to participate (e.g. neighbor
community)
Yields in 2016 by participant type
3000
2800
2500
2500
2200
2000
1500
1000
500
0
Choose to participate
Choose not to participate
Inelegible to participate
Did AR have a
negative impact?

8. Case I and II
In the end, with these naïve comparisons, we cannot tell
if the program had an impact
We need a comparison group that is as identical in
observable and unobservable dimensions as possible, to
those receiving the program, and a comparison group
that will not receive spillover benefits.

9. We need to keep in mind…
B&A
E&NE
Compare: Same individuals Before
and After they receive P.
Problem: Other things may have
happened over time.
Compare: Group of individuals
Enrolled in a program with group
that chooses not to enroll.
Problem: Selection Bias. We don’t
know why they are not enrolled.
Both counterfactuals may lead to
biased estimates of the impact.

10. Quantitative
Experimental
• RCT

11. RCT
1. Population
3. Randomize
treatment
2. Evaluation sample
X
Comparison
Treatment
= Ineligible
= Eligible
External Validity
Internal Validity

12. Quantitative
Experimental
• RCT
• Choice experiments, auctions, games

13. Choice experiments, auctions, games
• An experiment is a set of observations generated in a controlled
environment to answer a particular question or solve a particular
problem.
• Subjects make decisions that are not part of their day-to-day
decision making (typically in a game environment), they know they
are part of an experiment, or both.
• Purposes:
1. Test theories
2. Measure what are considered “unobservables” (e.g.
preferences, beliefs)
3. Test sensitivity of experimental results to different forms of
heterogeneity

14. Choice experiments, auctions, games
• Examples:
-behavioral game theory
-ultimatum games
-dictator games
-trust games
-public good games
-coordination games
-market experiments (auctions)
-risk- and time-preference experiments

15. Quantitative
Quasi-experimental designs
• Double-Difference (Diff-in-Diff)

16. Probability of adoption
Impact =(A-B)-(CD)=(A-C)-(B-D)
Not
participants
C=0.81
D=0.78
A=0.74
Participants
Impact=0.11
B=0.60
T=0
Before
T=1
After
Time

17. Impact =(A-B)-(CD)=(A-C)-(B-D)
Probability of adoption
Not enrolled
C=0.81
D=0.78
A=0.74
Impact<0.11
Enrolled
B=0.60
T=0
Before
T=1
After
Time

18. Example from Malawi:
Total land used (acres)
Treatment Group
(Randomized to
treatment)
Counterfactual
(Randomized to
Comparison)
Impact
(Y | P=1) - (Y | P=0)
Baseline (T=0)
[MARBES] (Y)
3.04
2.13
0.91
Follow-up (T=1)
[MARBES] (Y)
??
??
??

19. Quantitative
Non-experimental
• Double-Difference (Diff-in-Diff)
• Matching

20. Propensity-Score Matching (PSM)
Comparison Group: non-participants with same
observable characteristics as participants.


In practice, it is very hard.
There may be many important characteristics!
Match on the basis of the “propensity score”,


Compute everyone’s probability of participating, based on
their observable characteristics.
Choose matches that have the same probability of
participation as the treatments.

21. Density
Non-Participants
Participants
Common Support
0
Propensity Score
1

22. Quantitative
Non-experimental
• Double-Difference (Diff-in-Diff)
• Matching
• RD

23. RD: Effect of fertilizer program on adoption
Goal
Improve fertilizers adoption for small farmers
Method
o Farms with a score (Ha) of land ≤2 are small
o Farms with a score (Ha) of land >2 are not small
Intervention
Small farmers receive subsidies to purchase fertilizer

24. Regression Discontinuity: Design at baseline
Not eligible
Eligible

25. Regression Discontinuity: post intervention
IMPACT

26. Quantitative
Non-experimental
• Double-Difference (Diff-in-Diff)
• Matching
• RD
• IV and encouragement design

27. Babati (WP2): Timeline and design of an evaluation
Feb
13
July
13
AugOct 13
Nov 13 –Mar. 14
Mar. 2016
200 receive
improved seeds
Initial planting at
demonstration
plots
Follow-up field
day: farmers rank
preferred seeds
Fertilizer and seed
distribution
Survey
800 farmers in 11
villages
200 receive
improved seeds and
fertilizer
200 receive
seeds, fertilizer and
contracts
200 receive no
additional
intervention
End-line survey: measure
impacts

28. Outline
• Qualitative

29. Qualitative
• Semi-structured or open-ended indepth interviews
• Focus groups
• Outcome Mapping
• Participatory Impact Pathways Analysis
(PIPA)

30. Outcome Mapping (OM)
• Contribution of AR to changes in the
actions, behaviors, relationships, activities of the ‘boundary
partners’ (individuals, groups, and organizations with whom AR
interacts directly and with whom it anticipates opportunities for
influence)
• It is based largely on systematized self-assessment
• OM is based on three stages:
1. Intentional design (Why? Who? What? How?)
2. Outcome and performance monitoring
3. Evaluation design

31. Outcome Mapping (OM)
By using OM, AR would not claim the achievement
of development impacts; rather, the focus is on its
contributions to outcomes. These outcomes, in
turn, enhance the possibility of development
the relationship is not necessarily a
impacts – but the relationship is not necessarily a
direct one of cause and effect
effect.

32. Qualitative
• Participatory Impact Pathways Analysis
(PIPA)

33. Participatory Impact Pathways Analysis (PIPA)
• PIPA begins with a participatory workshop where stakeholders make
explicit their assumptions about how their project will achieve an
impact. Participants construct problem trees, a visioning exercise and
network maps to help them clarify their 'impact pathways‘ (IPs).
• IPs are then articulated in two logic models:
1. The outcomes logic model -> the project's medium term
objectives in the form of hypotheses: which actors need to
change, what are the changes, which strategies are needed to attain the
changes.
2. The impact logic model -> how, by helping to achieve the
expected outcomes, the project will impact on people's livelihoods.
Participants derive outcome targets and milestones, regularly revisited
and revised as part of M&E.

34. Outline
• Mix methods

35. Mixed methods
• Combination of quantitative and
qualitative research methods to
evaluate programs

36. Conclusions
• We cannot do everything in every megasite…
• Quantitative surveys are being conducted/planned in
every country
• IFPRI has comparative advantage in quantitative
approaches, we shall split the tasks with the research
teams on qualitative methods -> mixed methods
• Is IFPRI M&E on the right track? What shall we be
focusing on more? What shall we not be doing ?

37. Africa Research in Sustainable Intensification for the Next Generation
africa-rising.net