This presentation from the International Initiative for Impact Evaluation (3ie) was part of ICRAF's Agroforestry Development Impact Seminar (ADIS) series.

1. Not forgetting the Forests
for the trees - The art and
science of useful impact
evaluations
Presentation at ICRAF
Nairobi, 3rd September, 2015
Dr. Jo (Jyotsna) Puri: Head of Evaluation and Deputy Executive Director,
Diana Lopez: Evaluation Specialist
Stuti Tripathi: Senior Policy and Evidence Uptake Officer

2. Who, Why, What and Why – 3ie
• 2008
• Largest grant giving
organization for impact
evaluations
• All areas
• Raison d’etre

3. Deforestation – do protected areas
help? Assessing trade-offs

6. Variable 1: Protected Areas

7. Variable 2: Road building

8. Predicting the location of
Deforestation

9. Predicting Location of Impacts of
the two variables
• Where is deforestation most likely to occur?
Which areas are vulnerable?
• Are Protected Areas Effective?
• Hypothesis: Plots devoted to the most
lucrative use. So plot is cleared if clearing it is
more profitable than letting it stay forested.
9

10. The Econometric model
• The econometric model that we estimate is thus
given by
• Zi : Plot attributes (Slope, Elevation, Impedance
weighted travel time, Soil Dummy, Population
density)
• Y1i*: Net profit from clearing
• Y2i*: Net utility from protecting a plot
otherwise0;0*
2
if1
222
*
2
otherwise0;0*
1
if1
1121
*
1


i
Y
i
Y
i
e
i
WB
i
Z
i
Y
i
Y
i
Y
i
e
i
YB
i
Z
i
Y



12. Data – A spatially explicit GIS with many
layers
• Roads data (1982)
• Land Use (1:1m, LDD)
• District level census data on population (1990)
• Physiographic: Elevation (DEM), Slope
• Soil (FAO)
• Socio-Economic: Population density
• Cost of Travel to the market; Roads (DCW and
LDD)
• Protected Areas (IUCN)
12

13. 13
Roads (1982) and
Forests Of North Thailand (1986)
NORTH
THAILAND
Elevat.shp
Elevat.shp
1 - 1000 feet
1000 - 3500 feet
3500 - 6000 feet
6000 - 7700 feet
Elevat.shp

14. Sampling
• Grid squares of 100ms, 28 Million data
points
• Sampled at 5 km (spatial autocorrelation):
6550 sampled pixels.
• Impedance weighted cost of travel
(Costdistance in Arc/Info)

15. NORTH
THAILAND
Elevat.shp
Elevat.shp
1 - 1 000 fe et
100 0 - 3 500 fe et
350 0 - 6 000 fe et
600 0 - 7 700 fe et
Elevat.shp

16. 16
Cleared Land (Y1 = 1) T- Stats
Slope (degrees) -0.088 -10.652
Elevation (ms.) -0.001 -8.095
Population density1990
(people/km2)
0.003 4.532
Log(cost) (1982)** -0.191 -9.729
Soil and Province Dummies Not Shown
Protected Area dummy (1986) -0.077 -0.332
Constant 1.295 8.870
Protected Area (Y2 = 1) Equation
Slope (Degrees) 0.034 5.297
Elevation (ms.) 0.001 9.058
Population density1990
(people/km2)
0.001 2.297
Log(cost) (1982) 0.192 7.477
Soil and Province Dummies Not Shown
Watershed dummy 0.188 3.543
Constant -4.098 -14.010
Log Likelihood -3714.7
No. of observations 4946

17. Results
• In a static model, protected areas are not very
successful in preventing deforestation; Agnostic
about Wildlife Sanctuaries.
• Roads hasten clearing.
• Model useful in predicting vulnerable areas.

18. Accuracy of Predictions
Actual 
Predicted
Cleared Forested % of
Predictions
Correct
Cleared 872 296 75%
Forested 657 3133 83%
% Correctly
predicted
57% 91%
18

19. 19
Long way still....Khao Sanam
Phriang
Nam Nao
Thung Salaeng
Suang
Predicted Threatened areas
Wiang Kosai
Mae Yom
Chae Son
Areas predicted to clear
after cost is reduced by 100
units

20. Exploring the potential for using
GIS/Big data:
What affects agricultural expansion?

24. What affects Deforestation in the
area?
• Assessing impact of population and road
bulding on the agricultural frontier Forest
Reserves.
• Uses panel data over 11 years (1986-1996)
24

25. Data Problems
• Village level data
• Unbalanced data set (Selection bias in villages?)
• Ambiguous property rights
• Groups of crops tracked. No consistent data on crop
area.
• Short run crop (soybean maize tobacco peanut)
• Long run crop (upland rice, tea, tree crops)
• Paddy rice
• Self-reported data: Confounded zeros and missing
values
• Data on prices of crops

26. The Econometric Model
• Log (Land devoted to crop i by village j in year t )= f(Pjt, Tjt ,
Wjit, Git, Nit, Lit , Ait, t)
• Where
• Pjt : Population of village j at time t
• Tjt : Time taken to travel by most favored mode from village j
to the market at time t
• Ajt: Proportion of adults in village j at time t
• Wjit : Availability of water in village j at time t for crop i
• Gjt : Use of credit in village j at time t
• Njt : Land productivity in village j at time t
• Ljt: Status of property rights in village j at time t
• t : time trend

27. Ag area
Coefficient
Intensity of
cultivation
Year 15.35** 0.0379
Area devoted to Paddy Rice 0.46** 0.01399**
Area devoted to Upland Rice 0.21* 0.0011
Area devoted to Soybean -0.19+ 0.0037*
Constant -641.17* 63.314**
Sigma-u 1054.16 14.0611
Sigma-e 375.89 16.392
0.89 0.4239
Observations 1979 2174
R-square Within 0.042
Groups 622 629
R-square Between 0.054
R-square Overall 0.056
Gaussian Wald Statistic (chi2, 4df) 85.5 20.03
Prob > Chi2 0 0.0005
Results: Agricultural area & Intensity

28. Results
•Overall, ease in access led to a
substitution between upland rice and
paddy rice
•Upland rice area decreased.
•Environmental AND livelihood
benefits!

29. Conclusions
• Magnitudes of policy instruments are clearly
important to judge trade-offs.
• The ‘where’ and the ‘how’ specially important.
• Crop wise exploration is important.
• Would have been excellent to have satellite and
ground truthed data on
• verifiable soil,
• crop and
• agricultural land data
• Boundaries of forest reserves and location.

30. Promoting Agricultural Innovation in Sub-Saharan
Africa and South Asia: an Impact Evaluation
Program
Challenges and Learnings
ICRAF-Nairobi. September 3rd 2015

31. General Features of the Interventions
• Target at the farmer association level
• Target supply and demand: multiple
intervention arm

32. Common challenges that can arise
• Complex: Difficulties in identifying the
outcomes of interest
• Measuring the synergies among the different
treatment arms
• Very likely to have to deal with contamination
• Eligibility and placement of geographical
areas and farmers is not always clear cut
• Women are usually underrepresented in the
farmer associations

33. Learnings (1)
• Doing qualitative work prior to first round of
data collection it is key in order to :
• Better understand of the the change the
intervention can have on the population of interest
• Better define the groups on which the intervention
should have a differential impact
• Better identify the possible sources of
contamination and possible channels of spillover
effects

34. Learnings (2)
• Doing a pilot survey is also key to:
• Better design of the data collection
instruments
• Identifying the constraints farmers may
have to participate in the program

35. Learnings (3)
• A good team work between researchers
and implementing agencies is key for the
project to move and implement
successfully.

36. Evidence uptake and use

37. Generating evidence that is useful
Helps to answer a specific policy
question
Rooted in the local context
Opportunity exists for evidence to be
used
Message gives solutions (that can be
drawn from the evidence)
Affordable, logistically possible,
politically feasible
RELEVANT
CONTEXTUAL
DEMAND DRIVEN
CLEAR
FEASIBLE

38. Evidence uptake in 3ie-funded
studies
0
2
4
6
8
10
12
14
16
18
Expand successful
progs
Close unsuccesful
progs
Change pol/prog
design
Inform discussions
of pol/prog
Inform design of
other progs
Inform global
poldiscussions
Improved culture of
use of evidence

39. Expand successful programmes
Wage subsidy programme in South Africa

40. 3ie’s ToC for evidence uptake
3ie funds high-
quality policy
relevant studies
Researchers
engage with key
stakeholders
Results in uptake
of study findings
We get applications
- that ask policy
relevant question
- Comprises of a
study team that
has
demonstrated
experience in
stakeholder
engagement
ASSUMPTIONS
Researchers have the
- Willingness and
commitment
- Understand how
evidence uptake
happens
- Have the tools and
resources to invest
in stakeholder
engagement
Study makes policy
relevant
recommendations
that are
- Evidence based
- Answer what
works and why
- Propose feasible
solutions given
context and cost
ASSUMPTIONS ASSUMPTIONS
Grant management is about making sure
that the assumptions hold!!

41. 3ie’s ToC for evidence uptake
3ie funds high-
quality policy
relevant studies
ASSUMPTIONS
We get applications
- that ask policy
relevant question
- Hold potential for
policy impact
- Comprise a study
team that has
demonstrated
experience in
stakeholder
engagement
Institution of
preparation phase
to facilitate context
understanding and
stakeholder engagement
Increased
weightage to policy
relevance and impact
How is
this
ensured

42. 3ie’s ToC for evidence uptake
3ie encourages
researchers to
engage with key
stakeholders
ASSUMPTIONS
Researchers have
Willingness and
commitment
- Understand how
policy influence
happens
- Have the tools
and resources to
invest in
stakeholder
engagement
Developing a plan
and setting aside a
budget for
engagement
Encouraging
involvement of in
country
researchers
Theory based
evaluations– what
works and why
How is
this
ensured

43. 3ie’s ToC for evidence uptake
Results in uptake of
study findings
Study makes
policy relevant
recommendations
that are
- Evidence based
- Answer what
works and why
- Propose feasible
solutions given
context and cost
ASSUMPTIONS
Ongoing
engagement
with 3ie more
dynamic model of
interaction
End of project
interviews to
learn and adapt
processes
How is
this
ensured

44. Ensuring quality and achieving
relevance to users
Use multiple prongs: integrated and regular use
of many communication channels: social media, media,
listservs, events, meetings, website
Early
Widely
Often
Establish
champions

45. Thank you
Jyotsna (Jo) Puri
jpuri@3ieimpact.org