A presentation by Sudhanshu Handa as part of the Innovations in Design and Measurement panel discussion at the International Symposium on Cohort and Longitudinal Studies in Developing Contexts, UNICEF Office of Research - Innocenti, Florence, Italy 13-15 October 2014

1. The ‘story’ of the Impact Evaluation of
Ghana’s Livelihood Empowerment against
Poverty (LEAP) Program
Using an existing longitudinal data
collection effort to evaluate the country’s
largest social protection program

2. Transfer Project: Multi-country research initiative to
understand impact of national cash transfer programs
in SSA
Program Time Period Evaluation Design (level)
Zimbabwe HSCT 2013-2015 District matched case-control
Zambia CGP/MCP 2010-2015 RCT (CWAC)
Malawi SCT 2013-2015 RCT (Village Cluster)
Lesotho CGP 2011-2013 RCT (Electoral Division)
Kenya CT-OVC 2007-2011 RCT (Location)
Ghana LEAP 2010-2012 Longitudinal PSM
“Making the whole greater than the sum of the parts”
www.cpc.unc.edu/projects/transfer

3. Ghana LEAP IE: Background
• May 2009 identification mission to provide options for
‘rigorous’ IE of LEAP
– LEAP reaching 30,000 households in all regions of Ghana
– Next expansion scheduled for late 2009
• Resistance to building a ‘delayed entry’ control group
– Would have to target households well in advance
– Feared political backlash of ‘leaving eligible households out
just to study them’
– Next GLSS not expected for some time
• Accidental trip to Univ of Ghana to see if they had any ideas

4. Ghana LEAP IE: Brokering a deal with Yale and
University of Ghana
• Yale/UG planning 10 year national panel survey in Ghana,
funded by Yale (benefits of a large endowment)
– First round scheduled Spring 2010
– Multi-topic survey, most key evaluation indicators already in qsn
• The ask: Could they include some LEAP households into their
sample?
– Would it jeopardize their survey? How many households? Different
protocols? Who would be in charge?
• Agreement!
– 700 households maximum; fully integrated into Yale/UG survey, same
everything (enumerators, qsn, training, protocols, data entry, etc)
– Literally as if they simply increased their sample size by 700
purposefully chosen households
– Marginal cost paid by Ministry of Social Welfare

5. Ghana LEAP IE: From baseline to follow-up
• Yale/UG follow-up panel delayed
– Originally scheduled for 2 years, now 4 years
– What to do with IE? Could we re-interview ‘matched’
sample from Yale/UG survey along with LEAP households?
– Would we disrupt their study? Respondent fatigue vs
opportunity to test their tracking protocol
• Agreement! Follow-up conducted 24-months later on LEAP
households and ‘matched’ comparison group from Yale/UG
survey
– Longitudinal DD PSM estimator that fulfilled key criteria for PSM to
mimic benchmark
– [High pressure modelling of participation: determined which
households would actually be re-interviewed]

6. How did it work? Could we find comparison
units in national survey?
0 2 4 6 8
Figure A2.1 Distribution of propensity score by sample
0 .2 .4 .6 .8 1
Propensity Score
Full ISSER Sample ISSER Matched Sample
LEAP

7. Needed to reweight comparison group using
inverse of propensity score
2
1
2.5
0
1.5
.5
Distribution of propensity scores (unweighted)
0 .2 .4 .6 .8 1
Pr(T)
LEAP COMPARISON GROUP
2
1
1.5
.5
0
Distribution of propensity scores (weighted)
0 .2 .4 .6 .8 1
Pr(T)
LEAP COMPARISON GROUP

8. Baseline characteristics of LEAP and ISSER samples
ISSER SURVEY
LEAP All
Rural
Matched Matched
weighted
(1) (2) (3) (4)
Household size 3.83 4.12 3.69 3.83
Children under 5 0.44 0.73 0.45 0.46
Children 6-12 0.77 0.84 0.76 0.83
Children 13-17 0.54 0.47 0.50 0.52
Elderly (>64) 0.76 0.31 0.65 0.83
Number of orphans 0.62 0.15 0.34 0.65
Orphan living in hhld 0.27 0.09 0.19 0.28
Head characteristics
Female Household 0.59 0.28 0.54 0.64
Age of Head 61 49 59 63
Widowed 0.39 0.13 0.30 0.41
Head has schooling 0.30 0.57 0.47 0.31
Household characteristics
No toilet 0.31 0.37 0.31 0.34
Pit latrine 0.30 0.46 0.42 0.31
P.C. spending (GHc) 55.46 67.05 60.06 47.47
Livestock owned 0.41 0.57 0.44 0.42
N 699 3136 699 699

9. One of the most widely cited IE results in Ghana
Impact of LEAP on Happiness (DD PSM Estimates)
Full
Sample
FHH Size≤4 Size≥5
Impact 0.158 0.233 0.206 0.088
(t-statistic) (2.20) (2.28) (2.30) (0.73)
Observations 3,036 1,634 1,937 1,099
LEAP Baseline Mean 0.395 0.357 0.382 0.418
ISSER Baseline Mean 0.597 0.589 0.587 0.614

10. Reflections on the LEAP IE experience
• Able to ‘piggy-back’ off existing panel survey effort to build
rigorous IE
– Not everything under control of IE team
– Cheaper, more inclusive, results led to actual program
changes
– Do we always need a $2m stand-alone IE?
• PIs of panel data sets in this room: what would you have
done?
– These opportunities almost always exist in developing
countries where ‘a lot is going on’
– Prior to 2009 nobody knew about this 10-year panel in
Ghana, now they do