The use of administrative data for program evaluation Rio

1. The use of administrative data for
program evaluation
Emanuela Galasso
Development Research Group
The World Bank
The role of Administrative records and complex surveys in the
monitoring and evaluation of public policies
Rio de Janeiro Nov 2014

2. Using administrative data: the question
• Administrative data: main objective is monitoring for decision
making– can trace and describe trends of key inputs and
outcomes of interest
• Question: how can it be used for impact evaluations?
• Complementary to survey data collection
• For either type of data sources, a pre-requisite for embarking in
an impact evaluation is to have a clear identification strategy
• Call for expanding access to administrative data prominent US
economists

3. Using administrative data for impact evaluations: the
potential
• Size: Universe of the target population
• Best placed to assess programs targeted to very
specific groups – to be oversampled in surveys (the
extreme poor, migrants), or geographic areas
• By-product of the existing monitoring of the social
policy: cost
• time: characterize dynamics of impact
• Flexibility: can potentially look at different target
groups, assess new variants

4. Using administrative data for impact evaluations: the
cons
• Data up-to-date? Differential capacity of municipalities
or agencies in data collection
• Representativeness? ex.registries of the poor (as
CadUnico) covers the entire target population?
• Data quality? Income (imputations), scope for
over/under reporting (ex. Colombia PMT)
• Scope: limited set of variables. May be missing key
outcomes or determinants of pathway of impact(ex.
noncognitive skills, mental health, parenting practices)

5. Using administrative data for impact evaluation: the experience of Chile Solidario
• Non-experimental evaluation: method of regression
discontinuity design
• Planned after the program had started
• Exploited the clear assignment :
• Eligibility based on proxy means score (CAS). Official cutoff by
municipality
• Households invited from the bottom up of the CAS distribution in
each municipality
• Program gradually rolled out, 4 cohorts 2002-2005, became law in
2004

6. Chile Solidario: objectives and approach
1. Social inclusion
• Targeting the extreme poor (and vulnerable):
bottom 5% CAS. From explicit demand for services
to identification and invitation
• Central role of psychosocial wellbeing in social
policy: 2 years of psychosocial support, local social
worker

7. Chile Solidario: objectives and approach
2. Social protection
• Short term cash transfer + guaranteed monetary
transfers
• Preferential access for social services (demand)
• Making the supply side available (supply) and
tailored to the needs of the poorest
• Beyond access to health and education: multiple
dimensions and ‘minimum conditions’

8. Chile Solidario: objectives and approach
3. Promotion
•.Identification of key skills and endowments to sustain
exit from poverty (demand)
• Human capital of young and current generation
• Psychosocial capital
• Making supply side available (supply) and tailored to
the needs of the poorest

9. Data: Panel Survey Chile Solidario
• CASEN 2003: identify a representative sample of ChS
participants and an observationally ‘comparable’ sample of
non-participants
• Rich set of outcomes (health, education, housing, income,
poverty/indigence) CASEN + added new modules:
intergenerational mobility, psychosocial dimension (since
2006)
• Representativeness? cohorts 2002-2003
• Mistake sampling scheme in 2004 sampled better off non-participants
• Followed sample 2004, 2006, 2007
• No baseline data

10. Data: administrative data
• Panel of the administrative registry: Ficha CAS
• Demographics, housing/durables, employment, income,
monetary subsidies
• In 2007 FPS : expanded into attendance to preschool,
school, health visits, more detailed employment module,
• Detailed geographical information: 13 regions, 346
municipalities, 7000 neighborhoods.
• Can distinguish families vs households
• Coverage: 1/3 in 2000/1, to 2/3 after 2007
• Ficha vigente – updated every 2 years. Substantial
churning families in/out

11. The key role of the national ID
• Registro Unico Tributario RUT. use it to merge it to a
large set of complementary data (SIIS presentation
yesterday)
• Merged registry with:
• Administrative data from the program (identity participants)
• administrative data on social workers, caseloads (quality of
implementation)
• participation on training/employment programs (unit level
records available at the time of the evaluation)

12. Descriptives: target of the bottom 5% in each municipality led to
under-coverage nationwide
Proportion of participant among eligible households:
Cumulative entry around 50-60% of the potential target population

13. Descriptives: social exclusion was higher among the
poorest
CAS Distribution
SUF take-up rate
Note: Only families eligible for SUF are considered (heads 20-50 years old).

14. regression discontinuity design: ‘effective’ cutoffs
• Exploit gradual roll out of the program over time to estimate effective
thresholds that vary by municipality, over time
• inference caveat: window CAS within municipality, larger support nationwide

15. Participation to the program by cohorts
• Eligibility defined by the ‘effective’ cutoff

16. Main result: take-up 2 years
11% 1.2%
2.3% 4%

17. Main results: employment 2 years

18. Going beyond average impacts to understand
mechanisms
• Program is tailored to the needs: it is important to
account for the initial conditions of beneficiaries
• compare households who had vs had not satisfied
outcomes before 2002)
• Program evolved over time (supply side):
• compare cohorts before/after 2004
• In theory with current Ficha Social+ expanded
historical information on programs could look at 6-10
years impact

19. Supply side articulation: employment
Preferential access to:
• Self-employment programs (AAE)
• Employability (PNCL)
Exclusive access to:
• Job placement (wage subsidy SENCE, PROFOCAP)
• Self-employment programs (PAME)
• employability (habilitacion sociolaboral SENCE, competencia laboral
mujeres PRODEMU, apoyo jovenes SENCE)
• Share programs exclusive to CS increased over time
• Lion share of programs for self-employment (75%)

20. Results by initial conditions: take-up
• Impact driven mainly by those who were previously
disconnected from social services (SUF)

21. Results by initial conditions: take-up
• Take-up Employment programs driven by those
inactive/unemployed before CHS

22. Final employment outcomes show the importance
of the supply side
• Spouse: activation employment by 20% only for those cohorts
exposed to the supply side expansion

23. To summarize
• Social protection: activation of the demand for a large array of
social programs
• Important demand side constraints (information, transaction costs,
psychic costs activation)
• Impact persists over time
• Social inclusion: activation demand for programs for those who
were previously disconnected
• Take-up of monetary transfers
• Take-up of labor market programs
• Promotion: No. Activation demand does not translate into final
outcomes if not met by supply side:
• Need comprehensive activation component: technical and soft skills
training, internship, job intermediation and placement

24. Quality of the social worker associated with better take-up rates and employment
Quantile of SW
quality
Average SUF
take-up rate
10 0.3602
25 0.5692
50 0.6141
75 0.7019
90 0.8824
Quantile of SW
quality
Avg. prop. Head
emp.
10 0.5414
25 0.7059
50 0.7222
75 0.8700
90 0.9816

25. Conclusions
• Social registries starting data source for impact evaluation:
move beyond monitoring
• Cross check with unit record data on access to services, use of
services)
• Can go to fine level of disaggregation, exploit time dimension
• Might be missing key outcomes (income, psychosocial
dimensions)
• Caveat representativeness, quality
• Even programs are already at scale, refine questions of impact:
quality of implementation, complementarity of services to target
groups, experimentation