Interpreting the evidence on impacts of programmes targeting adolescents by Sonya Krutikova (Institute for Fiscal Studies, EDePo) Young Lives International Conference - Adolescence, Youth and Gender: Building Knowledge for Change, University of Oxford 8 and 9 September 2016

1. © Institute for Fiscal Studies
Interpreting the evidence on impacts of
programmes targeting adolescents
Sonya Krutikova (Institute for Fiscal Studies, EDePo)

2. © Institute for Fiscal Studies
Main aim
The development of a strong unifying conceptual
framework has hugely strengthened the state of
knowledge and design of programmes in the field of Early
Childhood (ECD)

3. © Institute for Fiscal Studies
Main aim
The development of a strong unifying conceptual
framework has hugely strengthened the state of
knowledge and design of programmes in the field of Early
Childhood (ECD)
→What can we learn from the ECD experience in using
programme evaluation to inform design of effective and
scalable programmes for adolescents

4. Outline
• Importance of a conceptual framework for better programme
evaluation and design
• Key components of the ECD conceptual framework
• Adapting the framework to adolescence
• Knowns and unknowns
• The potential of this approach to evaluation for improving
programme design: example from ECD
© Institute for Fiscal Studies

5. The importance of a conceptual framework for
effective programme design
A key ingredient for success of ECD agenda = development of a
unifying framework (Heckman et al):
• Way of interpreting and organising existing evidence
• Understanding the mechanisms through which programmes
work/don’t work – using the framework in programme evaluation
• Identifying outstanding questions and designing studies to
answer them
• Developing scalable, cost-effective programme/policy designs
based on these foundations
© Institute for Fiscal Studies

6. Framing the evidence on adolescents/young
people
• Clear linkages between ECD and adolescence:
– Biologically early childhood and adolescence are the key
developmental stages in HC formation
– Primary aim of programmes in both areas = build human capital
– Both are fields of significant policy and programming activity
– Growing body of experimental evidence
© Institute for Fiscal Studies

7. Framing the evidence on adolescents/young
people
• Clear linkages between ECD and adolescence:
– Biologically early childhood and adolescence are the key
developmental stages in HC formation
– Primary aim of programmes in both areas = build human capital
– Both are fields of significant policy and programming activity
– Growing body of experimental evidence
• However:
– Less coherence in what we are learning from programme evaluation
& how it links to progress in other areas e.g. psychology
– Less clarity about theory of change/assumptions underlying
programme and policy designs
→ Scope for incorporating approaches in ECD to strengthen design
and evaluation of programmes in field of adolescence
© Institute for Fiscal Studies

8. Key components of the ECD framework
(Attanasio, 2015)
– Process of human capital formation:
• Multidimensional
• Dynamic – current HC depends on past levels, some fixed factors (e.g.
Parental education) and some factors that change over time including
investment
• Windows of opportunity – investment in certain periods is particularly
salient
• Investment is chosen by different actors (focus in ECD on parents)
© Institute for Fiscal Studies

9. Key components of the ECD framework
(Attanasio, 2015)
– Process of human capital formation:
• Multidimensional
• Dynamic – current HC depends on past levels, some fixed factors (e.g.
Parental education) and some factors that change over time including
investment
• Windows of opportunity – investment in certain periods is particularly
salient
• Investment is chosen by different actors (focus in ECD on parents)
– Decision process determining investment behaviour:
• Preferences incl. in relation to investment in a specific child, all of the
children in the household
• How decision making process works within the household e.g. role of
mother and father’s preferences in final decision
• Constraints:
– Financial resources
– Information/beliefs re returns to investment
© Institute for Fiscal Studies

10. Adapting this framework to adolescence
• Process of human capital formation:
– Change in the developmental domains of interest
– Change in the types of investments that matter: moving away
from parental investments to own and school
© Institute for Fiscal Studies

11. Adapting this framework to adolescence
• Process of human capital formation:
– Change in the developmental domains of interest
– Change in the types of investments that matter: moving away
from parental investments to own and school
• Decision process determining investment behaviour:
− Adolescent preferences may be different from adults e.g.
Importance of peers, importance of future vs present,
importance of social norms
− Intra-household decision making – bargaining between
parents and children
− Constraints:
− Ability to make forward looking choices?
© Institute for Fiscal Studies

12. Knowns and Unknowns: Production Process
• Knowns:
– Domains of importance: Windows of opportunity different from early
childhood e.g. early childhood - sensory, motor and language skills;
adolescence - development in executive functioning, socialisation
processes, socio-emotional skills
– Types of investments:
• Own investments become important
• quality of school matters
© Institute for Fiscal Studies

13. Knowns and Unknowns: Production Process
• Knowns:
– Domains of importance: Windows of opportunity different from early
childhood e.g. early childhood - sensory, motor and language skills;
adolescence - development in executive functioning, socialisation
processes, socio-emotional skills
– Types of investments
• Own investments become important
• quality of school matters
• Unknowns:
– Windows of opportunity
– Can development in adolescence be effectively increases through
intervention
– How does past investment affect productivity of investments in
adolescence
– What are the key dimensions of school quality
© Institute for Fiscal Studies

14. Knowns and Unknowns: Decision making
• Knowns:
– Adolescent preferences: Preferences and behaviour of peers matter
a lot more in adolescence than other stages
– Ability to make forward looking choices: Adolescents are just as able
to accurately assess risks as adults
© Institute for Fiscal Studies

15. Knowns and Unknowns: Decision making
• Knowns:
– Adolescent preferences: Preferences and behaviour of peers matter
a lot more in adolescence than other stages
– Ability to make forward looking choices: Adolescents are just as able
to accurately assess risks as adults
• Unknowns:
– How do peer preferences and behaviour enter into the decision
making process
– What are the intra-household dynamics of making decisions about
adolescents
– How is school quality produced
© Institute for Fiscal Studies

16. Back to programme evaluation and designing
effective, scalable programmes: an example
• Evidence from the Jamaica studies (Walker et a)
© Institute for Fiscal Studies

17. Back to programme evaluation and designing
effective, scalable programmes: an example
• Evidence from the Jamaica studies (Walker et a)
• Design and evaluation of more scalable version of the
programme (Attanasio et al)
© Institute for Fiscal Studies

18. Back to programme evaluation and designing
effective, scalable programmes: an example
• Evidence from the Jamaica studies (Walker et a)
• Design and evaluation of more scalable version of the
programme (Attanasio et al)
• Find significant programme impacts (Attanasio et al, BMJ)
© Institute for Fiscal Studies

19. Back to programme evaluation and designing
effective, scalable programmes: an example
• Evidence from the Jamaica studies (Walker et a)
• Design and evaluation of more scalable version of the
programme (Attanasio et al)
• Find significant programme impacts (Attanasio et al, BMJ)
• Use additional data to estimate a model which allows to
disentangle the mechanisms – find that effects set in through
change in maternal investment behaviour
© Institute for Fiscal Studies

20. Back to programme evaluation and designing
effective, scalable programmes: an example
• Evidence from the Jamaica studies (Walker et a)
• Design and evaluation of more scalable version of the
programme (Attanasio et al)
• Find significant programme impacts (Attanasio et al, BMJ)
• Use additional data to estimate a model which allows to
disentangle the mechanisms – find that effects set in through
change in maternal investment behaviour
• Design of the next project:
– A) focus on measuring beliefs about the production function
– B) Given this mechanism could the same effects be attained in a
(much more scalable) group setting?
© Institute for Fiscal Studies

21. Conclusion
 An approach like this has potential to:
a) bring more coherence to evidence being generated through
evaluation of adolescent programmes
b) Design future programme evaluation projects in a way that
would systematically build on what we know
© Institute for Fiscal Studies