The document presents findings from the 'Act with Her' program aimed at improving the wellbeing of very young adolescents in Ethiopia, focusing on data from a mixed-methods impact evaluation. It highlights the need for further evidence on the effectiveness of community-based girls' clubs, indicating significant short-term positive outcomes related to mental health, resilience, and knowledge among participants. The evaluation also emphasizes the importance of understanding context diversity and the potential benefits of targeting additional demographics, such as boys and parents.

1. Act With Her Ethiopia: Short-run findings on programming
with Very Young Adolescents
Sarah Baird (George Washington University) and Nicola Jones (ODI)
Primary school students in Ebenat, Ethiopia © Nathalie Bertrams / GAGE 2020

2. Presentation overview
Introduction to GAGE
Existing evidence base and contribution of GAGE impact evaluation
of the Act with Her programme
Research sample and mixed methods impact evaluation design and analytical approach
Short-term findings on primary and secondary outcomes
Next steps in terms of data collection and analysis
Potential to do more (with co-funding)

3. GAGE timeline, age of cohorts and gender equality milestones
YOUNGER
COHORT
10-12 years 12-14 years 13-15 years 15-17 years
GAGE launched
(2016)
2017 2018 2019 2020 2021 2022 2023
Baseline Midline 1 Midline 2 Endline
OLDER
COHORT
15-17 years 17-19 years 18-20 years 20-22 years
SDGs launched
(2015)
Family
Planning
Summit
London (2017)
BMGF
Gender
Strategy
(2018)
ICPD + 25
(2019)
Beijing + 25
Decade for
Action launch
(2020)
Generation
Equality
Forums
(2021)
Women
Deliver
Conference
(2022)
SDGs + 10
(2025)
Very young adolescent stage Mid-adolescent stage
Youth/ adolescent to early adulthood transition
COVID-19
virtual data
collection

4. UNICEF’sMakani
programme
(Jordan)
Community based
organisations
(Jordan)
Act with Her
(Ethiopia)
Plan International –
Champions of Change (Rohingya) ,
starting 2021 World Bank adolescent education
(Bangladesh)
LOST& Na’baa
(Lebanon)
Room to Read
(Nepal)
Programmes GAGE is evaluating
Update: Makani
programme adapting
to covid 19 and gov.
engagement
Update: engagement
& application of
midline findings
Update: engagement
& application of
participatory findings
Update: engagement &
application of
participatory findings
Update: Programme now
rolling out in Jan 21 due to
covid 19

5. Adolescent girl, Amhara, Ethiopia © Nathalie Bertrams /GAGE 2020
Situating the
GAGE/AWH impact
evaluation within the
evidence base

6. Existing evidence on programming with VYA…
 There is very mixed and still limited evidence base on the impact of community-based girl’s
clubs on SRH, nutrition and broader empowerment outcomes (Simac et al., 2019; Austrian et al.
2020; Bandeira et al. 2018; Bandeira et al. 2020; Buehren et al. 2017; Ozler et al. 2020).
Just now starting to be published!
• Take-up generally low (~20%)
• Typically target older and wider age ranges (e.g.
13-22)
• Longest follow up is 4 years post intervention
• Limited programming with boys, parents or
systems (some with cash/asset transfer); and no
experimental evidence on added value of these
components.
These studies also
have some limitations:

7. Contribution to the evidence…
GAGE partnered with Pathfinder to address gaps in evidence on programming with VYA, and
recognising need for co-designing a multi-arm cluster RCT with long-run follow up to begin to
fill the evidence gap.
GAGE’s evaluation of AWH-E is the first experimental evaluation (that we are aware of) that will
be able to assess:
• Whether programming for VYAs (11-13) is effective in improving girl’s wellbeing in the short- (8-10
month) medium- (24 month and 40-48 months), and long-run (new funding would be required)
• Added value of interventions that target boys/parents, promote systems strengthening, and asset-
transfers
• Powered to assess regional effects (Ethiopia’s heterogeneity is akin to working in 3 different
countries)
• Additional questions with next phase of older cohort programming (discuss at the end)
But patience is needed in measuring more traditional adolescent outcomes, particularly in
SRH, child marriage and economic empowerment, as these girls are still VYA (on average 13
years old) so we are measuring outcomes early on in the causal pathway….

8. GAGE 3C’s Conceptual Framework

9. Impact evaluation
sample and
methodology
Adolescent pregnant girl, Ethiopia © Nathalie Bertrams / GAGE 2020

10. GAGE longitudinal research sample

11. Context diversity is critical in understanding
programming starting points
Zones selected based on:
 high prevalence of child marriage
(as proxy for conservative gender
norms) and then
 districts that are designated food
insecure (as proxy for econ
vulnerability)
In South Gondar:
 96% of girls and 89% of boys are
enrolled in school
 Girls have completed a half year
“more” school than boys: 4.8
versus 4.3
In East Hararghe
 85% of boys and 69% of girls are
enrolled
 Boys have completed a half year
“more” school than girls” 5 vs 4.5
In Zone 5, Afar:
 Only 63% of teens are enrolled
 Young adolescents have missed
24% of school days in the last two
weeks
 Average grade attainment is only
2.3 years
School enrolment varies tremendously—and in different ways
Figures
from GAGE
baseline

12. Context diversity is critical in understanding
programming starting points
In Amhara:
 Age of marriage climbed from 14.4 to
16.2 between 2005 and 2016
 “Only” half of girls are cut in Amhara,
3% are infibulated
In Oromia:
 Age of marriage was comparatively
unchanged, 17.1 to 17.4
 Infibulation is rarer in Oromia (7%),
where girls are cut later
In Afar:
 Age of marriage dropped from 16.7
to 16.4
 Nearly all girls under 15 are cut in
Afar—70% are infibulated
Similar diversity exists in terms of child marriage and FGM/C
Figures
from GAGE
baseline
Figures from
DHS

13. Quantitative sample
4,518 VYAs in 173 kebeles
Kebeles within woredas
selected based on mapping of
access to markets,
infrastructure and services
(high and low)
 Amhara: 1,113 girls and 819
boys (74 Kebeles)
 Oromia: 1,190 girls and 878
boys (79 kebeles)
 Afar: 298 girls and 220 boys
(20 kebeles)
Aged 10-12 during GAGE
baseline data collection
(late 2017; midline late
2019 8 months after
programming start)
Randomly selected from a door-to-door
household listing in communities
Randomly assigned kebeles to different
programming arms (and control) with
implementation starting in early 2019

14. Summary statistics (Oromia and Amhara, control)
Oromia Amhara
n=377 n=423
HH size 7.154 5.388
HH head literate 0.269 0.378
HH currently receives PSNP benefits 0.306 0.355
FIES score(0-8, higher is morefood insecure) 5.173 3.537
HH experiences any food insecurity 0.888 0.766
Panel A: Household level variables

15. Summary statistics (Oromia and Amhara control)
Girlsn=216 Boysn=161 Girlsn=241 Boysn=182
Age 12.921 12.969 13.000 13.054
Enrolled in school during most recent session 0.619 0.819 0.987 0.851
=1 if hungry in thepast 4 weeks 0.229 0.368 0.042 0.138
Number ofmeals with animal protein previous day 0.086 0.097 0.164 0.188
=1 if sourceof information on puberty 0.500 0.693 0.828 0.772
Has begun menstruation 0.106 na 0.050 na
=1 if can namea modern method of contraception 0.053 0.195 0.667 0.561
=1 if Ever Married 0.046 0.006 0.046 0.000
=1 if Has savings 0.686 0.675 0.344 0.260
=1 if Has not experienced/witnessed HH violencein
last 12 months
0.444 0.638 0.451 0.567
=1 if Has an adult s(he)trusts 0.452 0.716 0.697 0.674
Panel B: Individual level variables
Oromia Amhara

16. Qualitative sample
1 kebele per intervention arm and 1 control
per region
(i.e. 5 kebeles in Amhara and Oromia and 2
in Afar)
275 adolescents (179 HS or AWH
participants)
Inclusion of socially disadvantaged adolescents
based on disability (10%) and marital status (10%)
but largely non-programme participants
Amhara:
 112 adolescents
(77 HS or AWH participants)
 73 parents
 10 mentors
 34 key informants
Oromia:
 112 adolescents
(77 HS or AWH participants)
 87 parents
 13 mentors
 33 key informants
Afar:
 51 adolescents
(25 AWH participants)
 48 parents
 7 mentors
 10 key informants

17. Analysis approach for short-term findings
 Her Spaces vs. Control (Arm 1 vs. Control)
 AWH-E vs. control: Combined treatment of AWH-E (curriculum) and AWH-E (curriculum +
systems) vs. control. (Arm 2+3 vs. control) Will call this AWH-E in this presentation.
 Her Spaces vs. AWH-E
Focus on three main comparisons:
 Asset transfer really changed the intervention (in both good ways and bad) so we
look at it in comparison to the other two AWH-E arms separately (not part of core
AWH-E package).
 Community level social norms and systems strengthening activities were 25%
complete at the time of this survey; and thus not surprisingly quantitative findings
show limited difference in impact on any outcomes with the addition of systems
strengthening (medium-term findings will focus on added value of systems
strengthening)
Why this approach?

18. Analysis approach for short-term findings
 All adolescents and parents in our study who live in an AWH community are
considered treated, whether they were part of the program or not.
 Generally considered the policy relevant outcome (as uptake of these programs is
always less than 100%, and often much lower)
These are intent to treat estimates:
 Interest in the program was high across regions (much higher than generally seen
with these programs)
 Still manually entering attendance data to give exact numbers on enrollment and
attendance (time consuming!!)—working for better process moving forward
Future quasi-experimental analysis will look at treatment on the treated
(e.g. are there larger impacts for those who actually participated in AWH)
Full pre-analysis plan can be found here. Trial is registered at AEA RCT Registry
(#AEARCTR-0004024), ClinicalTrials.gov (#NCT03890237), and went through the
BMGF review process.

19. Analysis approach for short-term findings
Qualitative data
was thematically
coded in MAXQDA
and disaggregated by
region, gender and
by participant status.
Qualitative data
used to triangulate
findings from quant
survey, to provide
context specificity to
findings, and to
disentangle potential
pathways to explain
positive, negative or
non-effects.
Key informant
interviews
provided additional
understanding of
community, district
and zonal political,
economic and social
dynamics.
Quotes
are used to illustrate
findings; where they
represent the voices
of outliers this is
noted.

20. Capturing impact pathways via longitudinal
evaluations (within a messy eco-system)
Exposure to knowledge
about menstruation,
pregnancy and risks of
early marriage
Retention of knowledge
about timing of
menarche, how to
manage menstrual
health, contraceptives,
how to report marriage
etc.
Translation of knowledge
into seeking of supplies
and support to manage
menstrual health,
prevent pregnancy and
resist child marriage
pressures
Availability of services
and resources, and shifts
in community norms to
support uptake of
sanitary products at
home and school, use of
contraceptives to delay
fertility, and decisions to
delay marriage
Improved menstrual
health for girls with
support by families and
community; delayed
fertility; delayed
marriage; successful
educational transitions
and acquisition of skills
to support future
economic empowerment
Captured by M&E
data
Captured by short-run
survey and qual data
Captured by longer-run
survey and qual data
Captured by longer-run
qual + key informant
interviews
Captured by
longer-run survey
+ qual data

21. High-level findings on
short-term outcomes
for girls
Adolescent girl in Oromia, Ethiopia © Nathalie Bertrams / GAGE 2019

22. Girls’ primary indices (subset)
 Indices ensure not ‘cherry picking’ outcomes of impact
 Coefficients generally all in the right direction (positive is good), with larger coefficients for the AWH
arm, Limited significant difference between Her Spaces and AWH in the short term, except for mental
health and attitudes
 SRH limited to those already menstruating (~10% of sample)
 Significant impacts on mental health, resilience, voice and agency and knowledge provide
a pathway to longer-run impacts.
Violence
Ideal Age
of Marriage
Health and
Nutrition
SRH
Mental
Health
Resilience
Voice and
Agency
Gender
Equitable
Attitudes
Knowledge
Her Spaces 0.071 -0.029 0.090 0.044 0.057 0.735** 0.189*** -0.080 0.292***
(0.062) (0.504) (0.080) (0.249) (0.131) (0.320) (0.069) (0.067) (0.086)
AWH (no assets) 0.029 0.055 0.069 0.201 0.272** 0.748*** 0.272*** 0.059 0.342***
(0.058) (0.481) (0.065) (0.197) (0.126) (0.282) (0.058) (0.062) (0.068)
P-value: B1 /= B2 [0.475] [0.844] [0.790] [0.520] [0.027] [0.969] [0.149] [0.030] [0.510]
Control Mean 0.000 22.435 -0.000 0.000 26.412 31.310 0.000 0.000 0.000
Observations 1553 1534 1612 189 1520 1372 1488 1584 1536

23. Girls’ primary indices (subset)
 Girls in communities assigned to receive the asset transfer also showed improvements
in Voice and Agency and Knowledge
 Did not see impacts on mental health or GEA (and coefficients are negative)
 Stronger impacts on economic empowerment
 Best program depends on outcome you care about (trade-offs)
Violence
Ideal Age
of Marriage
Health and
Nutrition
SRH
Mental
Health
Resilience
Voice and
Agency
Gender
Equitable
Attitudes
Knowledge
Economic
Empowerment
AWH (no assets) 0.029 0.055 0.069 0.201 0.272** 0.748*** 0.272*** 0.059 0.342*** 0.101
(0.058) (0.481) (0.065) (0.197) (0.126) (0.282) (0.058) (0.062) (0.068) (0.082)
AWH assets 0.120* -0.230 -0.063 0.250 -0.032 0.337 0.169** -0.072 0.332*** 0.214**
(0.067) (0.535) (0.066) (0.193) (0.191) (0.307) (0.074) (0.078) (0.085) (0.098)

24. Normalization of menstruation (qual and quant)
‘Thanks to the education, we don’t feel shame
about it currently.’ (Girl, Her Spaces)
‘We learn to manage menstruation with a
clean piece of cloth so that we can attend class
conveniently. The menstrual flow will not spill
down on our legs.’ (Girl, assets)
Girl in East Hararghe © Nathalie Bertrams / GAGE 2020
 Sig. positive impacts on improved menstrual
hygiene and menstrual management
(many girls not menstruating yet)
 Where groups were running well, there was
good learning regarding puberty and
menstrual hygiene management.

25. Room for improvement on knowledge about SRH (control)….
Girlsn=216 Boysn=161 Girlsn=241 Boysn=182
% Correctly identifiied foods that contain iron vs.
not
0.480 0.498 0.607 0.569
Correctly identifies healthy number ofmain meals
per day for an adolescent
0.833 0.839 0.983 0.917
Correctly answers: Approximately how often
does menstruation usually occur? (Once per
month, every four weeks, or once per 28
days=1, any other frequency=0, DK=0)
0.281 0.176 0.584 0.374
Correctly answers: Getting pregnant early in
puberty can be bad for the health of the girl
and the baby. (True=1, False=0, DK=0)
0.641 0.588 0.693 0.641
Correctly identifies the legal age of marriage
for girls 0.068 0.210
Correctly identifies that FGM/C hasrisks 0.157 0.213 0.510 0.389
Knows whereto get help for violence 0.292 0.522 0.269 0.55
Knows whereto keep money 0.734 0.759 0.887 0.93
Knowledge
Oromia Amahra

26. ‘They told us that they’ll give her a pill right away and they’ll
make sure the offender gets proper punishment too. If a girl
is raped, they told us that she has to take examinations at
the clinic.’ (girl, Her Spaces)

27. ‘Since I joined this program, I understand that child marriage is a harmful
tradition. Therefore, I would tell them that I will only marry once I completed
my education and once I am self-sufficient.’ (12-year-old girl, AwH)

28. Mixed effects on parent-adolescent communication (quant and qual)
‘Previously I was shy to express my thoughts
even in the classroom. But now I can express
my feelings and thoughts.’
(13-year-old girl, assets)
‘If they’re planning to make me skip school so
that I’ll do something for them, I’ll tell them
whatever it is they want me to do, I’ll get it
done after school.’ (Girl, assets)
 Girls report being better able to speak up—
at home and at school.
 BUT– the overwhelming theme is
that girls are obeying more specifically
because of what they’ve learned in AWH.
‘The thing is whenever they order us
something we used to disobey them but now
that we took the lessons we don’t do that
anymore….’ (Girl, Her Spaces)
‘I used to say no and be rebellious... But
now…I will always say yes.’
(Girl, curriculum)
 HOWEVER —improved obedience is improving mother-child relationships—and
may be strengthening resilience because girls feel more supported.

29. Shifting gender attitudes is not straightforward
‘The learners listen to the education when they attend it. They forget
it when they go out of the education session. They put it aside when
they meet male adolescents.’ (13-year-old boy, AwH, Oromia, reflecting on
girls’ submissive behaviour around boys)

30. Summary and
next steps
Adolescent girls in East Hararghe, Ethiopia © Nathalie Bertrams / GAGE 2019

31. Summary
Largest short-run impacts are on adolescent voice and agency, and girls’ knowledge, with suggestive
evidence on mental health, resilience, and menstruation.
• Suggests pathways for longer-run impacts, with current data collection (2021) set to give another
data point.
• Can now assess more SRH outcomes: most if not all girls should be menstruating, can now ask
more sensitive SRH and child marriage questions (also important role for qualitative)
• Will also be able to speak to questions around added value of systems strengthening work
(and specifically systems strengthening via the school system)
Virtual COVID-19 surveys will also allow us to assess whether AWH-E was protective during the
pandemic
Random sample surveyed prior to the election, rest afterwards: assess how election impacts
outcomes, and how AWH interacts with this.

32. Key
hypotheses
to
explore
in
subsequent
analyses Promise of MHM
• Programme efforts to shift norms around menstruation indicate positive traction among adolescent
girls, boys, parents, community stakeholders; we hypothesise we are most likely to see sustained
impact in this domain.
Supportive
parents
• The role of supportive parents is critical and given mothers vs fathers play leading role in reinforcing
different conservative gender norms (e.g. FGM/C vs child marriage vs GBV) programme arms which
involve both parents are more likely to see sustained impact in gender norm change
Sustained gender
norm change
• Time-bound adolescent empowerment programming is likely to see greater impacts on sustained gender
norm change where i) at least upper primary education for girls is widespread, ii) shifts in norms have
already taken hold to an extent (e.g. declining trends in FGM/C in Amhara vs Afar or Oromia), iii) synergistic
partnerships between religious leaders, government officials and programme are facilitated and resourced
Context
complexity
• Norm change is messy and complex and requires a longer-term horizon; adaptive programming cycles
informed by in-depth learning about context-specific dynamics is critical to maximise a return on
investments rather than moving to rapid scale up
Asset/ cash
transfers
• Assets or cash transfers can play a complementary, reinforcing role by helping to reduce economic
barriers to girls’ agency and education, but targeting and messaging need to be carefully managed to
avoid backlash

33. Next Steps: Older Cohort Programming + Longer term VYA impacts
 Will also introduce AWH OA into urban areas (Debre Tabor, Amhara)
allowing us to assess differential impacts of programming in rural vs.
urban areas
First round of
VYA
programming
in research
kebeles
(2019-21)
Second
midline
OA
programming
in research
kebeles
(2021-22)
Arm 1 (HS)
Arm 2
(AWH
curriculum only)
girls + boys
OA
Arm 3
(AWH)
girls + boys +
parents +
systems
strengthening
Arm 4
(AWH + Asset)
Control
OA OA

34.  Undertake in-depth studies of positive and negative outlier adolescents in terms of programme
participation to better understand impact dynamics – including in urban vs rural vs pastoralist communities
 Undertake inter-generational case studies to explore interactive effects of adolescent and parent
participation, and complementary role of systems strengthening to improve programming
What else would we like to do (but need co-funding for 2022/2023 data round (and beyond))
 Biomarkers: STIs, cortisol, height/weight, anemia, etc. including with older cohort (for 2022 data collection)
 Valuable for looking at impacts; and also for broader assessment of SRH/nutrition of this population
More outcome measurement related to mental health, nutrition and SRH
 Rare opportunity to see whether impacts improve when the program launches again, building
on lessons learned from the first wave.
 Could compare systems strengthening work via schools vs non-school routes (e.g. via health
extension or justice system) [although not currently in implementation plan]
Evaluate second cohort of VYA (with improved programming)
 Use lab in the field experiments
 Validate new measures of gender attitudes including with two age cohorts
 Add in tools such as the implicit association test
Additional innovative gender measurement
Ethnographic analyses

35. GAGE
recent
reports
on
mid-line
and
AwH
impact
evaluation
f
indings
For
more
resources
see:
https://www.gage.odi.org/publications/

36. Recent
journal
publications
on
Ethiopia
https://www.gage.odi.org/publication/b
etwixt-and-between-adolescent-
transitions-and-social-policy-lacunae-in-
ethiopia/
https://www.gage.odi.org/publication/constrai
ned-choices-exploring-the-complexities-of-
adolescent-girls-voice-and-agency-in-child-
marriage-decisions-in-ethiopia/
https://www.gage.odi.org/publication/leave-
no-adolescent-behind-addressing-inequities-
in-educational-attainment-in-rural-ethiopia/
https://www.gage.odi.org/publication/gender-
norms-violence-and-adolescence-exploring-how-
gender-norms-are-associated-with-experiences-of-
childhood-violence-among-young-adolescents-in-
ethiopia/
https://www.gage.odi.org/publication/do-restrictive-
gender-attitudes-and-norms-influence-physical-and-
mental-health-during-very-young-adolescence-evidence-
from-bangladesh-and-ethiopia/
https://www.gage.odi.org/publicati
on/intersecting-inequalities-
gender-and-adolescent-health-in-
ethiopia/
https://www.gage.odi.org/publication/explor
ing-the-role-of-evolving-gender-norms-in-
shaping-adolescents-experiences-of-
violence-in-pastoralist-afar-ethiopia/

37. Contact Us
WEBSITE
www.gage.odi.org
TWITTER
@GAGE_programme
FACEBOOK
GenderandAdolescence
About GAGE:
 Gender and Adolescence: Global Evidence
(GAGE) is a nine-year (2015-2024) mixed-
methods longitudinal research programme
focused on what works to support
adolescent girls’ and boys’ capabilities in the
second decade of life and beyond.
 We are following the lives of 20,000
adolescents in six focal countries in Africa,
Asia and the Middle East.