SPIR Impact Evaluation Midline Webinar

SPIR Impact Evaluation Midline Webinar
April 2, 2020
Photo credit: Laura Zimmerman

Webinar agenda
TIME
(GMT+3)
AGENDA ITEM PRESENTER
3:00-3:20pm Opening remarks and overview of
study design
Michael Mulford – Chief of Partya
Dan Gilligan - Deputy Division Directorb
3:20-3:35pm Program delivery Heleene Tambet - Research Analystb
3:35-3:55pm Livelihoods outcomes Jessica Leight - Research Fellowb
3:55-4:10pm Child nutrition and health care Harold Alderman – Senior Research Fellowb
4:10-4:25pm Gender norms, women’s
decisionmaking, and their wellbeing
Melissa Hidrobo – Research Fellowb
4:25-4:55pm Questions and answers Lucy Billings – Senior Project Managerb
4:55-5:00pm Closing remarks Dan Gilligan - Deputy Division Directorb
a World Vision International
b International Food Policy Research Institute – Poverty Health and Nutrition Division

SPIR Impact Evaluation Midline Webinar
Introduction and Study Design
April 2, 2020

Motivation for the SPIR impact evaluation
• As part of the SPIR learning agenda, the impact evaluation
provides evidence on impact and for learning
• SPIR: integrated ‘graduation model’ social protection program
• Graduation model - multisectoral investments (asset building,
income generation) to provide a “big push” to promote
sustained poverty alleviation
– Ex: BRAC Targeting the Ultra-Poor (TUP) (Banerjee et al. 2015)
• implemented in 6 countries, including Ethiopia
• broad impacts on food security, consumption, assets
• no impact on women’s empowerment
– Still seeking effective multifaceted packages for nutrition, child
development, women’s empowerment and mental health

SPIR will fill a number of evidence gaps
• effectiveness of “graduation model” programs
• the promise of poultry value chains as a female-friendly
investment for the poor
• the impact of promoting higher aspirations
• whether reducing poverty improves mental health
• effect of cash transfers on intimate partner violence

SPIR Livelihood (L) and Nutrition (N)
Interventions
Intervention Components
L • Village Economic and Social Associations (VESAs)
• financial literacy
• agriculture and livestock value chain development
• home gardening and forage production
L* • L activities, plus
• Social Analysis and Action (SAA) to improve women’s
access to markets
• aspirations promotion activities, and
• poultry or cash livelihood transfers for poor women
N • Nutrition behavior change communication (BCC)
• WASH activities
• cascade training of health extension worker supervisors
N* • N activities, plus
• Timed and Targeted Counseling and CPNP
• male engagement in BCC
• Interpersonal Therapy in Groups (IPT-G) for depression

SPIR impact evaluation design
SPIR Livelihoods
+ SAA
+ aspirations
+ poultry/cash
SPIR
Livelihoods
Nutrition
SPIR Nutrition
+ TTC + CPNP +
male engagement
+ IPT-G
T1 T3
SPIR Nutrition
T2 C
Note: All components receive PSNP benefits.
Livelihoods
L*+N* L +N*
L*+N PSNP only
A Randomized Controlled Trial - 4 Treatment Arms

SPIR Livelihoods
+ SAA
+ aspirations
+ poultry/cash
SPIR
Livelihoods
Nutrition
SPIR Nutrition
+ TTC + CPNP +
male engagement
+ IPT-G
T1 T3
SPIR Nutrition
T2 C
Livelihoods
Main treatment arms:
T1=L*+N* T3=L+N*
T2=L*+N T4=PSNP
Sub-treatments:
A=aspirations PP=poultry package
CG= cash grant
T1+A
T2+A

SPIR Livelihoods
+ SAA
+ aspirations
+ poultry/cash
SPIR
Livelihoods
Nutrition
SPIR Nutrition
+ TTC + CPNP +
male engagement
+ IPT-G
T3
SPIR Nutrition
C
Livelihoods
Main treatment arms:
T1=L*+N* T3=L+N*
T2=L*+N T4=PSNP
Sub-treatments:
A=aspirations PP=poultry package
CG= cash grant
T1+CG T1+A+
CG
T2+A+
CG
T1+PP T1+A+
PP
T2+A+
PP
T2+CG
T2+PP

SPIR impact evaluation data collection
• Baseline survey: February – April 2018
– informed the context: differences between Amhara,
Oromia
– tested balance of the randomization
• Midline survey: July 25 – October 12, 2019
– evidence on delivery and participation
– provides lessons about implementation
– impacts measured at an intermediate stage of program
delivery

Midline survey sample
• baseline-midline panel: PSNP client households with a child
<age 3 and primary female caregiver at baseline
• midline supplemental sample: 4 PSNP client households per
kebele with a child age 0-23 months
• Attrition was low (n=114)
– attrition rate: 2.8%; Amhara, 2.6%; Oromia, 3.0%
– reasons: most moved out of area; only 17 refusals
Midline sample Amhara Oromia Total
Kebeles 112 80 192
Baseline panel households 1,921 1,393 3,314
Midline panel households 1,862 1,358 3,220
Midline supplemental 441 307 748
Midline total 2,303 1,665 3,968

Midline survey research outputs
• Midline survey report: March 4, revised April 1
• Learning briefs:
1. livelihood interventions: focus on impact of poultry and cash
livelihood transfers
2. SPIR impact on child nutrition and caring practices
3. SPIR impact on women’s empowerment, mental health and
intimate partner violence
• Research papers under development

SPIR Impact Evaluation Midline Report
Evidence around program participation

Overview
We will report patterns by treatment arm from questions about
SPIR program participation and exposure to related services as
reported in the midline survey:
– VESA group membership
– Livelihoods’ programming
– Discussions on gender, health & nutrition
– Livelihood transfers & aspirations treatment
– Exposure to health services

VESA membership
Someone in household
By gender

• Past 12 months
• Conditional on
attending at least
once, median
number of
meetings =
12/year
Attended at least one VESA discussions

Participation in financial education training

SPIR value chain trainings, by gender
• General value chain
development
discussion within
VESA groups
• Trainings on specific
value chains offered
selectively
• Most popular:
– Females: poultry
– Males: sheep/goat,
poultry

Membership in producer marketing groups
associated with SPIR value chains
• Offered in limited
locations
• Popular value
chains:
– Poultry
– Sheep/goat

VESA discussions: gender topics
Recalled topics Female Male
Workload sharing 71.8% 75.2%
Assigning tasks to
boys/girls
69.7% 69.8%
Respect/understanding 63.7% 61.6%

VESA discussions: nutrition topics
IYCF practices 30.6% 29.9%
Maternal nutrition 21.8% 19.3%
Nutritious food
preparation
17.4% 17.3%

VESA discussions: hygiene/health topics
Handwashing 30.8% 29.5%
Food hygiene 35.0% 26.2%
Latrine use and waste
management
25.8% 28.0%

Summing up: VESA membership and
discussions
• About 80% of HHs in T1—T3 have a member who belongs to a
VESA group
– Men report higher membership rates of their household
• Slightly below half of respondents in T1—T3 have attended
VESA discussions
• However, conditional on having attended, the median
frequency of participation is high at 12/year
• Between ~40-50% of attendants report exposure to gender,
nutrition and health discussions in all three treatment arms
– Health and hygiene topics have the highest reported exposure
– Conditional on attending, individual gender topics are most widely
recalled

POULTRY/CASH TRANSFER
&
ASPIRATIONS TREATMENT

Receipt of poultry start-up package, by treatment

Number of pullets received vs pullets remaining
Only HHs assigned to treatment
Reasons for reduction

Receipt of cash, by treatment assignment

Amount of cash, by treatment assignment

ASPIRATIONS TREATMENT
• 4 short documentaries of 15 minutes each
• Each tells a life story of an individual significantly improving
their well-being despite adverse conditions
– All subjects from Amhara or Oromia
– Two videos about men, two about women
• Common themes:
– Ordinary rural, rather poor residents who take proactive action
(starting a small business, diversifying income, improving farming
practices, etc.)
– Highlight personal qualities: perseverance, determination, reliability
– Individuals succeed largely through their own efforts

Aspirations movie screening
Sample assigned to treatment

Summing up: livelihood transfers & aspirations
• Among assigned households, 95% in T1 and 93% in T2
report receiving the poultry package
– Females are 17 percentage points more likely to report receiving
the package
• Reported receipts of cash grants are 60% in T1 and 66% in
T2 within assigned households
– Difference between females’ and males’ reports is smaller at 11
percentage points
• Reductions in chicken flock consistent with poultry business
cycle, although deaths of chickens are also reported
• Aspirations video screenings have low reported attendance
of 41% among females and 50% among males

EXPOSURE TO HEALTH SERVICES
• SPIR trains Community
Health Facilitators,
Health Extension
Workers (HEW) and
Health Development
Army (HAD) leaders in
N* kebeles (T1 and T3)
to facilitate the
programming
• More interaction
with HDAs in T1 and
T3—especially in
past 3 months,
especially at home

SPIR Impact Evaluation Midline Findings
Evidence around livelihoods

Overview
• Our primary objective is to report the effects of SPIR
programming on five categories of outcomes.
– Livestock ownership and associated income
– Credit access and savings
– Outside labor
– Investment in cropping
– Aspirations

Analysis
• In the report, we present the effect of SPIR programming on
three samples of interest:
– The full sample of all households, analyzing the effects of
T1, T2, and T3 relative to the control arm.
– The poor only sample of households, analyzing the effects
of poultry and cash transfers (in conjunction with
aspirations) as well as T3 relative to the control arm.
– The sample of non poor households (not eligible for cash
and poultry), analyzing the effects of T1 and T2 (in
conjunction with aspirations) as well as T3 relative to the
control arm.

This presentation
• Here, we present the separate effects for poor and non-poor
households only for livestock related outcomes.
• For the other outcomes of interest, there was little evidence
of heterogeneity comparing across the poor and non-poor
samples.
• Accordingly, we focus on the effects observed in the pooled
sample.

Livestock ownership: Pooled results

Livestock ownership: Non-poor sample

Livestock ownership: Poor sample

Other livestock outcomes: Poor sample

Summing up
• For poor, households, the poultry transfer had a substantial
positive effect on increasing household ownership of and
income from poultry.
• Comparable investments by households who received the
one-time cash transfer are smaller; the increase in the
probability of owning poultry is 39 percentage points for
poultry, compared to 9 percentage points for cash.
• Poultry households also show differentially larger investment
in other forms of livestock vis-à-vis cash recipient households.

Summing up, cont.
• Non-poor households exposed to base L* programming also
increase their engagement in poultry production, showing an
increase in the probability of owning poultry of between 8
and 10 percentage points.
• There is no effect for households exposed only to L
programming only.

Savings and credit: Full sample

Summing up
• For savings and credit, the effects are consistent across all
three treatment arms.
• There is a dramatic increase in the probability that women
report any savings (between 36 and 49 percentage points),
driven primarily by women reporting savings in VESAs; this
pattern is observed for both poor and non-poor women.
• By contrast, there is little evidence of any shift in credit
access.
• The substantial effect on female savings is consistent with the
observed pattern of consistently high participation in VESA
groups.

Expenditure on crop inputs: Full sample

Summing up
• In general, there is little evidence of any effect on labor
outside the household, or household investment in cropping.
• For poor households, there are some moderate effects on
outside labor for certain subtreatment arms, but this evidence
is not consistent and should be interpreted tentatively.
• Unfortunately, the midline survey did not collect data on
household engagement in non-agricultural businesses; this
will be explored further at endline.

Female aspirations: Full sample

Summing up
• In general, there is little evidence of any of the interventions
(including the cross-randomized aspirations video treatment)
on reported aspirations of male or female respondents for
their children’s education.
• Households in T1 only do show some evidence of an increase
in aspired income, but this does not seem to reflect the
effects of the targeted cross-randomized aspirations
intervention.
• The aspirations intervention was also characterized by
relatively low participation (less than 50%) in the data
reported by households, though this pattern will be explored
further in administrative data.

SPIR midline evidence around livelihoods
• There is evidence of substantial positive effects of both the
poultry and the cash transfers randomized to poor
households, though the effects of the poultry transfer seem
to be significantly larger for the variables measured.
• There is also evidence of positive effects of L* programming
for non-poor households, particularly for livestock production
and female savings.
• The effects of L programming are smaller and often not
statistically significant.
• There is little evidence of any effect of the cross-randomized
aspirations treatment, and little evidence of experimental
effects on aspirations overall.

Evidence on nutrition and health care

Bottom Line on Top
While the overall goal on nutrition is to reduce stunting,
it is too early to see results on this cumulative measure
There are promising improvements in a number of
health seeking behaviors, among them:
➢ Regular measurement – with follow up
➢ Contact with HEW and HDA
➢ Contributing to improvements in breastfeeding
➢ Small improvements in complementary feeding
But there is a long way still to go on diet diversity

Proportion of index children measured in past 6 months
The frequency, however, is less than planned for

Advice given after measurement and CPNP participation
Growth monitoring is becoming growth promotion

Participation in food demonstrations and CLTSH events

Breastfeeding practices – already good - improved
But complementary feeding still has a long way to go

Food groups consumed, children 6–23 months
Still, even with a slight increase in diet diversity the N* kebeles are far below
the recommended 5 food groups (including breast milk)

The effect of poultry transfers on egg consumption
The increase in egg consumption observed for children has a
parallel with mothers

Conclusions
The N* interventions—and to a lesser degree the N interventions-
have shown success in increasing exposure to health services
including BCC, food demonstrations, and WASH.
Similarly, communication measures increased significantly; over
half of caregivers for the children found to be underweight
participated in camps aimed at addressing malnutrition.
These increases reflect the goals of the SPIR design. Nevertheless,
the frequency of contacts between households and service
providers is less than planned and may indicate start-up
processes at the time of the midline.
This then provides a basis to expect continued improvements over
the SPIR program.

Evidence around gender norms, women’s
decisionmaking, and their wellbeing

Overview
• Objective is to report the effects of SPIR programming on
gender norms, women’s decision-making, depression, and
intimate partner violence (IPV)
• Conceptual framework
– L and N activities: bring women and men together to provide
them with knowledge and training over a wide range of topics,
thereby improving social capital and catalyzing women’s
empowerment.
– L* and N*: address constraints on women’s role in
intrahousehold decision-making, mobility, choice of livelihood
activities, and access to markets that derive from cultural and
social norms (through SAA); target women for the poultry and
cash transfers; include men in the nutrition BCC; and provide
IPT-G for women and men screened for depression.
– At the time of the midline survey, however, neither the male
engagement groups nor IPT-G groups had begun.

Analysis
• In the report, we present the effect of SPIR programming on
three samples of interest:
– The full sample of all households, analyzing the effects of T1
(L*+N*), T2 (L*+N), and T3 (L+N*) relative to the control arm.
– The poor only sample of households, analyzing the effects of
poultry and cash transfers (in conjunction with aspirations) as
well as T3 relative to the control arm.
– The sample of non poor households (not eligible for cash and
poultry), analyzing the effects of T1 and T2 (in conjunction with
aspirations) as well as T3 relative to the control arm.
• For this presentation we focus on the full sample results only

Gender norms
• Gender equitable indicators
– binary indicator that equals one if the respondent says
a husband is not justified in beating his wife under any
of the four circumstances (primary male and female).
– binary indicator that equals one if the respondent says
it is acceptable for a woman to travel alone to each of
the three specified places (primary male and female).
– binary indicator if respondent’s gender equitable
norms score is above the median (list of 10 gender
norms questions, primary male only)

Gender norms
• T1 marginally increases
the probability that a
man is above the
median on the gender
equitable scale by 6 pp
• T2 increases the
probability that men
report that it is never
justified for husbands
to beat their wives and
that it is acceptable for
women to travel alone
by 6 pp

Women’s decision-making
• Decision-making with respect to input into productive
decisions on horticulture, large livestock, small
livestock, and poultry
– for all animal categories, the question was about both
raising and processing of meat/milk/eggs.
• Two types of productive decisions for each activity:
– the quantity of inputs that should be used
– the quantity of the output that should be sold or
consumed at home.
• We create binary indicators that equal one if a woman
reports having input into most or all decisions for each
activity and decision type

Decision-making
• T2 marginally increases
the probability that a
woman reports
contributing to most
decisions related to
input use for small
livestock by 6.4 pp and
sale or consumption of
the output from large
livestock by 7.6pp

IPV
• Internationally validated standardized IPV measures
from the WHO Violence Against Women Instrument
– Instrument asks a woman if she has ever experienced the
act of violence and if she had experienced it in the last 13
months.
– For each type of violence, multiple behaviorally specific
questions were administered in order to reduce under-
reporting
• Create 4 indicators for marital control, emotional
violence, physical violence and sexual violence in the
last 13 months

IPV
• No impacts of any
treatment arm

Summing up
• T1 and T2 improves men’s but not women’s gender equitable
attitudes/norms (not significantly different from T3)
• T2 improves women’s decision-making around small and large
livestock but not horticulture or poultry (significantly different
from T1 and T3)
• T2 and T3 decrease the probability that a women has a PHQ-9
of 8 or higher, but no impacts on PHQ-9 scores of 10 or higher
and no impacts on men’s depression (not significantly
different from T1)
– impacts are concentrated among women with relatively low
levels of depressive symptoms
• No impacts on IPV of any treatment arm for the full sample

Conclusion
• Caveats:
– Many impacts are marginally significant (significant at the 10
percent level)
– Testing many indicators which leads to issues of multiple
hypothesis testing
• But, the handful of positive impacts across T1, T2, and T3
are promising given that not all activities had been rolled
out by midline and some outcomes might take more time
to change, such as personal norms regarding gender

SPIR Impact Evaluation Midline Webinar

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to SPIR Impact Evaluation Midline Webinar

Similar to SPIR Impact Evaluation Midline Webinar (20)

More from International Food Policy Research Institute (IFPRI)

More from International Food Policy Research Institute (IFPRI) (20)

Recently uploaded

Recently uploaded (20)

SPIR Impact Evaluation Midline Webinar