Targeted Evaluation of Five Programs Supporting Orphans and Vulnerable Children: Background and Methods Florence Nyangara, PhD MEASURE Evaluation/Futures Group Dissemination Meeting, September 3 rd , 2009 Washington, DC

The number of OVC and their corresponding programs increasing SSA - 12 million orphans (0-17 yrs), 2003 2010 - over 18 million orphans (O) Other millions are made vulnerable – HIV/AIDS, dire poverty, war, etc (V) Response – increased attention to the plight of OVC (funds, programs) Children on the Brink , 2004 Sub-Saharan Africa’s population of children orphaned by AIDS increasing Background

SSA - 12 million orphans (0-17 yrs), 2003

2010 - over 18 million orphans (O)

Other millions are made vulnerable – HIV/AIDS, dire poverty, war, etc (V)

Response – increased attention to the plight of OVC (funds, programs)

Response to the OVC Crisis OVC programs – emergency response to areas most HIV-affected Strategies used were based on existing cultural support systems, conventional wisdom, and lessons learned from other program areas; Support community-based responses (capacity & resources) Household/family support (capacity & resources) Direct support to families & OVC (access to essential services) Gap – lack of evidence to guide OVC programs Call for evidence based programming 2006 - USAID funded MEASURE evaluation to conduct targeted evaluations to fill this evidence gap Background

OVC programs – emergency response to areas most HIV-affected

Strategies used were based on existing cultural support systems, conventional wisdom, and lessons learned from other program areas;

Support community-based responses (capacity & resources)

Household/family support (capacity & resources)

Direct support to families & OVC (access to essential services)

Gap – lack of evidence to guide OVC programs

Call for evidence based programming

2006 - USAID funded MEASURE evaluation to conduct targeted evaluations to fill this evidence gap

Evaluation Goals Find out “what works” in terms of intervention models and program components cost effectiveness, and outcomes (benefits) for OVC and their caregivers in resource poor settings Provide evidence to guide program decisions such as; Scaling-up of best practices (models, strategies), and Modify & improve interventions - to make them effective

Find out “what works” in terms of

intervention models and program components

cost effectiveness, and

outcomes (benefits) for OVC and their caregivers in resource poor settings

Provide evidence to guide program decisions such as;

Scaling-up of best practices (models, strategies), and

Modify & improve interventions - to make them effective

Research Preparation Activities Funds were made available – USAID/PEPFAR/PHE (4) and USAID/Tanzania mission funded (1) program evaluation. Research team formed - MEASURE Evaluation Extensive literature reviews (early 2006) Consultation meetings with stakeholders Identified OVC programs to be evaluated Research protocol developed Ethical approvals obtained – US, Kenya, and Tanzania Identified local research partners (PSRI – KE; AXIOS - TZ)

Funds were made available – USAID/PEPFAR/PHE (4) and USAID/Tanzania mission funded (1) program evaluation.

Research team formed - MEASURE Evaluation

Extensive literature reviews (early 2006)

Consultation meetings with stakeholders

Identified OVC programs to be evaluated

Research protocol developed

Ethical approvals obtained – US, Kenya, and Tanzania

Identified local research partners (PSRI – KE; AXIOS - TZ)

Programs Evaluated Evaluated Five programs : 2 in Kenya & 3 in Tanzania They have different intervention models with v aried combinations of child, family/household, and/or community centered approaches (multi-faceted). *** Although, the approaches vary, the goal for all of these programs is to improve the well-being of OVC and their families.

Evaluated Five programs : 2 in Kenya & 3 in Tanzania

They have different intervention models with v aried combinations of child, family/household, and/or community centered approaches (multi-faceted).

*** Although, the approaches vary, the goal for all of these programs is to improve the well-being of OVC and their families.

Programs Evaluated Kenya (2) Kilifi OVC Project, Catholic Relief Services (CRS) Operating in Kilifi District for two years Community Based HIV/AIDS Prevention, Care and Support Program (COPHIA), Pathfinder & Integrated AIDS Program (IAP) Operating in Thika District for 4 years Tanzania (3) CARE Tumaini Project, Allamano, CARE, FHI (Allamano) Operating in Iringa Region for five years Mama Mkubwa & Kids Club, Salvation Army (TSA) Operating in Mbeya Region for 2-years Jali Watoto , Pact/SAWAKA ( Jali Watoto ) – Field funded Operating in Karagwe, Kagera Region for four years

Kenya (2)

Kilifi OVC Project, Catholic Relief Services (CRS)

Operating in Kilifi District for two years

Community Based HIV/AIDS Prevention, Care and Support Program (COPHIA), Pathfinder & Integrated AIDS Program (IAP)

Operating in Thika District for 4 years

Tanzania (3)

CARE Tumaini Project, Allamano, CARE, FHI (Allamano)

Operating in Iringa Region for five years

Mama Mkubwa & Kids Club, Salvation Army (TSA)

Operating in Mbeya Region for 2-years

Jali Watoto , Pact/SAWAKA ( Jali Watoto ) – Field funded

Operating in Karagwe, Kagera Region for four years

Overview of Programs & Strategies Evaluated

Key Research Questions Impact of indirect support: How do efforts targeted at the structural systems surrounding children– household and community– affect: Children well-being Caregivers well-being Community attitudes and support of OVC & families? Impact of direct support on child outcomes What is the impact of educational, health, legal support, and other direct services on child & families?

Impact of indirect support:

How do efforts targeted at the structural systems surrounding children– household and community– affect:

Children well-being

Caregivers well-being

Community attitudes and support of OVC & families?

Impact of direct support on child outcomes

What is the impact of educational, health, legal support, and other direct services on child & families?

Methods Case studies (2006 and 2007) Site visits, interviews, program document review Provide understanding of program strategies, components, goals, and expected outcomes Document lessons learned from implementation Case Studies available http://www.cpc.unc.edu/measure/ovc Program expenditures (2006) Expense data collected and social costs estimated Quantified the costs corresponding to specific intervention (e.g. food supplementation, psychosocial service, educational support)

Case studies (2006 and 2007)

Site visits, interviews, program document review

Provide understanding of program strategies, components, goals, and expected outcomes

Document lessons learned from implementation

Case Studies available http://www.cpc.unc.edu/measure/ovc

Program expenditures (2006)

Expense data collected and social costs estimated

Quantified the costs corresponding to specific intervention (e.g. food supplementation, psychosocial service, educational support)

Methods Outcome evaluation (2007 and 2008) Post-test study design with intervention & comparison groups Exposed Vs. Non-exposed Surveyed children age 8-14 “or 7-15” & their caregivers Up to 2 children per household Four questionnaires were applied in each household: Q1: Household Questionnaires Q2: Parent/Guardian/Caregiver Questionnaire Q3: Parent/Guardian/Caregiver Regarding Child Questionnaire Q4: Child (age 8-14 “or 7-15”) Questionnaire

Outcome evaluation (2007 and 2008)

Post-test study design with intervention & comparison groups

Exposed Vs. Non-exposed

Surveyed children age 8-14 “or 7-15” & their caregivers

Up to 2 children per household

Four questionnaires were applied in each household:

Q1: Household Questionnaires

Q2: Parent/Guardian/Caregiver Questionnaire

Q3: Parent/Guardian/Caregiver Regarding Child Questionnaire

Q4: Child (age 8-14 “or 7-15”) Questionnaire

Ideal Impact Assessment R

Post-test Study Design Used Exposed Non-exposed X O 1 O 2 The Groups are Not Randomly Assigned ** Jali Watoto – Study compared intact groups of intervention versus comparison

Research Designs and Sampling

Principles Guiding Questionnaires Development Capture multiple measures for each domain Use existing standardized scales where possible (PSS, SES) Intervention Exposure questions to be specific to each program To facilitate comparison across countries and program models, same survey instruments were used except intervention modules Multi-faceted programs necessitated sufficient questions across multiple domains Multiple perspectives on child well-being ( child and caregiver ) Measures of caregiver, household & community well-being

Capture multiple measures for each domain

Use existing standardized scales where possible (PSS, SES)

Intervention Exposure questions to be specific to each program

To facilitate comparison across countries and program models, same survey instruments were used except intervention modules

Multi-faceted programs necessitated sufficient questions across multiple domains

Multiple perspectives on child well-being ( child and caregiver )

Measures of caregiver, household & community well-being

Outcomes Examined Psychosocial well-being – multiple measures – standard scales used where appropriate ( child and caregivers ) Education – enrollment & attendance ( child ) Health – self-reported health status and access to health services ( child and caregiver ) HIV-prevention – HIV-knowledge ( child ) & HIV-testing ( caregiver ) Legal protection – birth registration, alternate caregiver Community support – stigma and in-kind support ( child & caregiver )

Psychosocial well-being – multiple measures – standard scales used where appropriate ( child and caregivers )

Education – enrollment & attendance ( child )

Health – self-reported health status and access to health services ( child and caregiver )

HIV-prevention – HIV-knowledge ( child ) & HIV-testing ( caregiver )

Legal protection – birth registration, alternate caregiver

Community support – stigma and in-kind support ( child & caregiver )

Study Strengths and Limitations Strengths: Yielded immediate data on program effects Results can be used to improve current programs Ethical - not withholding services for experiment sake Limitations: Post-test design – no baseline data - impossible to make conclusions concerning change in outcomes resulting from program exposure Selection bias - self-selection to participate and those who did not -makes it difficult to conclude with certainty that the interventions are responsible for the observed differences

Strengths:

Yielded immediate data on program effects

Results can be used to improve current programs

Ethical - not withholding services for experiment sake

Limitations:

Post-test design – no baseline data - impossible to make conclusions concerning change in outcomes resulting from program exposure

Selection bias - self-selection to participate and those who did not -makes it difficult to conclude with certainty that the interventions are responsible for the observed differences

Analyses Plan Who are the OVC/MVC program beneficiaries? Effects of community level interventions i.e. Community care and support meetings/sensitization Effects of household or caregiver level interventions i.e. Community volunteer or Health Worker home visits Caregivers participation in OVC care seminars Effects of child level interventions i.e. Kids clubs Basic needs support (e.g., education, health, legal)

Who are the OVC/MVC program beneficiaries?

Effects of community level interventions i.e.

Community care and support meetings/sensitization

Effects of household or caregiver level interventions i.e.

Community volunteer or Health Worker home visits

Caregivers participation in OVC care seminars

Effects of child level interventions i.e.

Kids clubs

Basic needs support (e.g., education, health, legal)

Statistical Analyses Descriptive analysis (Univariate) Bivariate analysis (ANOVA and Chi-square) Multivariate (logistics, and linear regression) Control variable: non-program factors e.g. socio-demographic Child Level – Age, sex, orphan status, relationship to caregiver, and number of different homes the child had lived in the past year. Caregiver level – Age, sex, marital status, education, illness, SES, and # children

Descriptive analysis (Univariate)

Bivariate analysis (ANOVA and Chi-square)

Multivariate (logistics, and linear regression)

Control variable: non-program factors e.g. socio-demographic

Child Level – Age, sex, orphan status, relationship to caregiver, and number of different homes the child had lived in the past year.

Caregiver level – Age, sex, marital status, education, illness, SES, and # children

Description of OVC Program Beneficiaries Who is enrolled in OVC programs (MVC Profile):- Although, these programs targeted geographic areas most affected by HIV/AIDS, MVC were identified and assisted regardless of specific causes of vulnerability Majority of children enrolled in OVC programs are vulnerable in several fronts & not just orphanhood

Who is enrolled in OVC programs (MVC Profile):-

Although, these programs targeted geographic areas most affected by HIV/AIDS, MVC were identified and assisted regardless of specific causes of vulnerability

Majority of children enrolled in OVC programs are vulnerable in several fronts & not just orphanhood

OVC Profile Orphans (over 66% across programs) Living in food insecure HH (over 80% across programs) P oorest households (< 2assets) – over 40% Living with chronically ill primary caregiver (over 20%) Living with caregiver aged 50+ (about 23%) Lived in two or more households in past year (14%)

Orphans (over 66% across programs)

Living in food insecure HH (over 80% across programs)

P oorest households (< 2assets) – over 40%

Living with chronically ill primary caregiver (over 20%)

Living with caregiver aged 50+ (about 23%)

Lived in two or more households in past year (14%)

Thank YOU! Key Findings are presented next….

Key Findings are presented next….

MEASURE Evaluation is funded by the U.S. Agency for International Development through Cooperative Agreement GHA-A-00-08-00003-00 and is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Futures Group International, ICF Macro, John Snow, Inc., Management Sciences for Health, and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government. Visit us online at http://www.cpc.unc.edu/measure.