A Toolkit for Evaluating the Impact of HIV/AIDS Programming on Children in Africa


Published on

Presented by Jenifer Chapman at the October 2013 American Evaluation Association Annual Conference.

Published in: Health & Medicine
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • {"27":"I wouldn’t read through these if only 15 mins\n","28":"I wouldn’t read through these if only 15 mins\n","7":"PURPOSE OF SLIDE: To show a diagram of the social-ecological model emphasizing the different levels that OVC programming can impact to improve child and family wellbeing.\nNOTES:\nThe PEPFAR approach to children in the epidemic is based on a social-ecological model that considers the child, family, community and country contexts and recognizes the unique yet interdependent contributions of actors at all levels of society to the well-being of children affected by HIV/AIDS. \n \nThis version is an AIDS-sensitive version that considers the ultimate goal at each level of society in achieving an AIDS free generation.\n \nIt also seeks to emphasize that Families are the first line of support and defense for children. And that Even in the most resource-deprived settings, families and communities have critically important strengths. This should be recognized by programs and built upon to achieve the best outcomes for vulnerable children affected by AIDS. \n","2":"PEPFAR, originally introduced in 2003 by then-President George W. Bush, remains the largest commitment in history by any nation to combat a single disease. In 2008, PEPFAR was reauthorized for  $48 billion over five years (2009 to 2013), with the goals of preventing 12 million new infections; treating 3 million people living with AIDS and caring for 12 million people, including 5 million orphans and vulnerable children.     The Reauthorization Act seeks to transition the U.S. response from an emergency approach to a focus on sustainability, including health system strengthening and partnership building.  The legislation allows the program to serve as a platform for expanded responses to a broader range of global health needs.  The Act also supports the availability and accessibility of female condoms.  \n \n","30":"I could elaborate on any of these if time allows\n","8":"PEPFAR and OVC\n-10% earmark for children made vulnerable by HIV and AIDS\n-falls under Care and Support \n--focuses on the social and emotional consequences and important responses to epidemic\n--updated our guidance in 2012: based in ecological model and care and protection for children emphasizing\nStrengthen families as primary caregivers of children;\nSupport the capacity of communities to create protective and caring environments;\nBuild the capacity of social service systems to protect the most vulnerable; and\nAllocate resources for children according to need in the context of HIV/AIDS by integration with the broader PEPFAR platform and response\n","3":"In the first phase of PEPFAR, activities were more of an emergency nature with the goal of using available interventions to reduce mortality and alleviate suffering from HIV disease as quickly and effectively as possible. \nMany lessons have been learned through examination of programs, including simple evaluations and operations research. \n \nAs would be expected in an emergency context state-of-the-art monitoring, evaluation, and research methodologies were not fully integrated or systematically performed.\nIn its second phase of PEPFAR-, emphasis on sustainability, programs must demonstrate value and impact to be prioritized within complex and resource-constrained environments. \nNow there is a greater demand to causally attribute outcomes to programs. Better attribution can be used to inform midcourse corrections in the scale-up of new interventions or to re- evaluate investments in programs for which impact is less clear.\n","9":"Change model\n","26":"I wouldn’t read through these if only 15 mins\n"}
  • A Toolkit for Evaluating the Impact of HIV/AIDS Programming on Children in Africa

    1. 1. A Toolkit for Evaluating the Impact of HIV/AIDS Programming on Children in Africa Jenifer Chapman, PhD MEASURE Evaluation/Futures Group
    2. 2. Stepping back for a minute  HIV/AIDS Programming = President’s Plan for AIDS RELIEF (PEFPAR)  Interagency effort  Introduced by President Bush in 2003  Largest Commitment in history by any nation to combat a single disease  Reauthorized in 2008 for $48 billion
    3. 3. PEPFAR I and II  First phase of PEPFAR (2003-2008) focus on emergency response—program reviews, simple evaluations, operations research  Second phase of PEPFAR (2009-2013) focus on sustainability, better attribution, program outcomes and impact
    4. 4. PEPFAR Mission & Goal (OVC)  Mission: To mitigate the social, emotional and economic impacts of HIV/AIDS on children and to reduce their risk and vulnerability while increasing their resilience  A goal: To care for 5 million orphans and vulnerable children  10% funding earmark for OVC
    5. 5. What is an “OVC”?  A child infected or affected by HIV  And what does that really mean?  PEPFAR 1 vs. PEPFAR 2 And how do we “care” for them?
    6. 6. Guidance!  6+1 areas of programming  Health, Education, PSS, Protection, Food & Nutrition, Shelter & Care, HES
    7. 7. PEPFAR’s approach to OVC programming
    8. 8. OVC Programs, under PEPFAR, aim to:  Improve the wellbeing of children and families  Strengthen families as primary caregivers of children  Support the capacity of communities to create protective, caring environments  Build the capacity of social service systems to protect the most vulnerable
    9. 9. Implementation  Community-based programs  Focus: case management (social work)  Home-visiting by community based volunteers (beneficiaries are targeted)  Community-wide interventions (HES, Kids Clubs)  6+1 domains of programming (health, PSS, etc.)  Referral networks
    10. 10. Evaluating PEPFAR’s contribution  Reviews of the OVC portfolio found that despite great investment, we know little about impact and “what works”
    11. 11. Not so long ago…  Few evaluations conducted at program level  Outcomes monitoring at case management level (good, but led to problems in evaluation practice)  Studies carried out were challenged:  Tool misuse, low data quality, inadequate measurement and poor choice of constructs, no comparability of measures, unethical practices  Poor availability of information for decision making The State of Evaluation Practice was:
    12. 12. A proposed solution Standardized questionnaires for use in a survey of children ages 0-17 years and their adult caregivers
    13. 13. The purpose  Standardize population-level child and caregiver wellbeing data beyond what is available from routine surveys  Produce actionable data to inform programs and enable mid-course corrections  Enable comparative assessments of child and caregiver well-being and household economic status across a diverse set of interventions and regions
    14. 14. But what do we measure?  Developing country context
    15. 15. Focus on PEPFAR OVC programs  Indicators need to reflect & be amenable to change by PEPFAR program intervention  HH interventions led by home visitors  Community interventions  Low direct funding per target, focus on linkages  Often inadequate services in vicinity
    16. 16. Who are these tools for?  Local and international research institutions and other implementing organizations with evaluation agenda  USAID Forward – supporting local researchers
    17. 17. So, where to start…  Two step, participatory process:  Build consensus around core impact indicators for PEPFAR-funded OVC programs  Develop OVC program evaluation (survey) tools
    18. 18. Distilling the core indicators  Our starting point: 6+1 domains of PEPFAR OVC programming  Step 1: Extensive literature search  Step 2: Gaps (HES, PSS) filled through targeted research  Result: >600 child/HH wellbeing questions/indicators  Step 3: Analysis against 8 criteria  Result: shorter list of questions for discussion 19
    19. 19. Inclusion criteria 1. Measures impact/outcomes 2. Amenable to change from program interventions 3. Relevant across a wide range of interventions 4. 5. 6. 7. 8. 20 Contributes to a holistic vision of child wellbeing Verifiable through another source Easy to implement Relevant across different regions / countries Relevant or easily adapted across age and sex
    20. 20. Finalizing the core indicators  External working group: solicited review from 49 stakeholders  Finalized core set of 12 child and 3 household measures 21
    21. 21. Developing the tools  Tools drafted with strong stakeholder input  Draft tools piloted in Zambia and Nigeria  Cognitive interviews to test key concepts (e.g. social support)  Household pre-test of full tools, procedures
    22. 22. Structure and content 1. Caregiver questionnaire (including questions on household) 2. Child questionnaire (ages 0-9 years), administered to caregiver 3. Child questionnaire (ages 10-17), administered to child with parental consent & child assent
    23. 23. Caregiver questionnaire Sections Core questions Section 1: Household schedule • • Household schedule* (10) Changes in household composition (4) Section 2: Background Information on Caregiver and Household • • • • Demographic information* (7) Work* (3) Access to money (3) Shelter (1) • Section 3: Food Security • Household food security (6) • Section 4: Caregiver Wellbeing and Attitudes • • • General health (2) Caregiver support (4) Parental self-efficacy (1) • Basic HIV/AIDS knowledge* (7) HIV testing* (3) Attitudes to condom educ (1) • Section 5: HIV/AIDS Testing, • Knowledge, Attitudes • • Section 6: Access to HIV Prevention, Care & Support *DHS, bold=core indicator • Household access to services (1) Optional modules • • Household Economic Status (forthcoming) Progress out of Poverty Index or similar (country specific) Dietary Diversity (1) Perceptions and experience of child discipline, violent discipline (forthcoming) Gender roles and decisionmaking power* (9) HIV/AIDS attitudes* (4)
    24. 24. Child questionnaire (ages 0-9) Sections Core questions Optional modules Section 1: Child Health and Welfare • • • • • • • • • Confirm demographics (5) General health & disability (4) Birth certificate (2) Vaccinations (11) Fever (<5 years)* (1) Diarrhea (<5 years)* (1) Experience of neglect (2) Slept under mosquito net* (1) HIV testing experience* (2) • • • Section 2: Education and Work • Section 3: Food Consumption Section 4: Access to HIV Prevention, Care & Support • School attendance*, progression/repeats, drop-outs, missed school days (5+ years) (9) Work for wages (2) Early childhood stimulation (2) Food consumption (2+ years) (8) • Child access to services (1) Section 5: Anthropometric Measures (of Children) • Weight*, Height*, MUAC • • *DHS, bold=core indicator • Fever: extended* (4) Diarrhea: extended* (3) Health for children living with HIV/AIDS (forthcoming) Dietary diversity (1)
    25. 25. Child questionnaire (ages 10-17) Sections Core questions Section 1: Background Information on Child Section 2: Diary Section 3: Education • • • • Section 4: Chores & Work • • • • • • • Confirm demographics* (5) Identity of caregiver (1) Daily log (6) School attendance*, progression/repeats, dropouts (9) Chores (3) Work (7) Food consumption (8) Alcohol consumption (3) Birth certificate (2) General health & disability (3) General support (4) • • Basic HIV/AIDS knowledge* (7) • HIV testing * (3) • Section 8: Sexual Experience Section 9: Access to HIV • Prevention, Care & Support Section 10: Anthropometric • Measures: Weight and Height • Section 5: Food & Alcohol Consumption Section 6: Health, Support & Protection Section 7: HIV Testing, Knowledge, and Attitudes Child access to services (1) Weight, Height, MUAC Optional modules • Dietary diversity (1) • Health for children living with HIV/AIDS (forthcoming) Perceptions/experience of violence (forthcoming) Child development knowledge (6) HIV/AIDS attitudes and beliefs (4) • Sexual behavior (13-17 yrs) (5)
    26. 26. Lessons from piloting  Access to money  Social support (and PSS generally)  The importance of validating translations and pre-testing  Addressing child headed households
    27. 27. You said a toolkit?  Tools & Manual  Template protocol with consent/assent forms  Methodological guidance  Data analysis guide  Data collector training manual and materials
    28. 28. Now….  Evaluation and data use are high on the agenda  We have standardized tools/measures and guidance for evaluating OVC programs/interventions with global buy-in  Tools misuse is being corrected  More information available for decision making The State of Evaluation Practice is:
    29. 29. Where can I find out more? Go to our website: http://www.cpc.unc.edu/measure/ our-work/ovc Keep in touch on ChildStatusNet: http://childstatus.net/ Email: Jenifer Chapman: jchapman@futuresgroup.com Janet Shriberg: jshriberg@usaid.gov
    30. 30. The research presented here has been supported by the President’s Emergency Plan for AIDS Relief (PEPFAR) through the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement GHA-A-00-08-0000300. Views expressed are not necessarily those of PEPFAR, USAID or the United States government. MEASURE Evaluation is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill in partnership with Futures Group, ICF International, John Snow, Inc., Management Sciences for Health, and Tulane University.
    31. 31. Questions?