Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Presentation by Rachel Steinacher, on IPA and RCTs

909 views

Published on

A presentation by Rachel Steinacher, Research Manager for IPA-Kenya (Innovations for Poverty Action), on IPA and RCTsThis was presented at the Commission for Science and Technology (COSTECH) in Dar es Salaam, Tanzania, on June 19, 2014, to an audience of researchers.

  • Be the first to comment

Presentation by Rachel Steinacher, on IPA and RCTs

  1. 1. Innovations for Poverty Action – Generating and Using Evidence
  2. 2. Outline  Introduction to Innovations for Poverty Action  What we do and why  Different Types of Evidence  Process evaluation  Qualitative data  Experimental evidence  Why research impact  Determining and defining impact  What can we use the results of RCTs for?  Example: Primary School Deworming  Conclusion  Opportunities at Innovations for Poverty Action
  3. 3. Innovations for Poverty Action  Founded in 2002, we partner with academics to design and evaluate potential solutions to global poverty problems using randomized evaluations.  We also work to mobilize and support these decisionmakers to use this evidence / these solutions to build better programs and policies at scale.  In collaboration with over 250 leading academics and implementing organizations, we have evidence from over 175 completed studies with over 225 in progress around the
  4. 4. Innovations for Poverty Action  Our Vision  More Evidence, Less Poverty  Our Mission  To discover and promote effective solutions to poverty problems around the world
  5. 5. Different Types of Evidence  We’ve been referring to ‘evidence’ but what we mean might not be clear as there are many different types of evidence  Types of evidence include (but are not limited to!)….  Process Evaluation Data (Monitoring and Evaluation)  Qualitative Data  Experimental Evidence
  6. 6. Process Evaluation  Process evaluation analyzes the extent to which program operations, implementation, and service delivery are being implemented properly.  When process evaluation is ongoing it is called program monitoring (as in Monitoring and Evaluation: M&E).  Process evaluations help us determine, for example:  Whether services and goals are properly aligned  Whether services are delivered as intended to the appropriate recipients  How well service delivery is organized  The effectiveness of program management  How efficiently program resources are used  Process evaluations are often used by managers as benchmarks to measure success, for example: the distribution of chlorine tablets is reaching 80% of the intended beneficiaries each week.  These benchmarks may be set by program managers and sometimes by donors.
  7. 7. Qualitative Data  Qualitative data is data that is captured in a non-numerical way.  Collection can happen in many different ways, including:  In-depth interviews  One-on-one interviews with a participant with the purpose of probing the thoughts, ideas, knowledge, etc. of the person being interviewed. These will often be recorded and transcribed for analysis later.  Focus groups  Same as an in-depth interview, but includes multiple participants  Direct observation (ethnography, etc.)  This differs from interviewing as the person collecting the data does not actively try to ask questions to the participant about the topic being studied. They simply watch and observe, with the goal of recording and interpreting what they say, their behaviour and attitudes. Sometimes this involves recording (video and/or audio) or writing notes about what they observe for analysis later.  Etc…
  8. 8. Qualitative Data  Qualitative data is useful to answer questions like:  Does this intervention ‘make sense’ in this context?  Is this intervention socially, politically, religiously acceptable?  How do people perceive / understand this intervention and the problem(s) it is meant to solve?  Hypothesis generation: what should we be studying / collecting data on to better understand the intervention or topic we’re interested in studying? Are we asking the right questions?
  9. 9. Experimental Evidence – Why Research Impact?  Experimental evidence allows us to determine impact. By impact, we mean changes / outcomes that have happened because the intervention happened.  This let’s us know what interventions…..  Are effective [How effective is it? In what context? For what demographic(s)? At what cost?]  Have no impact (Why wasn’t it effective? Was the program implemented properly?)  Are harmful (Even something that seems like common sense can have devastating consequences; see the ‘Campbell-Somerville Study’)
  10. 10. Why Research Impact – Influencing Programs and Policy  We can use this information to try to influence the decisions being made by governments, NGOs, and other policymakers / program implementers to make evidence-based decisions
  11. 11. Determining Impact  Different research methods yield different types of evidence….  Some of this evidence demonstrates the impact of a program  Some evidence cannot be used to demonstrate impact. in fact, it can lead us to believing things that aren’t true, even though the data seems to support a certain conclusion….  Correlation does not imply causation
  12. 12. Evidence that Pirates Prevent Global Warming
  13. 13. Determining Impact  Impact is captured by analyzing a comparison between: 1. the outcome some time after the program has been introduced 2. the outcome at that same point in time had the program not been introduced (the ”counterfactual”)
  14. 14. Impact: What is it? Time PrimaryOutcome Impact Intervention
  15. 15. Impact: What is it? Time PrimaryOutcome Impact Intervention
  16. 16. Constructing the Counterfactual  Counterfactual is often constructed by selecting a group not affected by the program  Randomized:  Use random assignment of the program to create a control group which mimics the counterfactual.  Non-randomized:  Argue that a certain excluded group mimics the counterfactual.
  17. 17. Counterfactual  The counterfactual represents the state of the world that program participants would have experienced in the absence of the program (i.e. had they not participated in the program)  Problem: Counterfactual cannot be observed  Solution: We need to “mimic” or construct the counterfactual
  18. 18. Non-Random Assignment HQMonthly income, per capita 1000 500 0 Treatment Control 1457 947
  19. 19. Demonstrating Impact through RCTs  RCTs are by far the most powerful research methodology by allowing us to determine the impact of a program by minimizing the likelihood that the treatment and control groups are meaningfully different from each other
  20. 20. Random Assignment Monthly income, per capita 1000 500 0 Treatment Control 1257 1242 HQ
  21. 21.  An example from IPA… What can we use RCT results for?
  22. 22. Primary School Deworming  One of Innovations for Poverty Action’s most famous studies, that is currently be scaled out across the Kenya
  23. 23. Primary School Deworming  Principal Investigators: Michael Kremer and Edward Miguel  Partners:  Investing in Children and their Societies (ICS)  Location: Western Kenya  Sample: 30,000+ primary school children, 6- 18 years old  Timeline: 1997-2001
  24. 24. Primary School Deworming  Policy Issue  Intestinal worms infect more than one in four people worldwide and are particularly prevalent among school-aged children in developing countries.  These intestinal worms are believed to have a negative impact on education, hindering child development as well as school attendance and reducing income later in life.
  25. 25. Primary School Deworming  Context of the Evaluation  Busia district is a poor and densely-settled farming region in western Kenya adjacent to Lake Victoria. It has some of the country’s highest worm infection rates.  One quarter of Kenyan student absenteeism is attributed to abdominal pains which likely due to intestinal worm infections. In addition, older children may miss school to take care of siblings who are sick with worm infections
  26. 26. Primary School Deworming  Details of the Intervention  This study evaluated the Primary School Deworming Project (PSDP), which was carried out by International Child Support in cooperation with the Busia District Ministry of Health.  The program randomly divided 75 schools into three equal groups which were phased into treatment over three years.  Within each group, a baseline parasitological survey was administered to a random sample of pupils. Schools with worm prevalence over 50% were mass treated with deworming drugs every six months.
  27. 27. Primary School Deworming  Details of the Intervention  In addition to medicine, treatment schools received regular public health lectures, wall charts on worm prevention, and training for one designated teacher.  The lectures and teacher training provided information on worm prevention behaviors—including washing hands before meals, wearing shoes and not swimming in fresh water.
  28. 28. Primary School Deworming  Results and Policy Lessons: Impact on Infection Intensity  Deworming reduced serious worm infections by half amongst children in the treatment groups.  Pupils that received treatment reported being sick significantly less often, had lower rates of severe anemia, and showed substantial height gains, averaging 0.5 centimeters.
  29. 29. Primary School Deworming  Results and Policy Lessons: Impact on School Attendance  Deworming increased school participation by at least 7 percentage points, which equates to a one-quarter reduction in school absenteeism.  When younger children were dewormed, they attended school 15 more days per year, while older children attended approximately 10 more school days per year.
  30. 30. Primary School Deworming  Results and Policy Lessons: Treatment Spillover  The entire community and those living up to 6 kilometers away from treatment schools benefited from “spillovers” of the deworming treatment.  Spillover effects occur because medical treatment reduces the transmission of infections to other community members.  Reductions in infection in non-treated children resulted in an additional 3 to 4 days of schooling per year.
  31. 31. Primary School Deworming  Results and Policy Lessons  Including the spillover benefits of treatment, the cost per additional year of school participation is US$3.27, considerably less than the cost of many alternative methods of increasing primary school participation.  Creation of the ‘National School-based Deworming Program’ and ‘Deworm the World’
  32. 32. National School-Based Deworming Programme National School Health Programme Ministry of Education Science and Technology Ministry of Health
  33. 33. National School-Based Deworming Programme GOAL: Treat 5 million children each year for 5 years TARGET: Treat ALL at risk children aged 2-14
  34. 34. Primary School Deworming  Scaling-up the Program  Additionally, in 2007 the ‘Deworm the World Initiative (DtW)’ was founded. It was managed and operated through Innovations for Poverty Action up until the end of 2013, when it our sister organization, ‘Evidence Action’ spun off with the goal of scaling evidence-based programs – including DtW  To date, DtW’s programs have reached 37 million children in 27 countries  Impact evaluation can and has had a profound impact on the world
  35. 35. In Sum  There are many types of evidence that one can collect and use to better understand an intervention or topic  Experimental evidence generated by RCTs is a powerful tool that we can use to promote evidence-based development work by influencing policymaking and program design.
  36. 36. Innovations for Poverty Action Moving Forward  Innovations for Poverty Action is a rapidly growing organisation with plans to expand in Tanzania (we currently have 1 project here)  We are always looking for:  Talented, innovative researchers to partner with on new research studies  For young researchers, this typically means partnering with a senior researcher to allow for mentoring and expert oversight
  37. 37. Innovations for Poverty Action Moving Forward  We are also looking for bright, eager people interested in working in working on our research projects as:  Project Associates (entry-level managers with a Bachelors degree + 0-2 years experience)  Project Managers (early to mid-career professionals with a Bachelors degree + 2-5 years experience)  Research Managers (early to mid-career managers with a Masters degree + 3-5 years of experience)  For more information, visit:  http://www.poverty-action.org/getinvolved/jobs

×