Presentation to the South African Monitoring and Evaluation Association Conference 2011

1. Can systematic reviews help identify what works and why? The case of microfinance in sub-Saharan Africa Presentation to 3rd Biennial SAMEA conference 8 September 2011 By Carina van Rooyen, Dr Ruth Stewart & Prof Thea de Wet

2. How do you want it – the crystal mumbo-jumbo or statistical probability?

3. Need to demonstrate impact Large development funders wants to know ‘what works’ in development Looking for evidence of effectiveness – evidence-informed development policy

5. “UK government support for aid organisations will be targeted at those agencies which demonstrate they can deliver best value for money while they improve the health, education and welfare of millions of people in the poorest countries…. We expect these charities to work hard to prove to UK taxpayers that they will and can make a real difference to the lives of the poorest and deliver real value for money.” ~ DFID 2010

7. Impact evaluations (IE)

8. IE about showing causality Causation: A change in X is related to a change in Y Not the same as correlation

9. Counterfactual crucial

10. Randomistas provide the answers? ‘Gold standard’ study design advocated by ‘randomistas’ – led by influential academics at the Abdul LatifJameelPoverty Action Lab (JPAL) Population

11. “Creating a culture in which randomised evaluations are promoted, encouraged and financed has the potential to revolutionise social policy during the 21st century, just as randomised trials revolutionised medicine during the 20th.” ~ Esther Duflo quoted in Lancet Editorial, “The World Bank is finally embracing science” (2004)

12. RCTs in development

13. But are RCTs sufficient? Methodological debates about RCTs raise number of concerns within the development community, including Dismissal of other evaluation techniques: hierarchies and ‘gold standard’ Lack of consideration of contextual information: over-simplificationwith generalisable information stripping out contextual details Narrow focus on linear causal relationships: experimental designs over-simplify complex issues

14. Hierarchy of evidence Randomised control double-blind trials Randomised control trials

16. RCTs questioned in development Narrow approach to evidence Trials are costly, have ethical dilemmas & are often lacking Solutions are urgently required Heterogeneity raises serious concerns about external validity of such trials

17. Systematic reviews (SRs) to the rescue? Can these concerns about RCTs be overcome through the use of SRs? Led by the Cochrane Collaboration, SRs routinely used in health care to combine results of RCTs Integrated into health policy internationally In development promoted by funders

18. SRs in the development field About 100 SRs in international development commissioned so far ~ Howard White (chair of IDCG) First SRs in development published: water and environmental sanitation (Waddington & Snilstveit 2009), HIV behaviour change (Noar et al 2009), microfinance (Stewart et al 2010)

19. SRs in the development field (cont.) Four registered SRs with IDCG (Campbell Collaboration) cash transfers for health & nutritional outcomes in poor families deworming for improving school attendance in school-aged children impact of farmer field schools effectiveness & sustainability of water, sanitation hygiene interventions in combating child diarrhoea IDCG expects to register more titles later in 2011 in CCTs in education, governance and anti-corruption, urban development, social protection & microfinance

20. What is a systematic review? Is about the evidence of effectiveness Thorough & systematic collection of all relevant evidence & its quality appraisal and synthesis Typically combine evidence from RCTs Designed to minimisebiases & errors inherent to traditional, narrative reviews

22. Analyse dataInterpret results & write a report, which is peer reviewed Comprehensive strategy to search for relevant studies (unpublished & published) Explicit & justified criteria for inclusion or exclusion of any study Statistical synthesis of data (meta-analysis) if appropriate and possible, or qualitative synthesis

23. Asking the right question? Paper or plastic?

24. Rigidity of SRs: Hierarchy of evidence?

25. Do you ever think sometimes, you might be overdoing the whole moisturiser thing?

26. Risks with methodological rigid SRs? Narrow menu of methodological options could mean reduction of development to simple interventions, in order to facilitate its measurement (Guijt et al 2011:4) “Those development programs that are most precisely and easily measured are the least transformational, and those programs that are most transformational are the least measurable.” (Natsios, ex USAID quoted in Guijt et al 2011:3)

27. Similar critiques against methodological rigid SRs in development that against RCTs in development

28. Our SR on the impact of microfinance on the poor in SSA

29. Our pragmatic approach Followed pragmatic approach for our SR in five important ways: Focused on REGIONAL rather than worldwide evidence

30. Our pragmatic approach (cont.) Multi-disciplinary nature of our team Using range of sources: not only electronic data bases (publication bias)

31. Our pragmatic approach (cont.) Methodological: Drew on well-conducted evaluations with comparative research design, including RCTs, but also non-randomised trials, quasi-experimental designs, and simple with-and-without studies For purists this ‘weakened’ confidence in evidence of impact ‘rigour’ narrowly defined in terms of statistically significant indication of difference with and without an intervention – internal validity (Guijt et al 2011:7)

32. Our pragmatic approach (cont.) We argue for ‘good enough quality’: rigour include aspects such as utility, external validity, method mastery, sense-making & substantiated methodological choice (Guijt et al 2011:7) In practice we also broadened the scope of our study Able to look at additional types of interventions & outcomes which haven’t yet been evaluated by RCTs Draw on evidence from additional countries

33. Details of 15 included studies 4 RCTs 2 quasi-experimental studies 9 with/without studies 11 = microcredit, 2 = savings, 2 = combined credit & savings Ethiopia, Ghana, Kenya, Malawi, Madagascar, Rwanda, South Africa, Tanzania (Zanzibar), Uganda & Zimbabwe Rural & urban initiatives

34. Is ‘good enough quality’ good enough?

35. Is ‘good enough quality’ good enough? (cont.)

36. Is ‘good enough quality’ good enough? (cont.)

37. “The quality of evidence about effectiveness should be judged not by whether it has used a particular methodology, but whether it has systematically checked internal and external validity, including paying attention to differential effects.” (Rogers 2010:195)

38. If methodological purist (exclude any study with indication of bias) possible conclusion that evidence not good, e.g. Duvendack et al’s SR on impact of microcredit worldwide

39. Clemens & Demombynes (2010:1) refer to luxury versus necessity White (2011a) refers to choice between technical quality & policy influence Risk of purist is that have nothing to say to policy makers as want definitive free-from-bias answer Risk of pragmatist is that, while providing policy makers with ‘better’ information than what otherwise would have, might have bias

40. Our pragmatic approach (cont.) Development of causal pathway in which we explored how microfinance works to be able to draw conclusions about why microfinance does or does not work & for whom What achieved (outcome) & how (process) Conventional SRs limited to evidence of effectiveness, but this more enhanced approach allowed informed conclusions to be drawn Evaluative ‘proving’ & improving

42. Causal pathway analysis 41

43. Use other MFI Social cohesion What we now think is happening Use same MFI Micro-credit Micro-savings Women’s empowerment Able to repay loan and avoid increase in debt Able to save Given to individuals or groups Default on loan, lose collateral and/or forced to borrow more Long-term benefits Spend money differently 1. Invest in immediate future: a. Business b. Productive assets c. Adult education d. Workers’ health & nutrition 2. Consumptive spending with scope for productivity: a. Add on housing b. Assets which retain value 3. Invest in long- term future: a. Children’s education b. Children’s health and nutrition 4. Consumptive spending (non-productive): Assets which do not retain value Actual increased income Actual decreased income Determined by external factors: Entrepreneurial ability Appropriateness of business in context Competition from other MFI clients Gender and power relations Scope for increased income via business or employment Improved capabilities Better able to deal with shocks FOR CREDIT CLIENTS ONLY Inability to repay loan

44. Some of our recommendations More and better impact evaluations of microfinance (especially savings) On-going discussion of how to deliver pragmatic systematic reviews for international development

45. Next steps SR methodology to be further enhanced to serve the needs of development Incorporating studies of poor people’s experiences, priorities & views (constructivist view): something similar has been done in health promotion, e.g. EPPI healthy eating review Combining reviews of published evidence with primary research, e.g. Thuthuka project Systemic approach to M&E and impact evaluations

46. Three challenges for M&E Consider findings of SRs to enhance individual programme evaluations, establishing what best available evidence shows & placing evaluation of individual projects within context of this broader evidence base Consider RCT designs as one part of solution to impact evaluation, and explore where evaluations which you are able to conduct can fit within broader evidence base to shed light on key issues in development Conduct pragmatic SRs to inform decision-making in development – flexibility

47. The latest research shows that we really should do something with all this research

48. Conclusion About what works for whom under what circumstances and how SRs help to think about strategic issues, rather than specific project intervention There are limitations with SRs & they are very reliant on existence & clear reporting of individual evaluations SR is only as good as the included studies (garbage in, garbage out)

49. Conclusion (cont.) But They are bigger than individual studies They take into account relevance, rigour & vigour With causal pathway analysis (theory of change), they go some way to translating research evidence into meaningful policy & practice insights

50. So, can SRs help identify what works and why? Based on our SR on the impact of microfinance on the poor in sub-Saharan Africa, yes But have to be pragmatic / flexible in approach to SR in the field of development

51. Source of cartoon: Guijt et al 2011:i

52. Source: http://www.how-matters.org/2011/05/24/rcts-band-aid-on-deeper-issue/ Not everything that can be counted counts, and not everything that counts can be counted ~ Albert Einstein

53. Thank you for listening cvanrooyen@uj.ac.za Presentation online at http://www.slideshare.net/carinavr

54. References / Acknowledgements Blattman C 2008 Impact evaluation 2.0. Presentation to the Department for International Development (DFID) London on 14 February 2008. Available at http://www.chrisblattman.com/documents/policy/2008.ImpactEvaluation2.DFID_talk.pdf Blattman C 2011 Impact evaluation 3.0? 5 lessons and reflections after a couple of more years of failure and success. Presentation to DFID on 1 September 2011. Available at http://www.chrisblattman.com/documents/policy/2011.ImpactEvaluation3.DFID_talk.pdf Cummings S 2010 Evaluation revisited 1: Rigorous versus vigorous. Blog posting at http://thegiraffe.wordpress.com/2010/06/17/evaluation-revisited-i-rigorous-vs-vigorous/ on 17 June 2010 Deaton A 2010 Instruments, randomisation and learning about development. Journal of Economic Literature 48: 424–455 Gertler PJ, Martinez S, Premand P, Rawlings LB & Vermeersch CMJ 2010 Impact evaluation in practice: Ancillary material. World Bank: Washington DC (www.worldbank.org/ieinpractice) Guijt I, Brouwers J, Kusters C, Prins E & Zeynaloba B 2011 Evaluation revisited: Improving the quality of evaluative practice by embracing complexity (conference report) Hughes K & Hurchings C 2011 Can we obtain the required rigour without randomisation? Oxfam GB’s non-experimental Global Performance Framework (3ie Working Paper 13). New Delhi: 3ie Rogers P 2010 Learning from the evidence about evidence-based policy. In Banks G (eds) Strengthening evidence-based policy in the Australian Federation. Melbourne VIC: Productivity Commission: 195-214 Photos and cartoons not acknowlegedon slide were found via Google Images