Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

IDS Impact, Innovation and Learning Workshop March 2013: Day 2, Paper Session 3 Richard Palmer Jones


Published on

Published in: Technology, Economy & Finance
  • Be the first to comment

  • Be the first to like this

IDS Impact, Innovation and Learning Workshop March 2013: Day 2, Paper Session 3 Richard Palmer Jones

  1. 1. Impact Evaluation, Replicationand EthicsRichard Palmer-JonesSchool of International DevelopmentUniversity of East AngliaPresented atImpact, Learning and Innovation: Towards a Research and Practice Agenda forthe FutureInstitute of Development Studies, Brighton (UK), Convening SpaceMarch 26-27, 2013
  2. 2. Modern impact evaluation• Addresses the attribution issue by:– Sophisticated econometric analyses of observational data– Randomised control trials– (and mixed methods, theory based, process tracing, andevaluation, agent based modelling)– phronesis• Methods which assert their status as science by– Mathematisation, quantification, neutrality and objectivity,and so on• But lack a crucial component of science– Replication• Repeatability, checking, internal, external, construct .. validity
  3. 3. Identification• Problem of unobserved confounding variables– Are benefits of microfinance due to loans from the MFI?• Placement and selection biases– More favourable areas– People who are more likely to benefit• Identification by:– Randomisation• Internal and external validity– In the absence of randomisation, or with compromised• multiple regression• Natural or quasi experiments– Single and double difference estimation• Instrumental variables estimation• Propensity Score Matching• Regression discontinuity• Panel data estimation
  4. 4. How robust are these methods?• Methods require assumptions– Many assumptions cannot be tested• Randomisation– Common threats to validity• Imperfect selection of subjects• Imperfect randomisation – subject agency• Imperfect adherence to treatment• Lack of blinding– Hawthorn & John Henry effects• Lack of external validity• Econometric results– Data mining, result polishing, researcher, sponsor, andeditor allegiance & reluctance, and HARKing
  5. 5. Replication in randomised studies• Replication is the sine qua non of science• Many RCTs do not yield the same results– Many of the problems listed above• Gave rise to systematic review and meta-analysis– Systematic review seldom resolves issues for all concernedeven when large number of good quality trials– Meta-analysis can make it appear that lots of weak resultscombine to produce a convincing (statistically andsubstantively meaningful one)• But missing studies– Publication bias – “Bad Pharma”– Researcher and institutional allegiance and reluctance» to publish results that are not from the right hymn sheet• Register of all studies in advance ….
  6. 6. Replication in observational studies• Pure replication– Checking the (original) data and code produce thereported results• Data and coding errors• Statistical replication– Is the study robust to plausible changes in datacleaning, variable construction, alternative equivalentdata• Scientific replication– Is the study robust to alternative accounts
  7. 7. Experiences from Replication World…In practice ….• Little replication in practice– More replication than realised?• Show me! [some key cases – but more generally?]– Low incentives for replication• Difficulties– Difficult access to data & code– Repeating the analysis is very taxing – detective work in the face ofincomplete documentation– [Publication bias & file drawer problem]– Deterrence• Imputation of Adverse motives– adversarial intentions» Political» Career advancement» Lack of originality• Belligerent refutation by original authors
  8. 8. Examples• Feldstein / Leimer & Lesnoy– Admit and contest with new results• Levitt / Lott– Law suite - dismissed• Hoxby and Rothstein– Beligerent contestation• Accusation of political motives and misreporting– Delayed publication (2004 –> 2007)• Acemoglu et al. and Albouy– Beligerent contestation– Severely delayed publication (2006 -> 2012)
  9. 9. Experiences from Replication World…in development practice ….• Randomisation studies (briefly)– Karlan and Zinman – randomly relax credit constraints• Observational studies– Boyce and Ravallion, 1991 -Declining real wages of agriculturallabourers in Bangladesh– Basu, Narayan, and Ravallion, 2002 - Benefits to illiterates ofbeing proximate to female literates– Pitt and Khandker, 1998 - The benefits of Microfinanceespecially when loaned to women …– Jensen and Oster, 2009 -The power of TV on the status ofwomen in India– Banerjee and Iyer, 2005 - The lasting adverse effects of colonialland revenue polices in India– [Macro-economics] Aid and Growth• Dollar & Burnside -> Mekasah and Tarp vs Doucouliagos & Paldam,JDS, forthcoming, 2013• Trade and aid -> . Perraton, 2011, Journal of Economic Methodology
  10. 10. Access tomicrofinancereduces creditconstraintsenables orincreases fixed& workingcapital, self-employmentIncreasesbusiness profitsIncreasesborrowing, orreduces costsof borrowingWageemployment,production,turnover, salesIncreasesincome and, orconsumption education and or healthexpenditure, child health andnutritional status , subjectivewell-beingChangesexpenditurepatternsWomenempowermentBusinesslosses+Failure to keepup repaymentsBorrowing fromother MFIs orinformalsources+Reducesincome and, orconsumption-Women dis-empowerment-inputseffectsimpactsfailures
  11. 11. 11• Pitt & Khandker’s quasi-experimental designSource: Armendáriz de Aghion and Morduch, 2005.“Treatment” Village “Control” VillageEligible nonparticipantsNot-eligible nonparticipantsWould beeligibleWould notbe eligibleEligibleparticipants0.5acrescultivablelandowned00.5Compareeligibleparticipantswith eligiblecontrols intreatmentvillagesCompareeligibleparticipantswith eligiblecontrols intreatment andcontrol villagesnon-eligibleparticipantsBut 20% ofparticipantshave morethan 0.5acresControlplacementbias withvillagefixedeffects
  12. 12. Results• WES-LIML-FE (Roodman & Morduch, 2011)– Modest impacts when borrowing by women– Disappear when outliers removed• Propensity Score Matching– Chemin, 2008• More modest effects, some negative• Does not distinguish by gender– (Duvendack and Palmer-Jones, 2012)• Modest, zero and negative impacts when borrowing bywomen• Impacts highly vulnerable to “hidden bias”– Hidden bias highly likely
  13. 13. Conclusions• Leamer, 1983– Let’s take the con out of econometrics• Ioannidis, 2005– Most published research findings are false• Manski, 2011– Use of econometric results in policy requires“incredible belief”• The devil is in the detail
  14. 14. Conclusions (cont)• Power and interests speaks to truth?• The development industry ….– Status and power of applied econometrics• Cognitive biases in (evaluation) research (belief)– see patterns where there are none;– see causal relations when there are none;– overvalue confirmation;– evaluate more favourably evidence that conforms with our prior beliefs– seek out confirmation;• Professional interests and the avoidance of cognitivedissonance– The disciplinary doxa– States of denial (with apologies to Stanley Cohen)– Economists’ ethics?
  15. 15. Country donors,e.g. ODA, USAIDThe ‘aid industry’ version 1Multilaterals,e.g. World bank, EUGovernmentagenciesInternationalNGOs, e.g OxfamNational NGOsLocal NGOsProjects/activitiesPEOPLE(from Gardner and Lewis, 1996)
  16. 16. The development industry (version 2)Multilateral aid agenciesUN …, WHO, FAO, …Bretton Woods (IMF, WB)Bilateral aid agenciesDFID, SIDA, NORAD,USAID, JICA, CIDA, ..NationalgovernmentsInternational NGOs(INGOs)CARE, OXFAM,Christian Aid,Save the ChildrenWWF,Media -TV,Newspapers,FreelancejournalistsInternationalcontractors,consultants,suppliersLocalgovernmentsNational NGOsmanagersfieldworkersProject &programmestaffActivists &lobbyistsDevelopmentacademicsPeople (beneficiaries)brokers, leaders, followers, patrons, clientsdiverse, differentiated, gendered, included/excluded,Politicians
  17. 17. Conclusions (cont)• Ethical analysis and publication– Respect• the interests of research subjects and researchers• interests of employers and funders• Peer groups and professions• Research as a social practise– Publish negative or null results– Enable replication