IDS Impact, Innovation and Learning Workshop March 2013: Day 2, Paper Session 3 Laura Camfield
Ensuring rigour and reliabilityLaura Camfield, University of East Anglial.firstname.lastname@example.org
Qualitative methods in modernimpact evaluation Who’s doing it? How are they doing it? How is its quality being assured? Andare existing checks sufficient? Ethical codes, peer review Archiving Restudies2
Endorsementfrom DFID (fundedreports by Garbarino andHolland, Stern) And fromNONIE, 3ie,World Bank IEG,FordFoundation, etc.
Growing interest in approaches tocausation that incorporatequalitative methods Small n/ case study Theory Based Realist Process tracing
Range of research designs limited Qualitative research typically post-hocrather than ex-ante Integrated analyses are rare Hard to access full methodological accounts– including how the analysis was done - orraw data
Growing numbers of mixed methodsdesigns, many ongoing 3ie – 700+ studies, 50 qual or mixedmethods (education, health, socialprotection, rural livelihoods) But qualitative component often smallor superficial,◦ e.g. sampling through participatory wealthranking or using criteria from PPA, ‘fieldvisits’, ‘qualitative surveys’
‘Empowering parents to improve education: evidence fromrural Mexico’◦ Reports data from 30 focus groups with parents from treatmentand control schools, but no analysis of differences betweengroups or how the programme promoted parental participation.For further information we are referred to an internal report ‘Changing Households’ Investments and Aspirations throughSocial Interactions’◦ Mentions ‘focus groups and semi-structured interviews with awide set of beneficiaries and other local actors’ which ‘suggeststhat aspirations and perspectives towards the future may havebeen a key part of program impact’ (p9). The methodology anddata underpinning these claims are in a report that has not beentranslated and isn’t available online
‘Assessing the Impacts of Farmer Field Schools onExcessive Fertilizer Use in China’ Detailed description of RCT methodology and sampling -‘qualitative data will also be collected to see what worked andhow it worked ‘Evaluating the Impact of Technology DevelopmentFunds in Emerging Economies: Evidence from LatinAmerica’ Qualitative methods not included in the methodology, butemerges like a rabbit from a hat ... ‘finally, based on qualitativeevidence from this study, financial support should beaccompanied by the infrastructure and technological services ofresearch centres and universities’
◦ Data production ‘Intellectual biographies’ of researchers andfieldworkers Quality of note taking – capturing embodiedknowledge?◦ Data management Transcription Translation◦ Analysis ‘Tertiary’ analysis◦ Representation Moving beyond quotes and text boxes Evidencing claims - ethnographic authority?9
1. Record of publications, presentations etc which explain and /ordraw on the archived data.2. Research designa) Brief context and logic of research design (in each data collectionperiod if appropriate).b) Is research exploratory or question driven? What are thequestions?3. What were the sampling decisions and how do they relate to theresearch questions?a) Was the desired sample achieved? How does the sample relate towider empirical evidence across the population and/or theoreticalissues?b) Are there implicit as well as planned ways in which the sample isstructured?4. An overview of what data is provided as part of the project.5. A descriptive profile of each participant specifying units of data(interview; diary; by wave).10
Rigour, Credibility, Relevance◦ 1. Was there a clear statement of the aims of the research?◦ 2. Is a qualitative methodology appropriate?◦ 3. Was the research design appropriate to address the aimsof the research?◦ 4. Was the recruitment strategy appropriate to the aims ofthe research?◦ 5. Were the data collected in a way that addressed theresearch issue?◦ 6. Has the relationship between researcher and participantsbeen adequately considered?◦ 7. Have ethical issues been taken into consideration?◦ 8. Was the data analysis sufficiently rigorous?◦ 9. Is there a clear statement of findings?◦ 10. How valuable is the research?
Validity in Quantitative and qualitative research12Quantitative researchQualitative researchBrymanGuba & Lincoln “Naturalistic inquiry”“trustworthiness and authenticity”Internal validity Legitimate causality claims Do the observations correspond well to thetheoretical constructs.Can you support claims that you observedwhat you claim?Credibility Multiple accounts of the same phenomenonresearcher validationrespondent validationtriangulationExternal validity Generalisability(representative samples)Can the qualitative findings be transferredacross social contexts.Transferability Research on small groups;Depth vs. breadth“Thick” description (Geertz)Reliability Repeatability, replicablity Could the qualitative research be replicated(access to qualitative inquiry “texts”)?Researchers adopt similar social rolesDo researchers agree about what they seeand here? – inter-observer credibilityDependability Trustworthiness by providing “audit trail” –extensive records of all aspects of the research,including raw dataRe-studiesObjectivity Objectivity, methodologicalrigourObjectivity in social science is spuriousValues more embedded in qualitativeresearchConfirmability Researcher worked in good faith
Possible solutions (Camfield and Palmer-Jones, 2013) Ethical codes, peer review Data archiving Restudies13
Social science ethical codes typically focused oncare of the subject, but see GoS’s (2007:1)emphasis on ‘respect, rigour and responsibility’ ‘act[ing] with skill and care in all scientific work’ ‘communicating results and intentions honestly andaccurately, and understanding that your work or itsoutputs will have an impact on society in its broadestsense’ Not all countries/ institutions/ disciplines haveethical codes or committees Ethics committees assess research designs, but donot monitor data production or presentation
Peer reviewers rarely see the data – noqualitative data deposit with journals – somuch is taken on trust/ reputation Grey literature and working papers may notreceive the same level of scrutiny, but are stillinfluential ‘Audit trail’ obscure in earlier examples
What does it offer?◦ Methodological insights◦ Increased use of data for research and teaching◦ Respect for respondents’ accounts and time◦ Judging validity of claims◦ Historical perspective Secondary analysis is difficult to do well,but within impact evaluation, mostqualitative analysis is secondary, or eventertiary
‘Scientific replication’◦ Goldthorpe and Lockwood (1962-3) tested the ‘affluent worker’hypothesis by taking highly-paid car assembly workers at theVauxhall factory in Luton as a ‘critical case’ to investigate whethereveryone was becoming middle-class. They concluded that thiswasn’t the case, albeit on the basis of a small quantitative sampleand (Savage argues) inadequate engagement with the data. Savage(2005:39) suggests that the conclusion arose because they hadfitted their data into a particular typology which closed offalternative interpretations. Methodological insights◦ Gillies and Edwards (2011:23) describe how in Townsend andMarsden’s (1965) study of single mothers, their “physicalattractiveness (or lack of it) is commented on [...and] perceivedintelligence and character was also subject to evaluation”.
Some examples from developing countries, e.g.ESRC funded restudy of Indian villages studied inthe 1950s by Bailey, Mayer and Pocock; vanSchendel, 1981 (Bangladesh); Breman et al., 1997(S. & S.E. Asia); Breman, 2007 (Gujarat, India), And some controversies, often due to ‘interpretiveoverreach’ (e.g. Freeman vs. Mead, Lewis vs.Redfield, Tierney vs. Chagnon) Why restudy? (Dis)confirming original findings Epistemological/ methodological insights Longitudinal perspective Running out of field sites...
Constructivist revisits (types 1 and 2) assume thesite being studied at two points in time does notchange so differences are due to the differentrelation of the ethnographer to the site (type I) ortheory that the ethnographer brings to the site(type 2) e.g. Weiners (1976) feminist reconstruction of Malinowski’s(1922) Argonauts of the Western Pacific highlighted theimportance of mortuary ceremonies in cementing women’scontrol over ancestral identity alongside Malinowski’scelebration of the ‘Kula ring’ Positionality and assumptions also important inevaluation
Realist revisits (types 3 and 4) studyhistorical change. Type 3 revisits focus oninternal processes in accounting fordifferences between authors’, while type 4emphasise external forces Hutchinson’s (1996) revisit to Nuerland (now SouthernSudan), which was studied by Evans-Pritchard (1940),explicitly to explore the impact of decolonization, war,Christianity, and transnational capital on the lives of theNuer Hard to reliably attribute and understand effects ofparticular interventions in complex, changingenvironments
Would Bemba society (N. Zambia) break down whenmen migrated to southern African mines becausethe slash and burn agricultural system (citimene)needed men to cut down the trees?◦ Richards – yes (conclusions enthusiastically adopted by colonialadministrators/ chiefs for their own reasons)◦ Moore and Vaughan (50 years later) – no (real threat was Zambiangovernment’s agrarian reforms which promoted the labour-intensive cash crop maize) But Moore and Vaughan’s analysis may also havebeen ‘one-sided, governed by specific feminist andFoucauldian assumptions’ (Burawoy, 2003:667) andeffects will be seen in future revisits…
What lessons can be learned forevaluation?◦ Watch out for interest groups!◦ Don’t forget about the women◦ Impacts of interventions are often unintended,multi-dimensional, and affect every member ofthe household, not just the target maize income and women’s labour controlled by men,women diverted from subsistence farming, familybecomes food insecure and women not entitled to takea share of the income to buy food, children’s healthsuffers...
Quality of research is an ethical issue Increasing role for qualitative research in impactevaluation requires increased reflection on currentstandards Epistemological and methodological problemscannot easily be detected or influenced throughconventional processes (standards, committees....) But qualitative researchers have own strategies forensuring rigour and reliability Data archiving and restudies contribute to these