Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Impact evaluation methods: Qualitative Methods


Published on

Presentation by Lini Wollenberg, Low Emissions Development Leader, CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS) at the Green Climate Fund Independent Evaluation Unit Learning-Oriented Real-Time Impact Assessment (LORTA)
Program Inception Workshop
July 24-26, 2018 Bangkok, Thailand

Published in: Science
  • Be the first to comment

  • Be the first to like this

Impact evaluation methods: Qualitative Methods

  1. 1. Lini Wollenberg, CGIAR Research Program on Climate Change, Agriculture and Food Security and Gund Institute, University of Vermont LORTA Program Inception Workshop, 24-26 July 2018 Impact Evaluation Methods: Qualitative Methods
  2. 2. Why qualitative methods? (1) Numbers reduce information! Use qualitative methods to capture things that cannot be easily measured with numbers - Where there is a need for assessing • system-level impacts • complexity • context, meaning, • depth, detail, nuance, • open-ended answers, unintended effects e.g. adaptation v. mitigation - Where things cannot be observed, and description is needed, e.g. historical extreme events, scaling potential of intervention, project contributions in complex impact pathways
  3. 3. (2) Flexibility and reduced expenses • Rapid assessment, e.g. exploring potential counterfactual explanations, activity data for mitigation, dealing with shifting adaptation strategies (3) People-oriented research • People are “in, not out” of assessment system: Participatory research, formative assessment, adaptive management • Ethics of measurement e.g. value of a life, value of ancestral lands, sensitive questions in humanitarian/emergency relief programs, • Where truth or knowledge is determined by social discourse and discussion - e.g. meaning, values, political issues, bridging multiple knowledge systems; critical social analysis Why qualitative methods? (2)
  4. 4. Examples of qualitative methods in impact assessment Bamberger 2012
  5. 5. Role of qualitative methods in the research process • Conceptual frameworks concerning social values, differences or power: human rights, legal, feminist or social inclusion frameworks • Research design: comparison; single case analysis; hypotheses may develop during research; flexibility for iterative, adaptive inquiry • Sampling: smaller numbers, purposive, representation of stakeholders • Data collection: may be less structured and open ended, more flexible, evolves with increased knowledge
  6. 6. Limits to qualitative methods Fit with scientific method • Limited statistical representativeness: can be highly context specific, small samples, difficult to generalize • Methods may not be as transparent or reproducible • Rich, but complicated information (spaghetti diagrams) • Causality can be difficult to isolate • Aggregation of results or comparison can be difficult Researcher as a person • Can require rapport or trust for optimal results • Researcher’s identity, perspective and behavior may affect methods (e.g. participant observation) Communicating results • Demand for numbers from policy makers, donors, media
  7. 7. Mixed methods Integrate approaches, not only in data collection, but in conceptual framework, design, analysis, interpretation Use multiple methods to • complement each other • triangulate results to test validity • inform each other (e.g. indicators for surveys) • support diverse ways of knowing or communicating and buy in from multiple audiences - Worry less about qualitative-quantitative divide and more about fit of method for aims - Problems arise more because of expectations and differences in team capacities and management
  8. 8. Example of mixed methods Bamberger 2012
  9. 9. Example of mixed methods Bamberger 2012
  10. 10. Qualitative assessment of climate action projects Some conclusions • Especially relevant to climate projects, e.g. assessing adaptation impacts, unexpected outcomes, complex impact pathways • Can facilitate learning and ownership to augment project outcomes. • Focus on quantitative or qualitative approach alone will limit results • Mixed method approaches, especially with sequential approaches, can ensure good mix of breadth and depth and effective project engagement and communication of results.
  11. 11. Some resources • GIZ. Impact evaluation guide for climate chage adaptation projects • Bamberger, M., Rugh, J., Church, M., Fort, L., Shoestring Evaluation: Designing Impact Evaluations under Budget, Time and Data Constraints, American Journal of Evaluation, 25(1), 2004, pp. 5 – 37. • Silvestrini, S. , Organizational Aspects of Evalu- ations in: Stockmann, R. (ed.), A Practitioner Handbook on Evaluation, Edward Elgar, Cheltenham, 2010. • Stockmann, R. (ed.), A Practitioner Handbook on Evaluation, Edward Elgar, Cheltenham, 2010.