Presentation by Lini Wollenberg, Low Emissions Development Leader, CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS) at the Green Climate Fund Independent Evaluation Unit Learning-Oriented Real-Time Impact Assessment (LORTA) Program Inception Workshop July 24-26, 2018 Bangkok, Thailand

1. Lini Wollenberg, CGIAR Research Program on Climate Change, Agriculture
and Food Security and Gund Institute, University of Vermont
LORTA Program Inception Workshop, 24-26 July 2018
Impact Evaluation Methods: Qualitative Methods

3. Why qualitative methods?
(1) Numbers reduce information!
Use qualitative methods to capture things that cannot be easily
measured with numbers
- Where there is a need for assessing
• system-level impacts
• complexity
• context, meaning,
• depth, detail, nuance,
• open-ended answers, unintended effects
e.g. adaptation v. mitigation
- Where things cannot be observed, and description is needed, e.g.
historical extreme events, scaling potential of intervention, project
contributions in complex impact pathways

4. (2) Flexibility and reduced expenses
• Rapid assessment, e.g. exploring potential counterfactual explanations,
activity data for mitigation, dealing with shifting adaptation strategies
(3) People-oriented research
• People are “in, not out” of assessment system: Participatory research,
formative assessment, adaptive management
• Ethics of measurement
e.g. value of a life, value of ancestral lands, sensitive questions in
humanitarian/emergency relief programs,
• Where truth or knowledge is determined by social discourse and
discussion
- e.g. meaning, values, political issues, bridging multiple knowledge
systems; critical social analysis
Why qualitative methods? (2)

6. Examples of qualitative methods
in impact assessment
Bamberger
2012

7. Role of qualitative methods in the
research process
• Conceptual frameworks concerning social values, differences or
power: human rights, legal, feminist or social inclusion frameworks
• Research design: comparison; single case analysis; hypotheses
may develop during research; flexibility for iterative, adaptive
inquiry
• Sampling: smaller numbers, purposive, representation of
stakeholders
• Data collection: may be less structured and open ended, more
flexible, evolves with increased knowledge

8. Limits to qualitative methods
Fit with scientific method
• Limited statistical representativeness: can be highly context
specific, small samples, difficult to generalize
• Methods may not be as transparent or reproducible
• Rich, but complicated information (spaghetti diagrams)
• Causality can be difficult to isolate
• Aggregation of results or comparison can be difficult
Researcher as a person
• Can require rapport or trust for optimal results
• Researcher’s identity, perspective and behavior may affect
methods (e.g. participant observation)
Communicating results
• Demand for numbers from policy makers, donors, media

9. Mixed methods
Integrate approaches, not only in data collection, but in conceptual
framework, design, analysis, interpretation
Use multiple methods to
• complement each other
• triangulate results to test validity
• inform each other (e.g. indicators for surveys)
• support diverse ways of knowing or communicating and buy in
from multiple audiences
- Worry less about qualitative-quantitative divide and more about fit of
method for aims
- Problems arise more because of expectations and differences in
team capacities and management

10. Example of mixed methods
Bamberger 2012

11. Example of mixed methods
Bamberger 2012

12. Qualitative assessment of climate action
projects
Some conclusions
• Especially relevant to climate projects, e.g. assessing adaptation
impacts, unexpected outcomes, complex impact pathways
• Can facilitate learning and ownership to augment project outcomes.
• Focus on quantitative or qualitative approach alone will limit results
• Mixed method approaches, especially with sequential approaches,
can ensure good mix of breadth and depth and effective project
engagement and communication of results.

13. Some resources
• GIZ. Impact evaluation guide for climate chage adaptation projects
https://www.adaptationcommunity.net/?wpfb_dl=260
• Bamberger, M., Rugh, J., Church, M., Fort, L., Shoestring
Evaluation: Designing Impact Evaluations under Budget, Time and
Data Constraints, American Journal of Evaluation, 25(1), 2004, pp.
5 – 37.
• Silvestrini, S. , Organizational Aspects of Evalu- ations in:
Stockmann, R. (ed.), A Practitioner Handbook on Evaluation,
Edward Elgar, Cheltenham, 2010.
• Stockmann, R. (ed.), A Practitioner Handbook on Evaluation,
Edward Elgar, Cheltenham, 2010.