1) Evaluation and impact assessment approaches can be categorized based on their goals (formative, summative, developmental) and methods (qualitative, quantitative, mixed). 2) In organizations like the CGIAR, impact assessment was initially implemented to ensure accountability but now aims to facilitate organizational learning. 3) For impact assessment to influence organizational change in the CGIAR, it needs to help build consensus on needed reforms, develop new frameworks for analyzing social processes, and work with coalitions of supportive stakeholders.

1. What we think is what matters: the role of impact assessment (IA) in the CGIAR change management Javier M. Ekboir 1

2. Overview of the presentation Review of evaluation and IA approaches How can philosophy of science contribute to evaluation and IA? The nature of organizational change Evaluation and IA in the history of the CGIAR Evaluation and IA in the on-going change process 2

3. Evaluation and IA can be categorized from two perspectives: goals and methods Goals: summative or judgment-oriented to determine the value of a program formative or improvement-oriented to identify steps to improve a program developmental to facilitate learning during implementation 3

4. Evaluation and IA can be categorized from two perspectives (2) Methods: Qualitative Quantitative rates of return and cost-benefit analysis other approaches, especially related to how to measure changes in indicators in non-experimental settings Combination of qualitative and quantitative 4

5. Which approaches and methods are appropriate depends on what important stakeholders think is appropriate! 5

6. The role of data in learning 6 Argentina’s amazingdiscovery Every data set (experimental or not) is open to an infinite number of hypotheses (underdetermination thesis) Let me show you the picture of a bird Or is it a rabbit? Data are collected and have a meaning within a “theoretical” framework (“theory-ladenness” of data) Data are important because they provide information to build and “confirm” theories

7. Hypotheses cannot be tested (T, H, A1, A2 … An) Then, what do researchers do? They “build explanations” But there is no consensus on what a valid explanation is Scientists build narratives that reflect how they see the processes they study 7

8. The social nature of knowledge The acceptance or rejection of a narrative depends on the consensus among the majority of scientists and other stakeholders about what is a good explanation 8 Consensus about climate change This does not mean that all knowledge is relative Just that knowledge is determined by the interaction of data and social conventions

9. Organizational change depends, among other factors, on: The nature of leadership 9 The organization’s culture The organization’s learning routines Governance In NGOs, also external consensus about what needs to be done

10. The nature of change in organizations Stakeholders influence organizational change according to their beliefs, resources and connections 10 Similar minded stakeholders form informal coalitions The more effective a coalition, the greater its influence on the change process

11. The nature of change in organizations (2) Influence can be gained in two ways: commanding resources convincing other stakeholders by presenting a compelling argument about what change is needed and how to measure it Evaluations do not bring organizational change by themselves, but influence the perception of what changes are needed 11

12. Why was impact assessment implemented in the CGIAR? In the 1990s, CGIAR centers had to justify their existence, i.e., accountability was the most important motive for IA For decades, economists had calculated rates of return to research, which was considered by some professionals a rigorous methodto measure impacts Some of these professionals occupied important positions among donors, in the science council and in the centers Over time, other approaches for IA were developed by CGIAR and external scientists, but they remained at the fringes of the system 12

13. Why was impact assessment implemented in the CGIAR? (2) In recent years, some donors and scientists realized that an alternative to the rates of return was needed New approaches for evaluation, IA and organizational learning are being adopted by donors, CGIAR scientists, other organizations working on development, and important segments of the evaluation community The issue today is what type of IA meets scientific standards and is usefulfor the donors and the CGIAR 13

14. IA and change management in the CGIAR Evaluations influence the change management process by helping to create a consensus about the new role of the CGIAR and the changes needed IA, as designed by the SC, did not induce change in the system How the new approaches for evaluation can influence change in the CGIAR depends on the system’s ability to use the new information This depends, among other factors, on the coalitions that influence the process 14

15. How can new evaluation approaches help change in the CGIAR? Generating new frameworks and information to help the Consortium, similarly-minded donors and DGs to provide effective leadership for change Synthesizing the new approaches for evaluation and IA developed by CGIAR scientists and providing a unifying framework 15

16. How can new evaluation approaches help change in the CGIAR? (2) Developingnew approaches for IA based on novel frameworks for the analysis of social processes Building a new, compelling argument for reassessing the role of science (including the CGIAR) and IA in development by bringing in new ideas and challenging orthodoxies 16

17. On its own, IA will have little impact on the change process To have an impact, it is necessary to work simultaneously along three lines of action: Give more visibility to the new methods for evaluation and IA developed by CGIAR and/or external scientists Develop a conceptual framework that unifies the new methods for evaluation and IA and provides a compelling argument for using evaluation as a tool for organizational learning Build a coalition of similarly-minded stakeholders 17

18. Thank you! 18