Latest Trends in Independent Evaluation at ADB


Published on

Independent evaluation has changed from the beginnings of evaluation activities in ADB some 30 years ago. Initially, the focus of evaluation was on assessing whether implementation was consistent with the intentions reflected in the appraisal of a project, and the extent to which the project achieved the expected economic and social benefits. Independent evaluation now shapes decision making throughout the project cycle and in ADB as a whole. This presentation offers more than an account of the early steps: it describes recent accomplishments and looks to the future.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Latest Trends in Independent Evaluation at ADB

  1. 1. The views expressed in this presentation are the views of the author/s and do not necessarily reflect the views or policies of the Asian Development Bank, or its Board of Governors, or the governments they represent. ADB does not guarantee the accuracy of the data included in this presentation and accepts no responsibility for any consequence of their use. The countries listed in this presentation do not imply any view on ADB's part as to sovereignty or independent status or necessarily conform to ADB's terminology. Latest Trends in Independent Evaluation at ADB Olivier Serrat 2010
  2. 2. The Changing Operations Evaluation Landscape There is new emphasis on aid effectiveness from increased demand for better development results. The aid community faces challenges caused by the crowded policy agenda, e.g., conflict reduction and prevention, fair international trade, environmental agreements, gender, a new international financial architecture, etc., many of which suffer from definitional problems and unclear time horizons, making them intrinsically difficult to evaluate. The aid community finds now often works with new partners in different partnership configurations; multi-agency consortia are becoming more common and agencies are moving beyond government-to-government links.
  3. 3. The Changing Operations Evaluation Landscape The growing influence of nongovernment and civil society organizations is adding dimensions, often positive, to debates about aid and its delivery. New instruments, e.g., sector-wide approaches, comprehensive development frameworks, the livelihoods approach, etc., are being tried to change the way the aid community works; they require significant shifts in how evaluation is approached. Aid delivery has been shifting to new horizons—from the project to the program to the country levels; there is also an array of issues, e.g., climate change, trade, migration, etc., that require regional or global approaches.
  4. 4. Agents of Evaluation in ADB In ADB, self-evaluation is carried out by the operations departments responsible for designing and implementing country partnership strategies, programs, projects, and technical assistance activities. Independent evaluation is conducted by the Independent Evaluation Department to systematically evaluate policies, strategies, programs, and projects, e.g., design, implementation, results, and associated business processes, to determine relevance, efficiency, effectiveness, impact, and sustainability.
  5. 5. Audiences for Evaluation Independent Evaluation Department (evaluations, monitoring and evaluation expertise, self-assessment) ADB Operations Departments (monitoring and evaluation, results agenda, applied learning) ADB Management (use and follow-up of evaluations, influence on ADB directions) ADB Board of Directors (oversight) International Evaluation Community (harmonization and joint evaluations) Developing Member Countries (use of evaluations, evaluation capacity development, joint evaluations)
  6. 6. Purpose of Independent Evaluation Development Effectiveness—to contribute to development results, i.e., maximize the development effectiveness of ADB's operations. Accountability —to confirm that ADB is doing the right things, that resources are properly allocated and used, that intended outcomes are realized. Generating Knowledge/Learning Lessons—to identify or derive lessons to improve the development impact of future policies, strategies, and operations.
  7. 7. Evolution of Independent Evaluation in ADB 1972—A unit is established in the Economics Office. 1978—A Post-Evaluation Office is set up. 1999—An Operations Evaluation Office is formed. 2001—An Operations Evaluation Department is constituted. 2004—The Evaluation Head reports to the Board of Directors of ADB through its Development Effectiveness Committee. 2009—An Independent Evaluation Department is created whose head is to be appointed by the Board.
  8. 8. The Role of the Development Effectiveness Committee The Development Effectiveness Committee in ADB was established in December 2000. The DEC's mandate is to help the Board of Directors of ADB carry out its responsibility of ensuring that the programs and activities of ADB achieve development effectiveness. Development effectiveness is assessed in part through independent evaluation. For the purpose of the DEC's work, development effectiveness is the measure of (i) whether ADB's programs and activities have resulted in the desired outcomes; and (ii) whether these programs and activities have made efficient use of ADB's available resources.
  9. 9. The Role of IED The core task of IED is the systematic and objective assessment of policies, strategies, programs, and projects, including their design, implementation, results, and associated business processes, to determine their relevance, effectiveness, efficiency, sustainability, and impact. • It evaluates the development effectiveness of operations. • It derives lessons to improve future operations. • It contributes to ADB's accountability for results. • It provides assistance to the DEC as the focal point for interaction with the Board of Directors. • It seeks to improve self- evaluation in ADB.
  10. 10. A Few Influential EvaluationsBuildingaResults-Based ManagementFramework, 2003— PromotingPortfolio Performance,2006— ImprovingCountry PartnershipStrategies, 2000s— ContributingtoSafeguard PolicyUpdate Involuntary Resettlement, 2006 Environment, 2006 Indigenous Peoples, 2007
  11. 11. Issues in Independent Evaluation The balance between accountability and learning, and between independence and engagement Conflict between closer engagement with developing member countries and Management Independence from the Board— is that advisable? Attribution or contribution—not all evaluations can address attribution issues How far should one move on self- evaluation and validation? How might one improve buy-in? Conduct fewer individual public sector project evaluations? Launch more country, sector, impact, and policy evaluations? Prepare more private sector evaluations?
  12. 12. Issues in Independent Evaluation How to improve credibility— through stronger evidence? How to make better use of evaluation results? Which evaluation methods to apply, e.g., qualitative or quantitative; rigorous or non- rigorous; experimental or non- experimental)? How to assess impact and sustainability? How to strengthen the accountability of Management? Recommendations must be clear, actionable, and monitorable How about the accountability of IED itself?
  13. 13. Knowledge Management for Independent Evaluation Efforts to increase external dissemination through, among other platforms, are well received. IED invests continuingly in a searchable lessons database system that includes well over 1,000 lessons to date. Efforts are being made to improve dissemination of findings. A Knowledge Management Unit was set up in IED in late 2006. To be used, evaluation findings must be available when needed by decision makers, in user-friendly formats. Evaluation findings only add value when they are used.
  14. 14. Matching Feedback Vehicles to Accountability and Learning
  15. 15. Matching Feedback Vehicles to Accountability and Learning
  16. 16. Matching Feedback Vehicles to Accountability and Learning
  17. 17. Matching Feedback Vehicles to Accountability and Learning
  18. 18. Matching Feedback Vehicles to Accountability and Learning
  19. 19. Matching Feedback Vehicles to Accountability and Learning
  20. 20. Matching Feedback Vehicles to Accountability and Learning
  21. 21. Matching Feedback Vehicles to Accountability and Learning
  22. 22. Matching Feedback Vehicles to Accountability and Learning
  23. 23. Improving Evaluation-Based Learning ADB's strategic frameworks are moving toward greater concentration on country, sector, and thematic areas, and greater decentralization. ADB sees itself as a learning organization where operations evaluation plays a key part, and is given greater institutional independence and greater priority. More knowledge is generated from evaluations using more sophisticated methods. More impact-orientation is needed for quality improvement, conceptual advancement of development, and greater external accountability.
  24. 24. Improving Evaluation-Based Learning ADB must systematize and institutionalize feedback systems for evaluation-based learning and accountability. Efforts are needed to manage the quickening flood of data and information, network the systems, and cater for upward feedback. ADB must intensify the internalization of lessons from evaluation. Implementation monitoring, that is, implementation of findings and recommendations from evaluation, should be strengthened by means of a Management action record system.
  25. 25. Improving Evaluation-Based Learning ADB should continue to publicly release proposed evaluation approach papers at an early stage and post the comments of external stakeholders. ADB should continue to recognize the need for greater participation of in-country partners and stakeholders in the feedback process. Moving from project evaluations to a higher-level country, sector, and thematic focus is an important way of increasing potential evaluation impact.
  26. 26. Possible Directions in Evaluation Assigning resources to broader evaluation studies Harmonizing evaluation standards (Jointly) Evaluating country/sector assistance programs Validating country partnership strategy and project/program completion reports Evaluating impact Developing evaluation capacity Promoting portfolio performance
  27. 27. Possible Directions in Evaluation Evaluating business processes Disseminating findings and recommendations Promoting monitoring and measuring of outcomes Institutional evaluations , e.g., human resource function, information technology, knowledge management, Asian Development Bank Institute Rigorous impact evaluations
  28. 28. • ADB. 2007. Independent Evaluation at the Asian Development Bank. asian-development-bank • ——. 2008. Conducting After-Action Reviews and Retrospects. and-retrospects • ——. 2009. Learning from Evaluation. • ——. 2009. The Most Significant Change Technique. • ——. 2011. Learning Histories. Further Reading
  29. 29. Quick Response Codes @ADB @ADB Sustainable Development Timeline @LinkedIn @ResearchGate @Scholar @SlideShare @Twitter