Evaluation Capacity Development


Published on

Strong national and sector monitoring and evaluation systems are necessary to assess efforts to reduce poverty, and to demonstrate results.

Published in: Leadership & Management
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Evaluation Capacity Development

  1. 1. The views expressed in this presentation are the views of the author/s and do not necessarily reflect the views or policies of the Asian Development Bank, or its Board of Governors, or the governments they represent. ADB does not guarantee the accuracy of the data included in this presentation and accepts no responsibility for any consequence of their use. The countries listed in this presentation do not imply any view on ADB's part as to sovereignty or independent status or necessarily conform to ADB's terminology. Evaluation Capacity Development Olivier Serrat 2014
  2. 2. The Planning, Monitoring, and Evaluation Triangle Planning MonitoringEvaluation Recommendations are made for future planning Plans show what to evaluate Monitoring revises plans during project implementation Plans show what needs to be monitored Evaluation highlights areas that need close monitoring Monitoring provides data to be used in evaluation
  3. 3. Define:Monitoring • Monitoring is the continuous collection of data and information on specified indicators to assess the implementation of a development intervention in relation to activity schedules and expenditure of allocated funds, and progress and achievements in relation to its intended outcome. Monitoring • Involves day-to-day follow-up of activities during implementation to measure progress and identify deviations; • Requires routine follow-up to ensure activities are proceeding as planned and are on schedule; • Needs continuous assessment of activities and results; • Answers the question, "what are we doing?" Monitoring
  4. 4. Define:Evaluation • Evaluation is the periodic assessment of the design implementation, outcome, and impact of a development intervention. It should assess the relevance and achievement of the intended outcome, and implementation performance in terms of effectiveness and efficiency, and the nature, distribution, and sustainability of impact. Evaluation • Is a systematic way of learning from experience to improve current activities and promote better planning for future action; • Is designed specifically with the intention to attribute changes to the intervention itself; • Answers the question, "what have we achieved and what impact have we had?" Evaluation
  5. 5. The Results Chain Outputs Outcome Impact ActivitiesInputs
  6. 6. Outputs, Outcome, Impact Outputs—the products, capital goods, and services that result from a project; they may also include changes resulting from the project that are relevant to the achievement of its outcome. Outcome—the likely or achieved short-term and medium-term effect of a project's outputs. Impact—the positive and negative, primary and secondary, long- term effect produced by a project, directly or indirectly, intended or unintended.
  7. 7. OECD-DAC Evaluation Criteria Relevance—This criteria examines the extent to which the objectives of a project matched the priorities or policies of major stakeholders (including beneficiaries). Effectiveness—This criteria examines whether outputs led to the achievement of the planned outcome. Efficiency—This criteria assesses outputs in relation to inputs. Impact—This criteria assesses what changes (intended and unintended) have occurred as a result of the work. Sustainability—This criteria looks at how far changes are likely to continue in the longer term.
  8. 8. The Results Chain and the OECD- DAC Evaluation Criteria Needs Objective Inputs Activities Outputs Outcome Impact Relevance Efficiency Effectiveness Sustainability
  9. 9. Challenges and Limits to Management Logic Degree of Control Challenge of Monitoring and Evaluation Impact What the project is expected to contribute to Outcome What the project can be expected to achieve and be accountable for Outputs What is within the direct control of the project's management Activities Inputs DecreasingControl IncreasingDifficulty
  10. 10. Life Cycle of Monitoring and Evaluation EA MT EPEA MT EPEA MT EP Key: EA = ex-ante, MT = mid-term, EP = ex-post
  11. 11. Trends in Evaluation Capacity Development Build Post-Evaluation Capability Build Monitoring and Evaluation Systems Establish Performance Management Systems
  12. 12. Evaluation Capacity Development Precondition 3 Precondition 2 Precondition 1 Three critical preconditions to success of evaluation capacity development are (i) substantive government demand, (ii) existence of a mandate by decree for evaluation, and (iii) stability in staffing such that a very high proportion of trained personnel remain in tasks for which they were trained.
  13. 13. Evaluation Capacity Development Good practice in developing evaluation capacity also recognizes that: • Monitoring and evaluation systems are a means to an end—benefits are obtained when results are used in decision making. • It is advisable to locate responsibility for monitoring and evaluation near the capable head of an organization. • Monitoring and evaluation systems should not become too complex or resource-intensive. • Monitoring and evaluation systems encompass data collection in the field and aggregation and analysis by end users. • Evaluation capacity development that concentrates on the oversight agency carries the risk that other entities may lack incentives to provide data and information. • Case studies help develop staff competency and confidence.
  14. 14. TA 6410–REG: Capacity Development for Monitoring and Evaluation
  15. 15. TA 6410–REG: Capacity Development for Monitoring and Evaluation Outputs 1. Proficiency in monitoring and evaluation is raised 2. Research and special studies on evaluation capacity development are conducted 3. Knowledge sharing and learning for monitoring and evaluation are boosted Outcome Improved ranges of skills, resources, systems, and attitudes for performance― in the evaluation agencies targeted―of results-based monitoring and evaluation of country partnership strategies, sector strategies, policies, programs, and projects in developing member countries of the Greater Mekong Subregion, namely, Cambodia, Lao PDR, and Viet Nam Impact Higher efficiency and effectivenes s in providing public sector services, leading to poverty reduction
  16. 16. Key: M&E = monitoring and evaluation, ECD = evaluation capacity development TA 6410–REG: Capacity Development for Monitoring and Evaluation Proficiency in M&E Tools, Methods, and Approaches for M&E Strategy and Policy Formulation for M&E Research and Special Studies for M&E Country Strategies for M&E A Strategy for ECD
  17. 17. Key: M&E = monitoring and evaluation TA 6410–REG: Capacity Development for Monitoring and Evaluation Knowledge Sharing and Learning for M&E Knowledge Sharing and Learning Platforms Knowledge Networks Partnership Arrangements with Evaluation Associations
  18. 18. Output Level: Assumptions and Risks Assumptions Risks ADB • Technical assistance activities integrate the chief lessons learned from past evaluations • Appropriate, integrated training programs can be planned, designed, or identified; and synergetic effects can be achieved • Training is conducted well and according to realistic schedules • The consultants and the selected evaluation agency staff coordinate activities effectively • The consultants have client management skills • The consultants and the selected evaluation agency staff maintain clear roles, responsibilities, and deadlines Clients • Evaluation agency staff are available to be trained ADB • The indicative activities and staffing schedule is too tight to permit productive sequencing of key activities TA 6410–REG: Capacity Development for Monitoring and Evaluation
  19. 19. Assumptions Risks Clients • Basic capacity exists and can be mobilized • The role and use of monitoring and evaluation in support of practices of knowledge management are understood • The funding agency has a clear vision about the intended outcome of the technical assistance and how it is to be achieved Clients • Evaluation agencies underestimate the importance of national ownership and leadership of the evaluation process and of building national monitoring and evaluation capacities TA 6410–REG: Capacity Development for Monitoring and Evaluation Outcome Level: Assumptions and Risks
  20. 20. Assumptions Risks ADB • Able to plan and support evaluation capacity development in line with international best practice Clients • Provide visible leadership, promote clear sense of mission, encourage participation, and establish explicit expectations on performance and rewards • Strategically approach change management and manage it proactively • Involve a critical mass of staff • Try, test, and adapt organizational innovations • Celebrate quick wins Clients • Lack of human security; armed conflict; economic policies that discourage pro-poor growth; weak scrutiny by the legislative branch of the executive branch; ineffective voice of intended beneficiaries; and corruption, clientelism, or patrimonialism do not provide a broadly enabling environment for monitoring and evaluation • Fragmented government with poor overall capacity; absent, noncredible, and/or rapidly changing policies; unpredictable, unbalanced, or inflexible funding and staffing; poor public service conditions; segmented and compartmentalized organizations; or insufficient commitment to an evaluation culture do not conduce to government effectiveness TA 6410–REG: Capacity Development for Monitoring and Evaluation Impact Level: Assumptions and Risks
  21. 21. ADB Center for Development and Research in Evaluation (Shanghai) International Program for Development Evaluation Training Asia-Pacific Finance and Development Center Regional Cooperation and Poverty Reduction Fund of the People's Republic of China TA 6410–REG: Capacity Development for Monitoring and Evaluation Technical Assistance Partnerships
  22. 22. TA 6410–REG: Capacity Development for Monitoring and EvaluationTajikistan • Amirov Fakhriddin K., State Budget Department, Ministry of Finance Lao PDR • Vixay Xaovana, Committee for Planning and Investment • Akhom Praseuth, Bank of Lao PDR • Bounthay Leuangvilay, Budget Department, Ministry of Finance Cambodia • Hou Taing Eng, Ministry of Planning • Im Sour, Cambodian Rehabilitation and Development Board • Suon Sophal, Cambodian Investment Board • Lors Pinit, Department of Investment and Cooperation • Hay Sovuthea, Supreme National Economic Council Malaysia • Arunaselam Rasappan, Center for Development and Research in Evaluation • Mariappan Mahalingam, Center for Development and Research in Evaluation Viet Nam • Tran Ngoc Lan, Ministry of Planning and Investment • Nguyen Dang Binh, Ministry of Planning and Investment • Pham Thai Linh, Ministry of Natural Resources and Environment • Nguyen Trang Thu, National Academy of Public Administration Selected Evaluation Agencies
  23. 23. Project Year 1 2 1. Proficiency in M&E is raised. 1.1 Proficiency in tools, methods, and approaches for M&E is raised. (i) Regional training-of-trainers (SHIPDET) (ii) National training-of-trainers 1.2 Proficiency in strategy and policy formulation for M&E is raised. (i) Thefoundations for M&E in GMS DMCs arestrengthened. (ii) GMS DMCs areassisted in setting long-term, consistent strategies. (iii) International training (IPDET) 2. Research and special studies on ECD areconducted. 2.1 Development of country strategies for M&E is supported. 2.2 Astrategy for ECD is suggested. 3. Knowledgesharing and learning for M&E areboosted. (i) Selected knowledgesharing and learning platforms areenhanced. (ii) Adviceon new and existing knowledgenetworks on M&E is extended to evaluation agency staff fromGMS DMCs. (iii) Partnership arrangements with interested evaluation associations promoted and concluded. Legend: Full-timeactivity Intermittent activity AFDC =Asia-Pacific Financeand Development Center, CeDRE =Center for Development and Research in Evaluation, IM&E =International Monitoring and Evaluation Specialist, IPDET =International Programfor Development Evaluation Training, M&E =monitoring and evaluation Activity TA 6410–REG: Capacity Development for Monitoring and Evaluation Technical Assistance Schedule
  24. 24. Training and Capacity Building Research and Special Studies Knowledge Sharing and Networking Strategic Direction for Evaluation Capacity Development Strengthened Evaluation Capacity Improved Service Delivery Leading to Poverty Reduction TA 6410–REG: Capacity Development for Monitoring and Evaluation Expected Short-, Medium-, and Long-Term Deliverables
  25. 25. Stage A1 (Apr. 2008) Introductory Training in M&E for Country Trainers (ToT) CeDRE In-Country Preparatory Work by Trainees In-CountryStage A2 (Apr. –Sept. 2008) Policy Level M&E Training (Round 1) IPDET 1Stage B1 (June 2008) Intermediate M&E Training for Trainees Lao PDRStage A3 (Oct. 2008) In-Country Stage 1 Down-line Training by Trainees In-CountryStage A4 (Nov. 2008–Mar. 2009) Advanced Level M&E Training for Trainees CambodiaStage A5 (Apr. 2009) Policy Level M&E Training (Round 2) IPDET 2Stage B2 (June 2009) In-Country Stage 2 Down-line Training by Trainees In-CountryStage A6 (Jul.–Sept. 2009) Wrap-Up Training & Certification of Country Trainers AFDC/ADBStage A7 (Oct. 2009) TA 6410–REG: Capacity Development for Monitoring and Evaluation Training Strategy
  26. 26. TA 6410–REG: Capacity Development for Monitoring and Evaluation Evaluation of Output 1 Output 1 had two components: (i) a three-stage training-of -trainers (ToT) program in monitoring and evaluation (M&E) (1–2 weeks per stage), provided by the TA's international consultant to 18 mid-level career officials recommended by ADB's Greater Mekong Subregion coordinators from the three countries covered (6 per country, including one national consultant each); and (ii) a 1-week country-based M&E downline training, provided to 20 key officials selected as M&E focal persons from relevant government agencies in the three DMCs. Most activities under this output were implemented and completed on track. Subsequently, the officials trained under the ToT program successfully conducted M&E downline workshops in their respective countries. Attendance at the Shanghai International Program for Development Evaluation Training (SHIPDET) was found particularly useful.
  27. 27. TA 6410–REG: Capacity Development for Monitoring and Evaluation Evaluation of Output 2 In synergy, the two components of output 2 circumscribed the M&E situation and clarified training needs in each of the three countries covered. Activities under this output were generally implemented well, even if the final M&E report was delayed. The detailed country case studies provided valuable information on the status, progress, and challenges of M&E. In each of the three countries covered, areas of investigation included (i) institutional structure, policy formulation, and strategy for M&E; and (ii) illustrative approaches to and practical application of M&E policy measures.
  28. 28. TA 6410–REG: Capacity Development for Monitoring and Evaluation Evaluation of Output 3 The first of output 3's two components set up an M&E Community of Practice website at www.linkingtoresults.org, which the national consultants engaged under the TA were tasked to promote through knowledge sharing on current M&E practices and issues in the three countries covered. Despite these efforts, the website was used sparingly due to (i) limited broadband access in government offices, especially in Cambodia and Lao, PDR; (ii) insufficient language skills in English, necessary to enable efficient online communications; and (iii) lack of time. The second component aimed to facilitate the emergence of national evaluation networks/societies. In the immediate, regulatory restrictions in the three countries covered delayed this effort; but, Cambodia and Viet Nam may soon move forward as registrations are reportedly being processed by the relevant authorities.
  29. 29. TA 6410–REG: Capacity Development for Monitoring and Evaluation Achievement of Outcome Despite limitations in output 3, the successful delivery of outputs 1 and 2 contributed to the achievement of the TA's outcome. This was reflected in the fact that the TA helped create, under the ToT program, an internal pool of at least 6 M&E resource persons in each of the three countries covered, who in turn were able to successfully conduct downline M&E training workshops for key officials selected as M&E focal points from concerned government agencies. Feedback from those attending the downline workshops indicated the usefulness of the workshops in building their understanding of M&E and its role in managing for development results (MfDR) as the workshops were customized to national contexts.
  30. 30. TA 6410–REG: Capacity Development for Monitoring and Evaluation Overall Assessment and Rating The overall performance of the TA was rated successful based on the combined ratings of relevant, efficient, effective, and likely sustainable. Most TA activities were generally implemented efficiently and completed on time. This led to good delivery of outputs, except for output 3 where use of the M&E CoP website fell short and national evaluation networks/societies were not immediately established. Notwithstanding, the successful delivery of outputs 1 and output 2 led to effectiveness in achieving outcome in terms of improved ranges of M&E skills and awareness of the importance role of M&E. In terms of design, the TA's ToT program was found relevant: it enabled the key officials trained as trainers to provide downline M&E training workshops to relevant staff of concerned agencies in their respective countries. This is expected to help sustain the outcome achieved.
  31. 31. TA 6410–REG: Capacity Development for Monitoring and Evaluation Major Lessons Five lessons stand out: (i) baseline situational analysis of M&E status and needs helped design relevant training courses; (ii) the training courses, which began with a ToT program, helped create pools of trainers able to deliver country-based M&E downline training workshops to staff in concerned agencies in the three countries covered, which ought to sustain and internalize the M&E knowledge and capacity gained; (iii) there should be no rush to design outputs susceptible to in-country obstacles, e.g., low levels of IT advancement, tight regulatory environments, unless they can be addressed first; (iv) the TA should have worked more closely with ADB's resident missions, which could have provided focal points in support of TA implementation; and (v) a follow-on experience sharing workshop should have been planned to estimate the magnitude of "actual" applications of the M&E knowledge and capacity gained and to gauge retention, on M&E-related tasks, of the officials trained.
  32. 32. Further Reading • ADB. 2007. Independent Evaluation at the Asian Development Bank. Manila. www.adb.org/publications/independent- evaluation-asian-development-bank • ——. 2007. Regional Technical Assistance for Capacity Development for Monitoring and Evaluation. Manila. www.adb.org/projects/documents/capacity-development- monitoring-and-evaluation-0 • ——. 2009. Learning from Evaluation. Manila. www.adb.org/publications/learning-evaluation
  33. 33. Further Reading • ADB. 2009. Regional Technical Assistance for Capacity Development for Capacity Development in Results-Based Monitoring and Evaluation for Countries under the Asian Development Bank's Central Asia Regional Economic Cooperation Program. Manila. www.adb.org/projects/documents/capacity-development- results-based-monitoring-and-evaluation-countries-under- asia • ——. 2011. Technical Assistance Completion Report: Regional Technical Assistance for Capacity Development for Monitoring and Evaluation. Manila. www.adb.org/projects/documents/capacity-development- monitoring-and-evaluation
  34. 34. Further Reading • Britton, B. and Serrat, O. 2013. Learning from Evaluation. www.slideshare.net/celcius233/learning-from-evaluation
  35. 35. Quick Response Codes @ADB @ADB Sustainable Development Timeline @Academia.edu @LinkedIn @ResearchGate @Scholar @SlideShare @Twitter