Evaluation Capacity Development
Upcoming SlideShare
Loading in...5
×
 

Evaluation Capacity Development

on

  • 511 views

Strong national and sector monitoring and evaluation systems are necessary to assess efforts to reduce poverty, and to demonstrate results.

Strong national and sector monitoring and evaluation systems are necessary to assess efforts to reduce poverty, and to demonstrate results.

Statistics

Views

Total Views
511
Views on SlideShare
509
Embed Views
2

Actions

Likes
0
Downloads
18
Comments
0

2 Embeds 2

https://twitter.com 1
http://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Evaluation Capacity Development Evaluation Capacity Development Presentation Transcript

  • Evaluation Capacity Development Olivier Serrat 2014 The views expressed in this presentation are the views of the author/s and do not necessarily reflect the views or policies of the Asian Development Bank, or its Board of Governors, or the governments they represent. ADB does not guarantee the accuracy of the data included in this presentation and accepts no responsibility for any consequence of their use. The countries listed in this presentation do not imply any view on ADB's part as to sovereignty or independent status or necessarily conform to ADB's terminology.
  • The Planning, Monitoring, and Evaluation Triangle Planning Recommendations are made for future planning Plans show what needs to be monitored Plans show what to evaluate Evaluation highlights areas that need close monitoring Evaluation Monitoring provides data to be used in evaluation Monitoring revises plans during project implementation Monitoring
  • Define:Monitoring Monitoring • Monitoring is the continuous collection of data and information on specified indicators to assess the implementation of a development intervention in relation to activity schedules and expenditure of allocated funds, and progress and achievements in relation to its intended outcome. Monitoring • Involves day-to-day follow-up of activities during implementation to measure progress and identify deviations; • Requires routine follow-up to ensure activities are proceeding as planned and are on schedule; • Needs continuous assessment of activities and results; • Answers the question, "what are we doing?"
  • Define:Evaluation Evaluation • Evaluation is the periodic assessment of the design implementation, outcome, and impact of a development intervention. It should assess the relevance and achievement of the intended outcome, and implementation performance in terms of effectiveness and efficiency, and the nature, distribution, and sustainability of impact. Evaluation • Is a systematic way of learning from experience to improve current activities and promote better planning for future action; • Is designed specifically with the intention to attribute changes to the intervention itself; • Answers the question, "what have we achieved and what impact have we had?"
  • The Results Chain Impact Outcome Inputs Activities Outputs
  • Outputs, Outcome, Impact Outputs—the products, capital goods, and services that result from a project; they may also include changes resulting from the project that are relevant to the achievement of its outcome. Outcome—the likely or achieved short-term and medium-term effect of a project's outputs. Impact—the positive and negative, primary and secondary, longterm effect produced by a project, directly or indirectly, intended or unintended.
  • OECD-DAC Evaluation Criteria Relevance—This criteria examines the extent to which the objectives of a project matched the priorities or policies of major stakeholders (including beneficiaries). Effectiveness—This criteria examines whether outputs led to the achievement of the planned outcome. Efficiency—This criteria assesses outputs in relation to inputs. Impact—This criteria assesses what changes (intended and unintended) have occurred as a result of the work. Sustainability—This criteria looks at how far changes are likely to continue in the longer term.
  • The Results Chain and the OECDDAC Evaluation Criteria Needs Impact Outcome Objective Inputs Activities Relevance Efficiency Effectiveness Sustainability Outputs
  • Challenges and Limits to Management Degree of Control Logic Impact Outputs Activities Inputs What the project can be What is within expected to achieve and the direct be control of the accountable project's for management Increasing Difficulty Outcome Decreasing Control What the project is expected to contribute to Challenge of Monitoring and Evaluation
  • Life Cycle of Monitoring and Evaluation EA MT EA EP MT EA EP MT Key: EA = ex-ante, MT = mid-term, EP = ex-post EP
  • Trends in Evaluation Capacity Development Build Monitoring and Evaluation Systems Establish Performance Management Systems Build Post-Evaluation Capability
  • Evaluation Capacity Development Precondition 3 Precondition 2 Precondition 1 Three critical preconditions to success of evaluation capacity development are (i) substantive government demand, (ii) existence of a mandate by decree for evaluation, and (iii) stability in staffing such that a very high proportion of trained personnel remain in tasks for which they were trained.
  • Evaluation Capacity Development Good practice in developing evaluation capacity also recognizes that: • Monitoring and evaluation systems are a means to an end—benefits are obtained when results are used in decision making. • It is advisable to locate responsibility for monitoring and evaluation near the capable head of an organization. • Monitoring and evaluation systems should not become too complex or resource-intensive. • Monitoring and evaluation systems encompass data collection in the field and aggregation and analysis by end users. • Evaluation capacity development that concentrates on the oversight agency carries the risk that other entities may lack incentives to provide data and information. • Case studies help develop staff competency and confidence.
  • TA 6410–REG: Capacity Development for Monitoring and Evaluation
  • 2. Research and special studies on evaluation capacity development are conducted 3. Knowledge sharing and learning for monitoring and evaluation are boosted Improved ranges of skills, resources, systems, and attitudes for performance― in the evaluation agencies targeted―of results-based monitoring and evaluation of country partnership strategies, sector strategies, policies, programs, and projects in developing member countries of the Greater Mekong Subregion, namely, Cambodia, Lao PDR, and Viet Nam Higher efficiency and effectivenes s in providing public sector services, leading to poverty reduction Impact 1. Proficiency in monitoring and evaluation is raised Outcome Outputs TA 6410–REG: Capacity Development for Monitoring and Evaluation
  • TA 6410–REG: Capacity Development for Monitoring and Evaluation Research and Proficiency in M&E Tools, Methods, and Approaches for M&E Strategy and Policy Formulation for M&E Special Studies for M&E Country Strategies for M&E A Strategy for ECD Key: M&E = monitoring and evaluation, ECD = evaluation capacity development
  • TA 6410–REG: Capacity Development for Monitoring and Evaluation Knowledge Sharing and Learning for M&E Knowledge Sharing and Learning Platforms Knowledge Networks Key: M&E = monitoring and evaluation Partnership Arrangements with Evaluation Associations
  • TA 6410–REG: Capacity Development for Monitoring and Evaluation Assumptions Risks ADB ADB • Technical assistance activities integrate the chief lessons learned from past evaluations • The indicative activities and staffing schedule is too tight to permit productive sequencing of key activities • Appropriate, integrated training programs can be planned, designed, or identified; and synergetic effects can be achieved • Training is conducted well and according to realistic schedules • The consultants and the selected evaluation agency staff coordinate activities effectively • The consultants have client management skills • The consultants and the selected evaluation agency staff maintain clear roles, responsibilities, and deadlines Clients • Evaluation agency staff are available to be trained Output Level: Assumptions and Risks
  • TA 6410–REG: Capacity Development for Monitoring and Evaluation Assumptions Risks Clients Clients • Basic capacity exists and can be mobilized • Evaluation agencies underestimate the importance of national ownership and leadership of the evaluation process and of building national monitoring and evaluation capacities • The role and use of monitoring and evaluation in support of practices of knowledge management are understood • The funding agency has a clear vision about the intended outcome of the technical assistance and how it is to be achieved Outcome Level: Assumptions and Risks
  • TA 6410–REG: Capacity Development for Monitoring and Evaluation Assumptions ADB Risks Clients • Able to plan and support evaluation capacity • Lack of human security; armed conflict; economic development in line with international best policies that discourage pro-poor growth; weak practice scrutiny by the legislative branch of the executive branch; ineffective voice of intended Clients beneficiaries; and corruption, clientelism, or • Provide visible leadership, promote clear patrimonialism do not provide a broadly enabling sense of mission, encourage participation, environment for monitoring and evaluation and establish explicit expectations on • Fragmented government with poor overall performance and rewards capacity; absent, noncredible, and/or rapidly • Strategically approach change management changing policies; unpredictable, unbalanced, or and manage it proactively inflexible funding and staffing; poor public service conditions; segmented and compartmentalized • Involve a critical mass of staff organizations; or insufficient commitment to an • Try, test, and adapt organizational evaluation culture do not conduce to government innovations effectiveness • Celebrate quick wins Impact Level: Assumptions and Risks
  • TA 6410–REG: Capacity Development for Monitoring and Evaluation Regional Cooperation and Poverty Reduction Fund of the People's Republic of China ADB (Shanghai) International Program for Development Evaluation Training Center for Development and Research in Evaluation Asia-Pacific Finance and Development Center Technical Assistance Partnerships
  • TA 6410–REG: Capacity Development for Monitoring and Evaluation Tajikistan • Amirov Fakhriddin K., State Budget Department, Ministry of Finance Viet Nam • Tran Ngoc Lan, Ministry of Planning and Investment • Nguyen Dang Binh, Ministry of Planning and Investment • Pham Thai Linh, Ministry of Natural Resources and Environment • Nguyen Trang Thu, National Academy of Public Administration Lao PDR • Vixay Xaovana, Committee for Planning and Investment • Akhom Praseuth, Bank of Lao PDR • Bounthay Leuangvilay, Budget Department, Ministry of Finance Cambodia • Hou Taing Eng, Ministry of Planning • Im Sour, Cambodian Rehabilitation and Development Board • Suon Sophal, Cambodian Investment Board • Lors Pinit, Department of Investment and Cooperation • Hay Sovuthea, Supreme National Economic Council Malaysia • Arunaselam Rasappan, Center for Development and Research in Evaluation • Mariappan Mahalingam, Center for Development and Research in Evaluation Selected Evaluation Agencies
  • TA 6410–REG: Capacity Development for Monitoring and Evaluation Activity Project Year 1 2 1. Proficiency in M&E is raised. 1.1 Proficiency in tools, methods, and approaches for M&E is raised. (i) Regional training-of-trainers (SHIPDET) (ii) National training-of-trainers 1.2 Proficiency in strategy and policy formulation for M&E is raised. (i) The foundations for M&E in GMS DMCs are strengthened. (ii) GMS DMCs are assisted in setting long-term, consistent strategies. (iii) International training (IPDET) 2. Research and special studies on ECD are conducted. 2.1 Development of country strategies for M&E is supported. 2.2 A strategy for ECD is suggested. 3. Knowledge sharing and learning for M&E are boosted. (i) Selected knowledge sharing and learning platforms are enhanced. (ii) Advice on new and existing knowledge networks on M&E is extended to evaluation agency staff from GMS DMCs. (iii) Partnership arrangements with interested evaluation associations promoted and concluded. Legend: Full-time activity Intermittent activity AFDC = Asia-Pacific Finance and Development Center, CeDRE = Center for Development and Research in Evaluation, IM&E = International Monitoring and Evaluation Specialist, IPDET = International Program for Development Evaluation Training, M&E = monitoring and evaluation Technical Assistance Schedule
  • TA 6410–REG: Capacity Development for Monitoring and Evaluation Training and Capacity Building Research and Special Studies Knowledge Sharing and Networking Strategic Direction for Evaluation Capacity Development Strengthened Evaluation Capacity Improved Service Delivery Leading to Poverty Reduction Expected Short-, Medium-, and Long-Term Deliverables
  • TA 6410–REG: Capacity Development for Monitoring and Evaluation Stage A1 (Apr. 2008) Introductory Training in M&E for Country Trainers (ToT) CeDRE Stage A2 (Apr. –Sept. 2008) In-Country Preparatory Work by Trainees In-Country Stage B1 (June 2008) Policy Level M&E Training (Round 1) IPDET 1 Stage A3 (Oct. 2008) Intermediate M&E Training for Trainees Lao PDR Stage A4 (Nov. 2008–Mar. 2009) Stage A5 (Apr. 2009) In-Country Stage 1 Down-line Training by Trainees In-Country Advanced Level M&E Training for Trainees Cambodia Stage B2 (June 2009) Policy Level M&E Training (Round 2) IPDET 2 Stage A6 (Jul.–Sept. 2009) In-Country Stage 2 Down-line Training by Trainees In-Country Stage A7 (Oct. 2009) Wrap-Up Training & Certification of Country Trainers AFDC/ADB Training Strategy
  • TA 6410–REG: Capacity Development for Monitoring and Evaluation Output 1 had two components: (i) a three-stage training-of -trainers (ToT) program in monitoring and evaluation (M&E) (1–2 weeks per stage), provided by the TA's international consultant to 18 mid-level career officials recommended by ADB's Greater Mekong Subregion coordinators from the three countries covered (6 per country, including one national consultant each); and (ii) a 1-week country-based M&E downline training, provided to 20 key officials selected as M&E focal persons from relevant government agencies in the three DMCs. Most activities under this output were implemented and completed on track. Subsequently, the officials trained under the ToT program successfully conducted M&E downline workshops in their respective countries. Attendance at the Shanghai International Program for Development Evaluation Training (SHIPDET) was found particularly useful. Evaluation of Output 1
  • TA 6410–REG: Capacity Development for Monitoring and Evaluation In synergy, the two components of output 2 circumscribed the M&E situation and clarified training needs in each of the three countries covered. Activities under this output were generally implemented well, even if the final M&E report was delayed. The detailed country case studies provided valuable information on the status, progress, and challenges of M&E. In each of the three countries covered, areas of investigation included (i) institutional structure, policy formulation, and strategy for M&E; and (ii) illustrative approaches to and practical application of M&E policy measures. Evaluation of Output 2
  • TA 6410–REG: Capacity Development for Monitoring and Evaluation The first of output 3’s two components set up an M&E Community of Practice website at www.linkingtoresults.org, which the national consultants engaged under the TA were tasked to promote through knowledge sharing on current M&E practices and issues in the three countries covered. Despite these efforts, the website was used sparingly due to (i) limited broadband access in government offices, especially in Cambodia and Lao, PDR; (ii) insufficient language skills in English, necessary to enable efficient online communications; and (iii) lack of time. The second component aimed to facilitate the emergence of national evaluation networks/societies. In the immediate, regulatory restrictions in the three countries covered delayed this effort; but, Cambodia and Viet Nam may soon move forward as registrations are reportedly being processed by the relevant authorities. Evaluation of Output 3
  • TA 6410–REG: Capacity Development for Monitoring and Evaluation Despite limitations in output 3, the successful delivery of outputs 1 and 2 contributed to the achievement of the TA’s outcome. This was reflected in the fact that the TA helped create, under the ToT program, an internal pool of at least 6 M&E resource persons in each of the three countries covered, who in turn were able to successfully conduct downline M&E training workshops for key officials selected as M&E focal points from concerned government agencies. Feedback from those attending the downline workshops indicated the usefulness of the workshops in building their understanding of M&E and its role in managing for development results (MfDR) as the workshops were customized to national contexts. Achievement of Outcome
  • TA 6410–REG: Capacity Development for Monitoring and Evaluation The overall performance of the TA was rated successful based on the combined ratings of relevant, efficient, effective, and likely sustainable. Most TA activities were generally implemented efficiently and completed on time. This led to good delivery of outputs, except for output 3 where use of the M&E CoP website fell short and national evaluation networks/societies were not immediately established. Notwithstanding, the successful delivery of outputs 1 and output 2 led to effectiveness in achieving outcome in terms of improved ranges of M&E skills and awareness of the importance role of M&E. In terms of design, the TA's ToT program was found relevant: it enabled the key officials trained as trainers to provide downline M&E training workshops to relevant staff of concerned agencies in their respective countries. This is expected to help sustain the outcome achieved. Overall Assessment and Rating
  • TA 6410–REG: Capacity Development for Monitoring and Evaluation Five lessons stand out: (i) baseline situational analysis of M&E status and needs helped design relevant training courses; (ii) the training courses, which began with a ToT program, helped create pools of trainers able to deliver country-based M&E downline training workshops to staff in concerned agencies in the three countries covered, which ought to sustain and internalize the M&E knowledge and capacity gained; (iii) there should be no rush to design outputs susceptible to in-country obstacles, e.g., low levels of IT advancement, tight regulatory environments, unless they can be addressed first; (iv) the TA should have worked more closely with ADB's resident missions, which could have provided focal points in support of TA implementation; and (v) a follow-on experience sharing workshop should have been planned to estimate the magnitude of "actual" applications of the M&E knowledge and capacity gained and to gauge retention, on M&E-related tasks, of the officials trained. Major Lessons
  • Further Reading • ADB. 2007. Independent Evaluation at the Asian Development Bank. Manila. Available: www.adb.org/publications/independent-evaluation-asiandevelopment-bank • ADB. 2007. Regional Technical Assistance for Capacity Development for Monitoring and Evaluation. Manila. Available: www.adb.org/projects/documents/capacitydevelopment-monitoring-and-evaluation-0 • ADB. 2009. Learning from Evaluation. Manila. Available: www.adb.org/publications/learning-evaluation
  • Further Reading • ADB. 2009. Regional Technical Assistance for Capacity Development for Capacity Development in Results-Based Monitoring and Evaluation for Countries under the Asian Development Bank's Central Asia Regional Economic Cooperation Program. Manila. Available: www.adb.org/projects/documents/capacity-developmentresults-based-monitoring-and-evaluation-countries-underasia • ADB. 2011. Technical Assistance Completion Report: Regional Technical Assistance for Capacity Development for Monitoring and Evaluation. Manila. Available: www.adb.org/projects/documents/capacity-developmentmonitoring-and-evaluation
  • Further Reading • Britton, B. and Serrat, O. 2013. Learning from Evaluation. Available: www.slideshare.net/celcius233/learning-fromevaluation
  • Olivier Serrat Principal Knowledge Management Specialist Regional and Sustainable Development Department Asian Development Bank knowledge@adb.org www.adb.org/knowledge-management www.facebook.com/adbknowledgesolutions www.scribd.com/knowledge_solutions www.twitter.com/adbknowledge