Models and approaches to decentralized evaluation system - considerations for CGIAR
ā¢Download as PPTX, PDFā¢
1 likeā¢798 views
Presented at the 2015 CGIAR Evaluation Community of Practice meeitng. CGIAR is moving towards a coordinated evaluation system to comprehensively cover the programs, insitutions, and activities. The presentation offers examples of decentralized evaluaitons as approached by other agencies, and aspects for CGIAR to consider.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Models and approaches to decentralized evaluation system - considerations for CGIAR
Similar to Models and approaches to decentralized evaluation system - considerations for CGIAR (20)
More from Independent Evaluation Arrangement of CGIAR
More from Independent Evaluation Arrangement of CGIAR (10)
Recently uploaded
Recently uploaded (20)
Models and approaches to decentralized evaluation system - considerations for CGIAR
- 1. Decentralized Evaluations Facilitated by Keith Child kchild@outlook.com
- 2. Where We Are ā¢ Mixed performance on CCEEs ā¢ Coincidental rather than planned linkage between IEA evaluations and CCEEs ā¢ Need for a transparent and searchable database for evaluations ā¢ Mixed levels of support requested from the IEA ā¢ Mixed implementation of existing guidance ā¢ Commissioned by those who designed/mange the CRP (in some cases) ā¢ Absence of rigorous quality control mechanism (credibility) ā¢ Lack of consistent management response and follow-up ā¢ Mixed capacity of M&E staff and/or limited time allocation for evaluation ā¢ Lack of evaluation/KPIs for evaluation function ā¢ Mixed resourcing for CCEEs
- 3. Target Outcomes Though Upgraded Use of Decentralized Evaluations ā¢ Greater emphasis on organizational learning, so that evaluations serve the diverse needs and priorities of CRPs and Lead Centers ā¢ An increased commitment to a corporate culture of accountability for achieving results and using resources efficiently ā¢ A more integrated and coherent evaluation system with strong linkages across the evaluation function ā¢ A renewed commitment to use evaluation as a means to improving impact
- 4. Who is decentralized? ā¢ UNICEF ā¢ UN Women ā¢ ILO ā¢ UDP ā¢ WFP ā¢ UN Habitat ā¢ ILO ā¢ UNDP ā¢ UNEP ā¢ DFID ā¢ USAID ā¢ Gates
- 5. What is a Decentralized Evaluation System? ā¢ Evaluation is the independent, systematic and objective assessment of an on-going or completed project, program, institution, policy or modality, its design, implementation and results. It determines the relevance and fulfilment of objectives, development efficiency, quality, effectiveness, impact and sustainability. ā¢ Decentralized evaluation is under the direct management of CRPs (roles and responsibilities more generally fall to the CRP)
- 6. Pros and Cons Potential Advantages ā¢ Actually reflects the decentralized nature of CGIAR ā¢ Evidence generated is relevant to the local context (fit for purpose) ā¢ Decentralized learning ā¢ More flexibility? Potential Disadvantages ā¢ Disagreement on roles and responsibilities ā¢ Higher transaction costs to manage a complex, hybrid system (e.g., capacity building) ā¢ Independence and credibility may be threatened ā¢ Overlap/duplication of efforts/ lack of corporate knowledge/ use of evidence
- 7. Hybrid System IEA: conducts independent evaluations (IEEs), set standards, guidelines and to provide guidance across the CGIAR, CRPs: primarily responsible for facilitating IEEs, conducting CCEEs and occasional join project evaluations according the standards and guidelines set by the IEA. IEA/CRP: 5 Review and Validation Studies (CCEE ++), which became CRP commissioned full evaluations
- 8. Recurring themes 1. Independence and impartiality 2. Quality assurance 3. Professionalization/capacity building 4. Monitoring the evaluation function 5. Integration and coherence of the evaluation function
- 9. 1. Independence and Impartiality When do we expect impartiality? All stages of evaluation process ā¢ planning of evaluation ā¢ selection of evaluation teams ā¢ conduct of the evaluation ā¢ formulation of findings and recommendations and recommendations
- 10. Possible Mitigation Measures 1.Submission of the draft report to the commissioning unit and the IEA simultaneously 2.āAudit trailā to track comments and responses by the evaluation team 3.Emphasize use of the Management Response to provide clarifications or express differences of opinion 4.Scale up M&E staff at the country and regional level 5.Update guidelines for decentralized evaluations 6.Communication / training to increase awareness
- 11. 2. Quality Assurance System Quality Assurance AssessmentQuality Assurance Mechanisms ā¢ During evaluation ā¢ Guidelines for CCEEs ā¢ Criteria for Peer Review (inception and draft reports) ā¢ Quality Assessment Oversight groups (ERG, Management group) ā¢ Real time support ā¢ Checklist ā¢ Concise assessment of quality after evaluation ā¢ Report on overall quality of evaluations reports ā¢ Strengthen internal evaluation capacity (during and after) ā¢ Contribute to organizational learning by identifying evaluations to be used in meta-analysis ā¢ Quality Assessment integrated into evaluation database
- 12. Internal Review Planning Designing Contracting Start Up Collecting Information Analysis Reporting Preparatory Phase Inception Phase Inquiry Phase Reporting Phase Evaluationā Output and QA TORS QA feedback Selection of team QA advice Inception report QA internal and external review QA feedback Draft report QA internal and external review QA internal and external review Final report QA validation
- 13. Quality Assessment Evaluation Quality Assessment Grid (UNFPA) ā¢ Simple, 8 criteria (based on 8 UNEG adapted standards) ā¢ Conducted by internally by EO ā¢ Low cost ā¢ 4 scoring levels Global Evaluation Reports Oversight System (UNICEF) ā¢ Complex (58 criteria, subdivided by 8 UNEG adapted standards) ā¢ Conducted by external reviewer/ consulting company ā¢ Results are sent to commissioning body to improve future evaluations ā¢ Rigour allows for pre-selection for meta evaluation report ā¢ High cost MIS for Evaluation Reporting: ā¢ ILO iTrack ā¢ UNICEF Evaluation database ā¢ UN Women Gate ā¢ UNFPA Evaluation Database
- 14. 3. Professionalization of Evaluation Staff ā¢ Moving beyond focal points ā¢ Capacity building ā¢ Mentoring ā¢ Certification/training ā¢ Sharing staff more effectively ā¢ Moving beyond the evaluation function alone
- 15. 4. M&E of the Evaluation Function Monitoring Possible KPIs include outputs and outcomes: 1. Human resources for M&E 2. Financial resources invested in evaluation 3. Evaluation coverage 4. Evaluation implementation rate 5. Quality of evaluation reports 6. Evaluation Reports with Management Response uploaded to IEA database 7. Number of staff that have completed certification programme 8. Percentage of CRPs that managed evaluation in a specific year compliant with QA standards Evaluation ā¢ Periodic evaluation of the evaluation function (3-5 years?) ā¢ Evaluating the evaluators ā¢ Sharing good practice and experience, stimulates change to achieve goals and standards ā¢ Criteria for evaluation: independence, credibility and utility ā¢ Can be aimed at either centralized or decentralized evaluation = flexible to achieve best fit ā¢ See: DAC/UNEG, Framework on Professional Peer Reviews of the Evaluation Function
- 16. 5. Integrated and Coherent Decentralized Evaluation System ā¢ Coherent within CRP and across CGIAR ā¢ CCEEs are building blocks for external evaluations, important source for internal learning for CRPs ā¢ Publication of evaluation schedule ā¢ Guidance for full CRP 2 Proposal calls for 5 year CCEE plan ā¢ Joint evaluations? ā¢ Thematic or Geographic focus ā¢ IEA co-ordination ā¢ What would an effective evaluation synergy tool look like? How would it function?
- 17. Questions?
- 18. Topics for Discussion 1. Strengthening the capacity to manage and conduct decentralized evaluations through the professionalization of the evaluation function across the CGIAR 2. Quality Assurance and Quality Control System to ensure that evaluations meet basic quality criteria 3. Monitoring the performance of the evaluation function ā KPIs/MIS (particular focus on decentralized evaluations) 4. Ensuring that evaluations are impartial and how to assess impartiality as a component of a QA system 5. Strengthened coordination of evaluations within a decentralized evaluation system 6. Wild Jack and Jill ā¦ everything these
- 19. Group Work 1. Based on the presentation and your own experience, discuss in groups of equal size one of the suggested focus areas. 2. Nominate in each group: 1) a rapporteur to present the results of the discussion and 2) a note taker to document it. 3. Include āconsensusā and āoutlierā ideas, and categorize them as according to who should take responsibility: ECOP members, IEA, Other Purpose: to develop a list of actionable recommendations that can feed into a larger discussion on how to strengthen the culture of decentralized evaluations in order to enhance their quality and usefulness.
- 20. Next Steps 1) Needs Assessment ā¢ I have developed a needs assessment tool. Please return it to me (kchild@outlook.com) by 30 November. 2) Advisory Working Group ā¢ The IEA would like to set up an Advisory Working Group to formalize a list of recommendations on how to strengthen the culture of decentralized evaluations in the CGIAR. Coordination of evaluation plans leading into second phase of proposals
Editor's Notes
- Introduce yourself Introduce your topic: decentralized evaluations, by which in our system means primarily CCEEs
- Interesting to know from centers on their level of guidance (faithfully have you followed guidance notes)
- Moving toward a more coordinated, cost effective evaluation regime some of the desired outcomes we are hoping to achieve are: Second Call for proposals requires a 300,000 per year budget for CCEEs
- Fortunately we are not alone.
- An evaluation is ā¦ (read from slide) So what is a decentralized evaluation? To the WFP and others, a decentralized evaluation is simply and evaluation commissioned by someone other than the Evaluation Office. In our case, a decentralized evaluation simply refers to an evaluation under the management of a CRP.
- What we have at the moment in the IEA is a sort of hybrid system. Interpret Graph It is important to first try to understand the organization and how the evaluation function can best serve it prior to looking in depth at the machinery. conducting independent evaluations, set standards, guidelines and to provide guidance across the CGIAR, whereas CRPs are primarily responsible for conducting facilitating IEEs, conducting CCEEs and occasional join project evaluations according the standards and guidelines set by the IEA. 5 RVS review and validation study became full IEA evaluation CCEE++ No figure pointing
- However, we face a constant danger of eroding impartiality: Consultants who only conduct evaluations for the CGIAR Pressure on consultants to āsoftenā findings and conclusions Reports are ācorrectedā by CRP staff Other challenges? Mitigation Measures: Submission of the draft report to the commissioning unit and the IEA simultaneously āAudit trailā to track comments and responses by the evaluation team Emphasize use of the Management Response to provide clarifications or express differences of opinion Scale up M&E staff at the country and regional level Update guidelines for decentralized evaluations Communication / training to increase awareness
- G
- ILO iTrack UNICEF Evaluation database UN Women Gate UNFPA Evaluation Database Once the reports are assessed, detailed feedback on how to improve future evaluations is sent to the office that commissioned the evaluation.Ā Meta-evaluationĀ reports (2013, 2012, 2011, 2010) are produced, disseminated and used to improve UNICEF decentralized evaluation function. As indicated in the EQA grid, the scoring scale comprises four levels: (1) unsatisfactory, (2) poor, (3) good, (4) very good. Annex to all ToRs IEA quality assessments RVS 5 quality assemssments Roles and responsibilities
- Moving beyond focal points ā making fp effective, avoiding ambiguity over role (multiple focal points), sufficient time allocation, etc. Neither accountability or learing can be achieved soley through the evaluation function. Each requires internal actions by senior Capacity building Mentoring Certification Hiring more M&E staff/ sharing staff more effectively
- This is not the same, but is closely related to building a quality assessment system because its often reported on the same MIS. A peer evaluation of the evaluation function is never implies a punitive decision or sanction, but in our case might help to strengthen a renewed decentralized evaluation system.
- Half of budgeted CRP activities should be covered by CCEEs in a CRP cycle (5 years) Livestock and Fish want to do a CCEE next year on Feeds, but they have no money. Tool building IEA can provide Needs assessment: is this something that they want help with? What are the criteria for selecting topics for CCEEs vs IEEs
- A lot of these ideas require formal changes in the way that a decentralized system is build. Often however, the problem is not so much with the system, as its implementation. This is to say that many of the basics to building an effective decentralized system my already be in place, but that we simply need to implement them more effectively or in their entirety.
- Including coherence IEA Role, their role/ other Discussion in plenary
- Reconvene to discuss