Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Decentralized Evaluations
Facilitated by Keith Child
kchild@outlook.com
Where We Are
• Mixed performance on CCEEs
• Coincidental rather than planned linkage between IEA evaluations and
CCEEs
• N...
Target Outcomes Though Upgraded Use of
Decentralized Evaluations
• Greater emphasis on organizational learning, so that ev...
Who is decentralized?
• UNICEF
• UN Women
• ILO
• UDP
• WFP
• UN Habitat
• ILO
• UNDP
• UNEP
• DFID
• USAID
• Gates
What is a Decentralized Evaluation System?
• Evaluation is the independent, systematic and
objective assessment of an on-g...
Pros and Cons
Potential Advantages
• Actually reflects the
decentralized nature of CGIAR
• Evidence generated is relevant
...
Hybrid
System
IEA: conducts independent
evaluations (IEEs), set standards,
guidelines and to provide
guidance across the C...
Recurring themes
1. Independence and impartiality
2. Quality assurance
3. Professionalization/capacity building
4. Monitor...
1. Independence and Impartiality
When do we expect impartiality?
All stages of evaluation process
• planning of evaluation...
Possible Mitigation Measures
1.Submission of the draft report to the commissioning unit and the IEA
simultaneously
2.“Audi...
2. Quality Assurance System
Quality Assurance AssessmentQuality Assurance Mechanisms
• During evaluation
• Guidelines for ...
Internal Review
Planning Designing Contracting Start Up Collecting
Information
Analysis Reporting
Preparatory Phase Incept...
Quality Assessment
Evaluation Quality Assessment Grid
(UNFPA)
• Simple, 8 criteria (based on 8 UNEG
adapted standards)
• C...
3. Professionalization of Evaluation Staff
• Moving beyond focal points
• Capacity building
• Mentoring
• Certification/tr...
4. M&E of the Evaluation Function
Monitoring
Possible KPIs include outputs and outcomes:
1. Human resources for M&E
2. Fin...
5. Integrated and Coherent Decentralized
Evaluation System
• Coherent within CRP and across CGIAR
• CCEEs are building blo...
Questions?
Topics for Discussion
1. Strengthening the capacity to manage and conduct decentralized
evaluations through the profession...
Group Work
1. Based on the presentation and your own experience, discuss in
groups of equal size one of the suggested focu...
Next Steps
1) Needs Assessment
• I have developed a needs assessment tool. Please return it to me
(kchild@outlook.com) by ...
Upcoming SlideShare
Loading in …5
×

Models and approaches to decentralized evaluation system - considerations for CGIAR

505 views

Published on

Presented at the 2015 CGIAR Evaluation Community of Practice meeitng. CGIAR is moving towards a coordinated evaluation system to comprehensively cover the programs, insitutions, and activities. The presentation offers examples of decentralized evaluaitons as approached by other agencies, and aspects for CGIAR to consider.

Published in: Government & Nonprofit
  • Be the first to comment

  • Be the first to like this

Models and approaches to decentralized evaluation system - considerations for CGIAR

  1. 1. Decentralized Evaluations Facilitated by Keith Child kchild@outlook.com
  2. 2. Where We Are • Mixed performance on CCEEs • Coincidental rather than planned linkage between IEA evaluations and CCEEs • Need for a transparent and searchable database for evaluations • Mixed levels of support requested from the IEA • Mixed implementation of existing guidance • Commissioned by those who designed/mange the CRP (in some cases) • Absence of rigorous quality control mechanism (credibility) • Lack of consistent management response and follow-up • Mixed capacity of M&E staff and/or limited time allocation for evaluation • Lack of evaluation/KPIs for evaluation function • Mixed resourcing for CCEEs
  3. 3. Target Outcomes Though Upgraded Use of Decentralized Evaluations • Greater emphasis on organizational learning, so that evaluations serve the diverse needs and priorities of CRPs and Lead Centers • An increased commitment to a corporate culture of accountability for achieving results and using resources efficiently • A more integrated and coherent evaluation system with strong linkages across the evaluation function • A renewed commitment to use evaluation as a means to improving impact
  4. 4. Who is decentralized? • UNICEF • UN Women • ILO • UDP • WFP • UN Habitat • ILO • UNDP • UNEP • DFID • USAID • Gates
  5. 5. What is a Decentralized Evaluation System? • Evaluation is the independent, systematic and objective assessment of an on-going or completed project, program, institution, policy or modality, its design, implementation and results. It determines the relevance and fulfilment of objectives, development efficiency, quality, effectiveness, impact and sustainability. • Decentralized evaluation is under the direct management of CRPs (roles and responsibilities more generally fall to the CRP)
  6. 6. Pros and Cons Potential Advantages • Actually reflects the decentralized nature of CGIAR • Evidence generated is relevant to the local context (fit for purpose) • Decentralized learning • More flexibility? Potential Disadvantages • Disagreement on roles and responsibilities • Higher transaction costs to manage a complex, hybrid system (e.g., capacity building) • Independence and credibility may be threatened • Overlap/duplication of efforts/ lack of corporate knowledge/ use of evidence
  7. 7. Hybrid System IEA: conducts independent evaluations (IEEs), set standards, guidelines and to provide guidance across the CGIAR, CRPs: primarily responsible for facilitating IEEs, conducting CCEEs and occasional join project evaluations according the standards and guidelines set by the IEA. IEA/CRP: 5 Review and Validation Studies (CCEE ++), which became CRP commissioned full evaluations
  8. 8. Recurring themes 1. Independence and impartiality 2. Quality assurance 3. Professionalization/capacity building 4. Monitoring the evaluation function 5. Integration and coherence of the evaluation function
  9. 9. 1. Independence and Impartiality When do we expect impartiality? All stages of evaluation process • planning of evaluation • selection of evaluation teams • conduct of the evaluation • formulation of findings and recommendations and recommendations
  10. 10. Possible Mitigation Measures 1.Submission of the draft report to the commissioning unit and the IEA simultaneously 2.“Audit trail” to track comments and responses by the evaluation team 3.Emphasize use of the Management Response to provide clarifications or express differences of opinion 4.Scale up M&E staff at the country and regional level 5.Update guidelines for decentralized evaluations 6.Communication / training to increase awareness
  11. 11. 2. Quality Assurance System Quality Assurance AssessmentQuality Assurance Mechanisms • During evaluation • Guidelines for CCEEs • Criteria for Peer Review (inception and draft reports) • Quality Assessment Oversight groups (ERG, Management group) • Real time support • Checklist • Concise assessment of quality after evaluation • Report on overall quality of evaluations reports • Strengthen internal evaluation capacity (during and after) • Contribute to organizational learning by identifying evaluations to be used in meta-analysis • Quality Assessment integrated into evaluation database
  12. 12. Internal Review Planning Designing Contracting Start Up Collecting Information Analysis Reporting Preparatory Phase Inception Phase Inquiry Phase Reporting Phase Evaluation– Output and QA TORS QA feedback Selection of team QA advice Inception report QA internal and external review QA feedback Draft report QA internal and external review QA internal and external review Final report QA validation
  13. 13. Quality Assessment Evaluation Quality Assessment Grid (UNFPA) • Simple, 8 criteria (based on 8 UNEG adapted standards) • Conducted by internally by EO • Low cost • 4 scoring levels Global Evaluation Reports Oversight System (UNICEF) • Complex (58 criteria, subdivided by 8 UNEG adapted standards) • Conducted by external reviewer/ consulting company • Results are sent to commissioning body to improve future evaluations • Rigour allows for pre-selection for meta evaluation report • High cost MIS for Evaluation Reporting: • ILO iTrack • UNICEF Evaluation database • UN Women Gate • UNFPA Evaluation Database
  14. 14. 3. Professionalization of Evaluation Staff • Moving beyond focal points • Capacity building • Mentoring • Certification/training • Sharing staff more effectively • Moving beyond the evaluation function alone
  15. 15. 4. M&E of the Evaluation Function Monitoring Possible KPIs include outputs and outcomes: 1. Human resources for M&E 2. Financial resources invested in evaluation 3. Evaluation coverage 4. Evaluation implementation rate 5. Quality of evaluation reports 6. Evaluation Reports with Management Response uploaded to IEA database 7. Number of staff that have completed certification programme 8. Percentage of CRPs that managed evaluation in a specific year compliant with QA standards Evaluation • Periodic evaluation of the evaluation function (3-5 years?) • Evaluating the evaluators • Sharing good practice and experience, stimulates change to achieve goals and standards • Criteria for evaluation: independence, credibility and utility • Can be aimed at either centralized or decentralized evaluation = flexible to achieve best fit • See: DAC/UNEG, Framework on Professional Peer Reviews of the Evaluation Function
  16. 16. 5. Integrated and Coherent Decentralized Evaluation System • Coherent within CRP and across CGIAR • CCEEs are building blocks for external evaluations, important source for internal learning for CRPs • Publication of evaluation schedule • Guidance for full CRP 2 Proposal calls for 5 year CCEE plan • Joint evaluations? • Thematic or Geographic focus • IEA co-ordination • What would an effective evaluation synergy tool look like? How would it function?
  17. 17. Questions?
  18. 18. Topics for Discussion 1. Strengthening the capacity to manage and conduct decentralized evaluations through the professionalization of the evaluation function across the CGIAR 2. Quality Assurance and Quality Control System to ensure that evaluations meet basic quality criteria 3. Monitoring the performance of the evaluation function – KPIs/MIS (particular focus on decentralized evaluations) 4. Ensuring that evaluations are impartial and how to assess impartiality as a component of a QA system 5. Strengthened coordination of evaluations within a decentralized evaluation system 6. Wild Jack and Jill … everything these
  19. 19. Group Work 1. Based on the presentation and your own experience, discuss in groups of equal size one of the suggested focus areas. 2. Nominate in each group: 1) a rapporteur to present the results of the discussion and 2) a note taker to document it. 3. Include “consensus” and “outlier” ideas, and categorize them as according to who should take responsibility: ECOP members, IEA, Other Purpose: to develop a list of actionable recommendations that can feed into a larger discussion on how to strengthen the culture of decentralized evaluations in order to enhance their quality and usefulness.
  20. 20. Next Steps 1) Needs Assessment • I have developed a needs assessment tool. Please return it to me (kchild@outlook.com) by 30 November. 2) Advisory Working Group • The IEA would like to set up an Advisory Working Group to formalize a list of recommendations on how to strengthen the culture of decentralized evaluations in the CGIAR. Coordination of evaluation plans leading into second phase of proposals

×