Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
SIAC Phase 1Evaluation
**Some** **preliminary** findings and recommendations for discussion
ISPC, 15 Sept 2016
Julia Compt...
SIAC: Strengthening Impact Assessment in the CGIAR
• Initially a three year project (2013-16) extended to 2017
• Funders: ...
SIAC has dramatically increased central resources for IA
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
2007 2008 2009 2010 2011 ...
Context: CRP Phase 2 IA budgets
Source: CRP proposals, compiled by SPIA
0.5%
1.7%
1.5%
1.1%
2.3%
3.9%
3.0%
3.2%
3.1%
2.5%
...
Four main SIAC activity areas
1 (Methods): Pilot and verify innovative methods for diffusion data
2 (Outcomes): Institutio...
ISPC
SPIA
SIAC
Funded projects
• CGIAR centers
• External incl.
MSU)
1
Independent experts
• For external
reviews,
consult...
Introduction to this SIAC evaluation
• Financed by SIAC and managed for (independence) by IEA
Objectives:
• Accountability...
Methods
• Limited time/ timing (most data collection in August).
• Analysis of project and other documentation
• Semi-stru...
Questions addressed
(in the order of this presentation)
Effectiveness (1) Outputs: Has SIAC produced its planned outputs?
...
Effectiveness (1): SIAC Outputs
- A very ambitious work programme, with many subactivities
- Most outputs (under project c...
Outputs continued
KEY
Completed
On track
Some setbacks
Discontinued
New or modified
Too early to tell
3. Assess the full r...
Some key outputs
Methods: DNA fingerprinting for varieties;
policy workshop with PIM; NRM pilots
Adoption data: central da...
Percentage of cost of SIAC studies by area of work
Varietal adoption
43%
Crop/Natural
Resource
Management
34%
Policy
6%
Ex...
Quality of science
• SIAC process (for most): competitive calls, workshops, peer review of
proposals, approval by Project ...
Effectiveness (2): SIAC Outcomes
Some progress, but could be stronger
• Methods have promise for broad uptake, eg DNA fing...
Organisation of Phase 1 has favoured productivity and high
academic quality with limited resources
• Outsourcing managemen...
Outputs after 5 years Outcomes Impacts
Systematic collection
of adoption data
institutionalized
Greater range of
types of ...
Outputs after 5 years Outcomes Impacts
Systematic collection
of adoption data
institutionalized
Greater range of
types of ...
Relevance
SPIA’s mandate (reflects demand) :
To (1) provide CGIAR members with timely, objective and credible
information ...
SPIA Mandate SIAC Relevance
1. Timely and credible
information on the
impacts at the system
level...
Broadly relevant, but...
Management – efficient, within constraints
• Understaffed for ambitious workplan - leading to outsourcing major
parts of t...
Governance - reconsider for Phase 2
PSC composition: SPIA, core donors, IA expert, Consortium Office
(observer) and IEA (o...
Preliminary recommendations (partial)
1. Revisit the SIAC Theory of Change...
2. More systematic use of reviews and “evide...
Thank you
Upcoming SlideShare
Loading in …5
×

Julia Compton SIAC Phase1 Evaluation

319 views

Published on

ISPC14

Published in: Science
  • Be the first to comment

  • Be the first to like this

Julia Compton SIAC Phase1 Evaluation

  1. 1. SIAC Phase 1Evaluation **Some** **preliminary** findings and recommendations for discussion ISPC, 15 Sept 2016 Julia Compton (independent), Tim Dalton (Kansas State University) and Sophie Zimm (IEA)
  2. 2. SIAC: Strengthening Impact Assessment in the CGIAR • Initially a three year project (2013-16) extended to 2017 • Funders: BMGF, and Window 1 (mostly DFID top-up funding) Two “top-level indicators of success”: a) “An expansion of the available set of impact studies, providing useful and credible information to guide future investments in the CGIAR.” b) “CRPs and Centers of the CGIAR have institutionalized impact assessment such that ex post impact assessment is regarded as an essential part of prudent research management for accountability purposes and as an input to ex ante strategic planning.”
  3. 3. SIAC has dramatically increased central resources for IA 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 AnnualSPIAbudget(US$million) Year SIAC W1 SIAC BMGF DIIVA study ISPC SIAC Phase 1 Source: SPIA data, with thanks to Ira Vater
  4. 4. Context: CRP Phase 2 IA budgets Source: CRP proposals, compiled by SPIA 0.5% 1.7% 1.5% 1.1% 2.3% 3.9% 3.0% 3.2% 3.1% 2.5% 4.4% 5.3% 0 1 2 3 4 5 6 CCAFS Climate Change Livestock Water Land Ecosystems Legumes/Dryland Wheat Fish Maize FTA Forests Agroforestry GRiSP Rice SIAC Roots Tubers Bananas A4NH Nutrition Health PIM Policies Inst Markets Average annual budget (US$ M) for Impact Assessment (labels = % of CRP total budget)
  5. 5. Four main SIAC activity areas 1 (Methods): Pilot and verify innovative methods for diffusion data 2 (Outcomes): Institutionalize the collection of diffusion data 3 (Impacts): Assess the full range of impacts from CGIAR research 4 (Building a community of practice): for epIA within the CGIAR and between the CGIAR and the development community more broadly
  6. 6. ISPC SPIA SIAC Funded projects • CGIAR centers • External incl. MSU) 1 Independent experts • For external reviews, consultancies 2 IA focal points (IAFPs) • one per CGIAR center 3 PSC SPIA chair (chair) SPIA Secretariat (1 staff) BMGF, DFID External expert (UC Davis) CO and IEA member (observer) CGIAR Fund Council EIAC FAO BMGF Administrative reporting Fund Use agreement Legend intellectual exchange Funding Project and financial reporting W1 money SIAC Organigram (2013- June 2016, before latest reforms) ISPC Chair and Exec DirectorCGIAR CO
  7. 7. Introduction to this SIAC evaluation • Financed by SIAC and managed for (independence) by IEA Objectives: • Accountability to SIAC funders (BMGF, DFID, System Council) • Learning wider lessons for SPIA, and improving the design of SIAC Phase II (SIAC Phase II proposal going to System Council in 2017) Scope of evaluation: SIAC, not SPIA. IEA evaluation of ISPC/SPIA planned 2017 (However, overlap since SIAC covers whole SPIA budget and workplan.)
  8. 8. Methods • Limited time/ timing (most data collection in August). • Analysis of project and other documentation • Semi-structured interviews /group discussions with >60 people • SPIA, ISPC, Fund Office, System Office • 11 CGIAR research leaders (CRP leaders/DGs/DDG-Rs) • 21 CGIAR Impact Assessment Focal Points • 5 donors - BMGF, DFID, USAID, ACIAR and Germany-GIZ • External IA experts who know SPIA/SIAC • CLEAR and 3iE
  9. 9. Questions addressed (in the order of this presentation) Effectiveness (1) Outputs: Has SIAC produced its planned outputs? Quality of Science: Is SIAC promoting high quality work and cutting edge methods? Effectiveness (2) Outcomes: Is SIAC on track for planned outcomes? Relevance : Does SIAC respond to need, demand and SPIA’s mandate? SIAC management and governance: Does this promote efficiency, accountability and transparency?
  10. 10. Effectiveness (1): SIAC Outputs - A very ambitious work programme, with many subactivities - Most outputs (under project control) achieved. - Evidence of reflection, learning and adjustments. KEY Completed On track Some setbacks Discontinued New or modified Too early to tell 1: Develop, pilot and verify innovative methods for diffusion data 1.1 Methods for crop varieties 1.2 Protocols for diffusion of NRM technologies 1.3 New institutional approaches for diffusion data 1.4 Disseminate best practices learned 2. Institutionalize the collection of the diffusion data needed for IA 2.1 Institutionalise collection of adoption data 2.2 Collect and validate NRM claims Organise (and institutionalise) POR results Institutionalise collection of adoption data
  11. 11. Outputs continued KEY Completed On track Some setbacks Discontinued New or modified Too early to tell 3. Assess the full range of impacts from CGIAR research 3.0 IA on nutrition and health 3.1 Long term/large scale epIAs 3.2 Experimental/quasi exp studies 3.3. Under-evaluated areas 3.4 Meta analyses at system level 4. Support Community of Practice for epIA with CGIAR/partners 4.1 Small grants 4.2 Capdev of IA in the CGIAR 4.3 Biennial conference on epIA 4.4 Quality star rating for CGIAR IA studies ex-4.5 Support RROs and NARs 4.5 SPIA website (but not yet one stop shop for CGIAR IA) 4.7 Support and capdev to Consortium At least three studies by post docs
  12. 12. Some key outputs Methods: DNA fingerprinting for varieties; policy workshop with PIM; NRM pilots Adoption data: central database of 12 crops and 17 countries with 130 crop x sub- region combinations CGIAR policy outcome database Capacity development: Center/Uni partnerships, training, IAFP meetings Evidence of CGIAR impact: Reviews of water management, livestock. Many other studies under way. Survey enumerators practice taking leaf samples for genotyping (Photo: SPIA)
  13. 13. Percentage of cost of SIAC studies by area of work Varietal adoption 43% Crop/Natural Resource Management 34% Policy 6% Extension/ Information 4% Nutrition/health 9% Livestock 4% Source: evaluation team analysis of approved research proposals
  14. 14. Quality of science • SIAC process (for most): competitive calls, workshops, peer review of proposals, approval by Project Steering Committee • We analysed processes and (accepted) proposals Findings: • Most proposals are good-to-high quality • Cutting edge methods in some areas (not always the objective) • Some proposals do not clearly specify testable hypotheses • One or two cases of poor research design: e.g. sample too small to test hypothesis • Few spell out plans to assess distribution of adoption/benefits by gender or by social groups (e.g. wealth/ethnic/caste) • Few contain human subjects statements (ethics) • Most calls now specify open data, but not all proposals are clear on plans • Calls managed by different people – consistency in stated requirements is increasing
  15. 15. Effectiveness (2): SIAC Outcomes Some progress, but could be stronger • Methods have promise for broad uptake, eg DNA fingerprinting • Slowly building a picture of CGIAR impacts, but could be more focused • Institutionalisation is more challenging (e.g. of adoption studies, or of links to prioritisation). Two main areas for improvement identified: a) Organisation of SIAC research in Phase 1 b) Theory of change of SIAC
  16. 16. Organisation of Phase 1 has favoured productivity and high academic quality with limited resources • Outsourcing management of key areas to trusted academics • Competitive calls. Most calls broad in topic with proposals chosen for high academic quality; opportunities to leverage other initiatives - Experimental studies - NRM impacts - Nutritional impacts - Testing large scale outcome claims by CGIAR • Lack of systematic gap analysis and linkages to key indicators (IDOs) Result: Many high-quality studies on the way, but also fragmentation and potential loss of opportunities for institutionalisation and learning
  17. 17. Outputs after 5 years Outcomes Impacts Systematic collection of adoption data institutionalized Greater range of types of research and types of impact covered by impact assessments Greater rigor and scale of CGIAR adoption estimates and impact assessments Greater transparency about CGIAR impacts Greater confidence of donors in CGIAR system More evidence- based decision- making within the CGIAR CGIAR system becomes more effective at reaching its System-Level Outcomes Increased number of impact assessments of CGIAR researchSIAC LOGIC MODEL
  18. 18. Outputs after 5 years Outcomes Impacts Systematic collection of adoption data institutionalized Greater range of types of research and types of impact covered by impact assessments Greater rigor and scale of CGIAR adoption estimates and impact assessments Greater transparency about CGIAR impacts Greater confidence of donors in CGIAR system More evidence- based decision- making within the CGIAR CGIAR system becomes more effective at reaching its System-Level Outcomes Increased number of impact assessments of CGIAR research Selection, design and management of SIAC activities Does not specify activities, outputs, links, assumptions and risks explicitly and in a testable way
  19. 19. Relevance SPIA’s mandate (reflects demand) : To (1) provide CGIAR members with timely, objective and credible information on the impacts at the system level... To (2) provide support to and (3) complement the Centers in their ex post impact assessment activities... To (4) provide feedback to CGIAR priority setting and ... links to ex ante assessment... Source: SPIA website http://impact.cgiar.org/about 7/9/16
  20. 20. SPIA Mandate SIAC Relevance 1. Timely and credible information on the impacts at the system level... Broadly relevant, but could be better focused. • More focus on SLOs/IDOs • More gap analysis • More consultation on priorities 2. Complement the Centers in their epIA activities SIAC has moved beyond epIA - potential overlap with Center research – needs to be clearer on its role • Increase consultation on priorities with Centers/CRPs • IPSC/SPIA evaluation should address wider issue of who does what 3. Support to the Centers in epIA Broadly relevant, but could be better focused - needs further analysis of comparative advantage • Demand for cross-cutting methods (working with expert CRPs), collection and centralisation of data, (guidance, quality assurance) • Concerns about narrow SPIA disciplinary expertise 4. Feedback to CGIAR priority setting and ... links to ex ante assessment... Potentially well placed in ISPC, but • no structured feedback loops to ex-ante and priority setting (?)
  21. 21. Management – efficient, within constraints • Understaffed for ambitious workplan - leading to outsourcing major parts of the work, and employing consultants • Efficient management, learning and improving • Routing part of funding through FAO - a major efficiency constraint
  22. 22. Governance - reconsider for Phase 2 PSC composition: SPIA, core donors, IA expert, Consortium Office (observer) and IEA (observer) • Approves forward work plans and research proposals • No adequate consultation mechanism for System Council and Centers/CRPs • Fund Council did not give clear guidance Potential opportunities: a) annual meeting of CGIAR research leaders b) new Committee in newly constituted System Council
  23. 23. Preliminary recommendations (partial) 1. Revisit the SIAC Theory of Change... 2. More systematic use of reviews and “evidence gap maps” (of key indicators e.g. IDOs) to identify key IA topics and research questions. 3. Systematic consultation of CGIAR research leaders on IA needs, priorities and responsibilities 4. Improve two-way feedback loops from IA to ex-ante modelling (would especially welcome discussion and comments from the ISPC meeting on this point) 5. Invest more strategically in supporting CGIAR to institutionalise and learn from IA. Suggestions include: • Regular reviews of IA in the CGIAR • Revisit design of Community of Practice for IA • Develop a (two-way) communications strategy 6. (for IEA) Upcoming ISPC / SPIA evaluation should analyse IA and related activities across the CGIAR and consider roles and responsibilities.
  24. 24. Thank you

×