Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Assessing Quality of Science in IEA Evaluations - Rachel Sauvinet-Bedouin


Published on


Published in: Science
  • Be the first to comment

  • Be the first to like this

Assessing Quality of Science in IEA Evaluations - Rachel Sauvinet-Bedouin

  1. 1. Assessing Quality of Science in IEA Evaluations Lessons-learned & considerations for the future Rachel Sauvinet-Bedouin, IEA ISPC 13th Meeting Item 5 May 2016
  2. 2. Relevance Efficiency Effectiveness Impact & Sustainability Quality of Science Governance & Management to SLO; Research Community; Legitimacy Quality of research and design Gender Partnerships & Capacity Development Quality of Science: an evaluation criteria
  3. 3. mgmt of QoS, processes for assuring QoS, incentives HR quality: team profiles (h-index); research design; data management IEA QoS Evaluation Framework Inputs Outputs Mgmt Process Inputs Research publications (bibliometric analysis); qualitative review of publications; peer assessments
  4. 4. IEA Workshop on Evaluating QoS Date: 10-11 December 2015, Rome Purpose: - Consolidate and strengthen the IEA’s approach to evaluating QoS; - Opportunity to explore the scope for achieving a common understanding and definition of OoS and how to link it to other aspects of performance in appraisal, monitoring and evaluation; Specific objectives among which: - Draw broader lessons for the CGIAR with the aim of preparing guidelines for evaluation of QoS 16 Participants: IEA/ISPC/IDRC/CO/Donor/CRP leader/Evaluation Team experts
  5. 5. Role and responsibility of Center vs CRP –aspects require assessment of Center products, practices and capacities  Difficult to identify shortcomings and target recommendations “What can the CRP management and oversight do to have a better handle on the QoS?” “How does the CRP use its authority to demand more from the Centers?” What have we learned from IEA experience? (1)
  6. 6. What have we learned from IEA experience? (2) • Quality of science in the context of agricultural research for development: what does this mean? • Assessing systems science and transdisciplinary research: additional challenges
  7. 7. Methodological issues • Complexity of the framework: seeing the forest from the trees? • Using quantitative and qualitative data • How to interpret evidence and findings • How to have a more systematic approach to assessing research design taking into account a broad range of disciplines and types of research
  8. 8. Outputs Research Process Resources and their Management - Human Resources - Financial Resources - Infrastructure - Research Planning and design - Research implementation and protocols - Scientific and peer review - Non peer reviewed - Non publication outputs IEA revised framework
  9. 9. 1) Is the framework a good starting point for reaching a common understanding on the various dimensions and aspects of QoS in for CGIAR? 2) What are the minimum set of indicators/information when assessing QOS in program appraisal, monitoring, evaluation? 3) Does this provide sufficient flexibility for tailoring the approach for these various functions? Questions for ISPC
  10. 10. Quality of Science in the Context of AR4D • Issue of QoS along IP: what should be the expectations? • How to integrate risk of failure in assessment of QoS? • How to benchmark? How to assess QoS at CRP level while most elements rest are the responsibilities of Centers? • How can CRP be accountable for QoS? • How can QoS of CRP be improved within this context? Further Considerations for ISPC
  11. 11. Conclusions • Revise IEA framework for assessing QoS in preparation for the next round of CRP evaluations; • Explore and address the many unresolved issues with concerted efforts from all those involved in design, appraisal, implementation, monitoring and evaluation of research for development
  12. 12. Thank you!