ISPC14

1. 1
08 September 2016
Towards a common framework on Quality of Science (QoS) in the CGIAR
Draft concept note and terms of reference for QoS Working Group
Background
The quality of the science conducted by the CGIAR System is perhaps the most important
determinant of the effectiveness of the organization in realizing its objectives, and contributing
to the Sustainable Development Goals. In the new CGIAR System, increased importance has
been given to ensuring that the agricultural research of the System contributes to development
outcomes under the framing of AR4D. Quality of science (QoS) in the CGIAR context is related
not only to the excellence and rigor of research, but also the ability to respond timely and
effectively to new emerging development challenges is essential for effective delivery of results.
In addition, CGIAR research needs to generate International Public Goods (IPGs), reflecting the
remit of the CGIAR.
The ISPC, as the major provider of scientific advice to the CGIAR System, is charged with
helping ensure scientific quality in the CGIAR Research Programs (CRPs). All other CGIAR
institutions, programs and System units involved in the design, implementation, appraisal,
monitoring and evaluation of research, are also dealing with QoS from their own perspectives.
While there is agreement across the CGIAR on the paramount importance of quality and
relevance of science and various segments of the System are actively involved in tasks that
assess some measure of QoS, the System does not have a common understanding or even a
common definition of Quality of Science in the CGIAR context. This problem was clearly
articulated in the main conclusions of the task force report of the ISPC which stated:
“attention to science quality and strategic thinking at the System level have suffered from the
disconnectedness between the different System entities created as part of the reform process,
coupled with the number of boards, panels, committees etc. which are involved in science
quality and developing strategies at different levels. There is no clear mechanism for
engagement between these bodies or co-ordination of their work on science quality”.
This causes significant problems in effectively designing, managing and evaluating the work
of the CGIAR. It also causes significant problems in adequately assessing staff performance.
Senior scientists who spend their time building productive partnerships that facilitate the
delivery of research results for development impact are quite likely to have less time for
publishing in peer-reviewed journals with high impact factors. A conventional measure of QoS
will not capture this type of contribution, i.e. non-publication outputs, and thus scientists
whose performance is judged by such criteria have little incentive to consider how they can
improve the delivery of their results.
QoS concerns four components, namely: inputs, processes, outputs and outcomes of science.
This involves the correct formulation of hypotheses, and the appropriateness of scientific
inputs, research methodologies and processes including ethical and gender considerations,
research outputs and outcomes (both intended and unintended), and reporting of results. It is
also important to recognize the importance of learning from “failure” and adjusting as well as

2. 2
documenting. There is a series of conceptual questions and managerial issues that need to be
resolved. For instance there is still tension between QoS and relevance issues in the context of
AR4D and the interpretation of data can often be a point of divergence (e.g. using publications
and bibliometrics as diagnosis tools or comparison metrics).
A special Task Force for strengthening the ISPC recommended expanding the remit of the ISPC
to ensure that science quality is consistent across the System1. The TF highlighted the need
for increased coherence, linkages and coordination between the different System entities
with respect to identifying and implementing actions required to ensure high quality of
science across the CGIAR. It was recommended that the ISPC should be tasked with ensuring
effective dialogue and exchange of information on QoS and future science direction in the
CGIAR. In the proposed new terms of reference for the ISPC, it is tasked with leading and
facilitating development of criteria for science quality, relevance and performance for CGIAR
research.
Working Group on QoS – proposed terms of reference
As a first step in responding to this challenge, the ISPC has called for the establishment of a
System-level Working Group on QoS (QoS WG). The ISPC chair requested nominations from
relevant system entities at meeting with CRP leaders in Montpelier in June 2016. At the same
meeting, the ISPC agreed to provide a draft concept note and terms of reference for the QoS
WG, which resulted in this draft note. The overall goal of the QoS WG is to enhance
coherence, linkages and coordination between the CGIAR System entities with respect to
quality of science.
Essentially there are three main questions that need to be answered by the QoS WG:
1) What is quality of science in the context of delivery of R4D research results and how
does the inclusion of development impact affect the way we should define QoS?
2) How we can actually measure this CGIAR-tailored QoS? What are the criteria and
methods of assessment?
3) How to incorporate and ensure QoS standards throughout the System?
The work of the QoS WG will be to respond to these questions through a series of meetings,
technical inputs and consultations and workshops in order to design a common framework
for defining and assessing QoS in the CGIAR, which will cover the scope, approach, tools,
performance measures and indicators. The QoS WG should also formulate a strategy to
ensure high QoS across the CGIAR. This will include consideration of how QoS should be
managed and overseen across Centers and CRPs.
All CGIAR System units and institutions (SC, SMB, SMO, IEA, CRPs and Centres) are stakeholders in
the QoS debate, for the design, implementation, appraisal, monitoring and evaluation of research.
The Consortium Office commissioned a study on the performance of CRPs, based on the analysis of
publication and citation data and identification of publications resulting from funds provided by
CGIAR through CRPs. A CGIAR Monitoring, Evaluation and Learning Community of Practice is
coordinating the efforts of CRPs for strengthening the monitoring, evaluation, learning and impact
1
http://cgiarweb.s3.amazonaws.com/wp-content/uploads/2015/09/Draft-ISPC-Task-Force-Report.pdf

3. 3
assessment (MELIA) at both the project and program levels, taking into account performance and
QoS indicators. The IEA has developed a framework for QoS in evaluation; a workshop on
“Evaluating the Quality of Science in the CGIAR2”, was organized with participation of
representatives of the CO, ISPC and IEA. Discussions focused on IEA’s approach to evaluating QoS in
the CRPs considering the recent experience and the evaluation framework.
Membership
The WG will be coordinated by the ISPC; members representing constituencies needed to be
agreed among:
- CRPs
- Centers (DDG-R)
- MEL COP
- IEA
- ISPC
- System Management Board /Office
Proposed activities and outputs
• Working Group on Quality of Science (QoS WG) in the CGIAR convened, with
representatives of the various actors and stakeholders, with ToR above.
• A White paper produced on current understandings of Quality of Science in the CGIAR:
definitions, criteria and indicators being used for assessing science quality within
CGIAR institutions and System units; and comparison with other relevant institutions
outside the System.
• Workshop to discuss white paper findings and build common framework for QoS in
CGIAR
• WG meetings
Next steps/timeline:
September 2016: Agreement on the WG ToR : ISPC14 Hyderabad, September 2016
September 2016: QoS WG membership agreed upon
October 2016: First meeting of QoS WG (virtual)
Jan 2017: QoS working group meeting (in-person)
March 2017: First draft ISPC White paper/ Overview on Quality of Science in the
CGIAR produced
April 2017: Workshop/Technical consultation on QoS in the CGIAR led by the QoS
WG
May 2017: Draft of document outlining CGIAR common framework for QoS
May 2017: QoS WG meeting (virtual)
June 2017: Final Report including the CGIAR Framework on Quality of Science
2
http://iea.cgiar.org/news/evaluating-quality-science-cgiar