ISPC14

1. Evaluating science quality
a strawman
Holger Meinke, ISPC

2. Background
– All research organisation and their funders are struggling with these
questions
– This has resulted in a narrow, metric-driven approach that evauates
the performance of individuals and then adds up these metrics as a
measure of the performance of the organisation as a whole
– The process is deeply unsatisfying and has led to a drive to measure
‘impact’
– Science has a credibility problem (e.g. climate change, GMO, the
image of scientist in the media, ‘mad’ scientists who are seen as
socially irresponsible, ‘white lab coats’, socially inept, ivory tower)
– Science has influenced everything around us, in many tangible but
more often intangible ways, but rarely on its own
– Science is never value free
– Science can be used for ‘good’ and ‘evil’, perspective mattersINSERT FACULTY NAME IN FOOTER 2

3. When can science compel (policy) action?
Value consensus
Uncertainty
Low High
Low Science used to select between
well defined options
Information may reduce ambiguity
about what the options are, but
does not help to select between
them
Science used to select preferred
option
There is clear agreement about
what the options are, why action
should be taken and what the best
course of action is
High Science used to justify divergent
approaches
More science leads to increasing
number of contended options,
paralysing decision processes
Strong commitment to act, but
not sure how
Policy may decide to act ahead of
science with varying
consequences.
Science can be used to build
governance systems for managing
societal change.
Source: Pielke (2007) – The Honest Broker, Ch2

4. Matching scientific knowledge with the
nature of the problem
Cause/effect Research goals Decision making
Level I Clear, deterministic
Disciplinary, reductionist
to provide solutions to
clearly defined goals
Control, solve well
defined problems with
obvious outcomes
Level II
Level III
Derived from Allenby & Sarewitz (2010) – The Techno-Human Con
Improving water use efficiency on-farm
through the introduction of a higher
yielding, more efficient variety

5. Matching scientific knowledge with the
nature of the problem
Cause/effect Research goals Decision making
Level I Clear, deterministic
Disciplinary, reductionist
to provide solutions to
clearly defined goals
Control, solve well
defined problems with
obvious outcomes
Level II Complex, emergent
Interdisciplinary,
systems to create
options for addressing
contended goals
Adaptive, seek consensus
on ways to manage
reasonably well
understood situations
Level III
Derived from Allenby & Sarewitz (2010) – The Techno-Human Con
Improving water use efficiency on-farm
through the introduction of a higher
yielding, more efficient variety
Optimising the use of limited water
resources across enterprises of a farming
operation

6. Matching scientific knowledge with the
nature of the problem
Cause/effect Research goals Decision making
Level I Clear, deterministic
Disciplinary, reductionist
to provide solutions to
clearly defined goals
Control, solve well
defined problems with
obvious outcomes
Level II Complex, emergent
Interdisciplinary,
systems to create
options for addressing
contended goals
Adaptive, seek consensus
on ways to manage
reasonably well
understood situations
Level III
Unforeseeable shifts
with no definable
boundaries
Design governance
systems capable of
agreement on ill-
defined and highly
contended goals
Seek agreement on
actions and develop
robust institutional
responses to anticipate
future conditions despite
persistent uncertainty
Derived from Allenby & Sarewitz (2010) – The Techno-Human Condition
Improving water use efficiency on-farm
through the introduction of a higher
yielding, more efficient variety
Optimising the use of limited water
resources across enterprises of a farming
operation
Introducing irrigation on a large scale as a
transformational technology that will alter
not only farming operations, but the fabric
of rural communities in a region

7. Matching scientific knowledge with the
nature of the problem
Cause/effect Research goals Decision making
Level I Clear, deterministic
Disciplinary, reductionist
to provide solutions to
clearly defined goals
Control, solve well
defined problems with
obvious outcomes
Level II Complex, emergent
Interdisciplinary,
systems to create
options for addressing
contended goals
Adaptive, seek consensus
on ways to manage
reasonably well
understood situations
Level III
Unforeseeable shifts
with no definable
boundaries
Design governance
systems capable of
agreement on ill-
defined and highly
contended goals
Seek agreement on
actions and develop
robust institutional
responses to anticipate
future conditions despite
persistent uncertainty
Derived from Allenby & Sarewitz (2010) – The Techno-Human Condition

8. Assumption
Science within the context of the CGIAR mission should be
an enabler of desirable, societal outcomes by
ensuring that appropriately targeted innovations
within the CGIAR’s mandate can take hold.
High quality science is and will remain the lifeblood of the CGIAR
and must be judged in light of its contribution to the CGIAR’s
mission.
INSERT FACULTY NAME IN FOOTER 8

9. Assumptions
– Science cannot achieve these audacious goals lone. This must be
recognised when evaluating science and passing judgement on
science quality.
– Hence, ‘quality’ also requires that these other components are drawn
into the science process appropriately (‘co-production of outcomes’)
– We need to recognise that impact of innovation in agriculture and food
systems is facilitated through value chains that co-innovate for mutual
benefit.
– Science needs to be defined broadly and inclusively – we should avoid
disciplinary arrogance
– Need approaches to assess quality of social sciences
– Need to evaluate the quality of the enabling environment
INSERT FACULTY NAME IN FOOTER 9

10. … encompasses a series of
scientific, technological, organisational,
financial & commercial activities.
H. Meinke
TI
A
Innovation is usually disruptive and frequently very disruptive. To be
‘successful’ it requires
• A degree of confidence and willingness to be disrupted
• A high level of inquisitiveness
• A strong degree of cultural aspirations
For innovation to be successful requires the right attitudes,
aspirations and abilities of stakeholders.
If impact via innovation is the goal, ‘science’ needs to be evaluated
broadly and include the scientists, the program and the organisation.
All of them need to be evaluated in terms of performance (science
conduct, process, facilities etc) AND behaviour (solution-oriented,
partnerships, governance, impact-driven communication etc).