This document discusses using probabilistic decision tools and modeling approaches for evidence-based impact evaluation and development policy decisions. It argues that traditional scientific methods are not well-suited for studying complex, real-world agricultural systems due to their interdisciplinary nature and many uncertainties. Instead, it advocates adopting a decision analysis approach which (1) incorporates all important system aspects, (2) models the system using all available information including uncertainties, (3) identifies key uncertainties, and (4) recommends options based on expected outcomes to support decisions that must be made without perfect information. The document provides a case study on using decision analysis to evaluate options for reducing reservoir sedimentation in Burkina Faso.

1. Probabilistic Decision Tools for
Evidence-Based Impact Evaluation
Modeling real-world complexities to support
development policy decisions.
Caroline Muchiri, Eike Luedeling, Keith Shepherd and Yvonne Tamba

2. Modeling the impact pathway of agricultural development research
• Aimed at supporting development decisions
• Contribute to development impact
• Therefore this requires our models to adequately
reflect the reality of agricultural systems
Agricultural
model
Support
decisions
Improved
decisions
Development
impact

3. Agriculture is inherently complex and interdisciplinary
Reproduced from: McConnell and Dillon, 1997. Farm Management for Asia: A Systems Approach. FAO
Farm Systems Management Series 13 (http://www.fao.org/docrep/w7365e/w7365e00.htm#Contents).
Managerial
subsystem
Technical
subsystem
Informal
structural
subsystem
Goals and
values
subsystem
Organizational
structural
subsystem
Environmental
suprasystem
• Culture
• Technology
• Education
• Political setting
• Legal setting
• Demography
• Sociology
• Climate
• Economics
System behavior arises
from a host of factors
and interactions
How can we study the
behavior of the whole
system?
Can our usual research
and modeling
approaches address
this?
Input-output flow of
material, energy,
information and influence
Decision-makers need
tools that allow
forecasts for the whole
system

4. Our usual approach to knowledge generation
The scientific
method
Inspired by https://commons.wikimedia.org/w/index.php?curid=42164616
Formulate
hypotheses
Make observations &
think of interesting
questions
Develop general
theories
Develop testable
predictions
Gather data to test
predictions
Refine, alter,
expand or reject
hypotheses
• Seeks for objective, widely valid
system behavior rules
• Reduces complexity to allow
hypothesis testing
• Dissects the system into researchable
fragments
• Struggles where many disciplines are
involved and rules are complex

5. System models from solid building blocks?
Can we ‘stack’ study results obtained
with the scientific method to gain
system understanding?
Study 1
Study 2
Study 3
Study 1
Doesn’t work if the blocks don’t fit
Insufficient
knowledge for
decision support
Or if we can’t get enough
blocks (gaps)

6. How to do a small study on agriculture
This is usually preferable, but
how can it be done?
Biophysical
aspects
Economic
aspects
Social
aspects
Environmental
aspects
Cultural
aspects
Biophysical
aspects
Biophysical
aspects
Economic
aspects
Social
aspects
Environmental
aspects
Cultural
aspects
An agricultural system
Given limited resources for
research, how should we study
this?
By addressing one
aspect in great detail?
Are our results still
relevant for the
agricultural system?
By doing a coarser assessment that
includes all essential dimensions?
OR

7. How do good decision-makers generate knowledge?
Structured decision analysis
Frame entire
decision context
Model the
situation using
all available
information
Quantitatively
project decision
outcomes
Characterize all
risks and
uncertainties
Identify key
uncertainties
Measure where
information
value is high
Recommend
preferable
option
• For systems that are too complex
to fully understand (with available
resources)
• For supporting decisions that
MUST be made without perfect
information
• “What’s the best option according
to our limited understanding?”

8. Key principles of decision analysis
Shepherd et al., 2015. Nature 523, 152-154. Luedeling and Shepherd, 2016. Solutions 7(5), 46-54.
1. Incorporate all important aspects into the model.
2. Model the system using all sources of information,
including local and expert knowledge
3. Explicitly consider uncertainties about inputs,
processes and outputs (probabilistic models)
4. Identify key uncertainties for measurement using
‘Value of Information’ analysis
5. Update model, when new information becomes
available

9. The decision analysis process
Adopted from Luedeling and Shepherd, 2016. Solutions 7(5), 46-54

10. Probabilistic simulation
Normal
model
Precise numbers
as input
42
Precise number as
output
Probabilistic
model
Distributions as
input, because
precise values are
unknown
Distribution as
output
• Allows working with
variables that we don’t
have perfect knowledge on
• Requires characterizing
and quantifying our
uncertainty about them
• Common methods are
Monte Carlo simulation
and Bayesian Networks

11. A case study - Decision analysis in agricultural development research
Reservoir Protection in Burkina Faso
• How can sedimentation be prevented? (dredging, check dams, buffer strips)
• Are the options available cost-effective?

12. Reservoir protection from sedimentation in Burkina Faso
Participatory assessment
What management
options are
available?
What are the risks,
costs and benefits
for each option?
What is known
about them?
What is the
plausible range of
outcomes?
What additional
information do
decision-makers
need?
What is the most
promising and cost-
effective course of
action?

13. Reservoir protection in Burkina Faso
Lanzanova et al., in press.

14. Reservoir protection in Burkina Faso
Profit margin of
vegetable
production
• Best option is combination
of dredging, check dams and
buffer strips
Lanzanova et al., in preparation.
• Probably positive outcome,
but small risk of net losses
• Additional information on
profitability of vegetable
production would facilitate
decision Net Present Value (NPV) for combined
intervention (in thousands of USD)
Expected Value of Perfect
Information
(EVPI; in thousands of USD)

15. Contrasting research paradigms
?
?
?
Knowledge generation Problem-solving
Scientific method Decision scienceWhere is agricultural research?
Agricultural Scientists’
methodological comfort zone
How far can we stretch it? It may be easier to reach our
objectives from here
Relatively simple
Predictable
Replicable
Disciplinary
Generalizable
Follows a small number of rules
Complex
Stochastic/unpredictable
No replication possible
Inter-/transdisciplinary
Behavior is context-dependent
Complex and unclear behavior rules
?
Main purpose
System characteristics
?

16. Conclusions
• Decision sciences are geared towards
solving problems
• They don’t aim at precision or ultimate
answers, but at offering comprehensive
advice for decisions and system
management
• If solving problems, supporting decisions
and facilitating development in complex
systems is our goal…
Decision sciences offer a more fitting
paradigm for agricultural modeling
The scientific method is a poor fit for
decision-oriented agricultural models
• Most agricultural models aim to solve
problems and support decisions
• This is (usually) not a quest for basic
laws of nature!
• We need research approaches that can
deal with the complexity and
uncertainty of the systems we work on
• We need pragmatic ways of supporting
decision processes

17. …we should adopt decision analysis thinking as
our research paradigm
Thanks for your attention!
k.shepherd@cgiar.org