Weighing evidence and assessing uncertainty in microbiological risk assessment

FEDERALINSTITUTE
FORRISKASSESSMENT
Weighing evidence and assessing
uncertainty in microbiological risk
assessment
Approaches for preparing appropriate scientific support
for decision making in complex questions
Matthias Greiner
Shaping the Future of Food Safety, Together.
Milan, 14-16 Oct 2015

Shaping the Future of Food Safety, Together. Milan, 14-16 Oct 2015 Page 2
Contents
What is evidence, uncertainty and WoE all about and
what are the application areas in microbiological food
safety?
Why do we need to apply these concepts more
systematically?
How can we use these concepts to deal with (the
limitations of our) knowledge?
 What
 Why
 How

Credit
To all members of the current Working Group of the EFSA Scientific
Committee “Guidance on The Use of the Weight of Evidence
Approach in Scientific Assessments”
External experts: Maged Younes, Emilio Benfenati, Qasim Chaudhry, Peter Craig, Geoff
Frampton, Matthias Greiner, Anthony Hardy, Andrew Hart, Christer Hogstrand, Gijs
Kleter, Claude Lambre, Robert Luttik, Alonso Siani
EFSA: Djien Liem (AFSCO), Sybren Vos (ALPHA), Elisa Aiassa (AMU), : Marco Binaglia
and Michaela Hempen (BIOCONTAM), Jaime Aguilera (FEEDAP), Anna Castoldi (FIP)
and Camilla Smeraldi, Antonio Fernandez-Dumont (GMO), Silvia Valtuena-Martinez
(NDA), Andrea Terron (PRAS), Roy Kirby (Quality Management), Jean Lou Dorne
(SCER)

Definitions and context
– Working definitions
– Current practices related to evidence and uncertainty in the area of
microbiological risk assessment
– Application contexts in which the weight-of-evidence (WoE) concept
is used
 What

Definitions
Evidence
– The available body of facts or information indicating whether a belief
or proposition is true or valid (Oxford Dictionaries)
Uncertainty
– Used in the EFSA Draft Guidance on Uncertainty as a general term
referring to all types of limitations in the knowledge available to
assessors at the time an assessment is conducted and within the
time and resources agreed for the assessment
Weight-of-evidence (WoE)
– Process in which all of the evidence considered relevant for a risk
assessment is evaluated and weighted (WHO/IPCS, 2009)

Context of statements about evidence and
uncertainty
Search in the EFSA Extranet
– Repository of documents of EFSA and collaborating partners
– Search not restricted to publication date or to a particular panel;
last update 10 Oct 2015
– Text snippets as retrieved (rather than the full documents) used as
text corpus
– Extracting sentences with the terms “evidence” or “uncertainty”
evidence AND food AND (microbio* OR outbreak* OR foodborne)
in Documents in the Directory, Collaboration Items
uncertain* AND food AND (microbio* OR outbreak* OR foodborne)

Context of EFSA statements about evidence and
uncertainty
Evidence: in the conclusion – Uncertainty: in the scientific background?
Data source:
EFSA Extranet

Context of statements about evidence and
uncertainty
Search in ScienceDirect
– Scientific contents source covering articles from over 2,500 journals
and more than 33,000 book titles
– Search covering freely accessible contents (guest access); not
restricted to publication date; last update 10 Oct 2015
– Using title, abstract and keywords as text corpus
– Extracting sentences with the terms “evidence” or “uncertainty”
TITLE-ABSTR-KEY(evidence AND food AND (microbio* OR outbreak* OR
foodborne))
TITLE-ABSTR-KEY(uncertain* AND food AND (microbio* OR outbreak* OR
foodborne))

Context of other scientific statements about
evidence and uncertainty
Data source:
ScienceDirect
Evidence: part of discussion – Uncertainty: close to regulatory science?

Weight-of-evidence concept
EFSA Extranet
– Citations in all areas: 7 results
– Citations in microbiological food safety: Zero results
ScienceDirect
– Citations in all areas: 1,636 results
– Citations in microbiological food safety: Zero results
("weight-of-evidence" OR "weight of evidence" OR "evidence synthesis") AND
AND food AND (microbio* OR outbreak* OR foodborne)
TITLE-ABSTR-KEY({weight-of-evidence} OR {weight of evidence} OR {evidence
synthesis}) AND TITLE-ABSTR-KEY(food AND (microbio* OR outbreak* OR
foodborne))

Weight-of-evidence concept
WHO Microbiological Risk Assessment Series
– WoE for causality inferences. Assessing quantity, quality and nature of the
results available from various study types; pathogen characteristics;
biological mechanisms; extrapolating from animal or in vitro studies to
humans (WHO, 2003)
– Weights proportional to sample size, expert beliefs and uncertainty (WHO,
2008)
– WoE will become increasingly prominent in risk assessments of
microbiological pathogens in food (WHO, 2009)
Advocating the WoE approach for integrating of
evidence from multiple sources

Application of the weight-of-evidence concept in
microbiological food safety and related areas
Foodborne outbreaks:
Support of emergency
measures
Scientific evidence on:
Population, agent,
vehicle, source,
adverse effect
Examples:
Health Canada, 2011
Vik et al., 2014
Food safety
intervention:
Decision support
Problem, intervention,
outcome
Examples:
Fazil et al., 2008
Other applications:
a) Dose-response models
b) Water management
c) Risk ranking
Various aspects depending
on the problem
Examples:
a) Moon et al., 2005
b) Olivieri et al., 2014
c) EFSA BIOHAZ, 2015

Why do we need to apply these concepts more
systematically?
– Complexity of questions in microbiological risk assessment
– Dealing with limits of our current knowledge
 Why

Complexity in microbiological food safety
Epidemiological triangle
Host
– Behaviours
– Acquired or inherent
susceptibility or vulnerability
Agent
– Species /genotype /phenotype
characteristics
– Gene transfer
Environment
– Biotic and abiotic factors for
growth
Environment
Agent
Host
Agent

Complexity in microbiological food safety
Host
Microbiome
Infectome
Adapted from Bogdanos et al., 2013

How can we use these concepts to deal with
(the limitations of our) knowledge?
– Risk assessment – an evidenced-based approach
– Must make limitations and uncertainties explicit
– Use WoE in cases of alternative sources of evidence
 How

Conceptual model of a food safety question

Uncertainty assessment
Risk question
Scenario
Model
Model parameters
Data
Calculations/Simulations
– Describe limits of current
knowledge
– Safeguard against over-
interpretation http://www.bfr.bund.de/cm/350/guidelines-on-
uncertainty-analysis-in-exposure-assessments.pdf
http://www.efsa.europa.eu/sites/default/files/consultati
on/150618.pdf

Combining evidence (I)
Meta-analysis situation
Multiple primary studies on the same question
and comparable outcome metrics (e.g. odds
ratios from epidemiological studies; sensitivity and
specificity of diagnostic tests)
– Be systematic  use systematic review
(PRISMA standard*)
– Use an instrument for assessing study quality*
– Use strict inclusion/exclusion criteria
– Investigate and account for heterogeneity
– Use established statistical models for summary
estimates and confidence intervals
*http://www.equator-network.org/
http://www.efsa.europa.eu/sites/de
fault/files/scientific_output/files/mai
n_documents/432.pdf

Combining evidence (II)
Weight-of-evidence situation
Combining information from primary studies with different types of
observations (e.g. in vivo, in vitro, in silico, epidemiological), different
study organisms (e.g. human and animal)
– Resembles the approach for meta-analysis (use systematic approach
as far as possible)
– ...
– No established statistical tools available for integrating evidences
(currently under review by EFSA working group)

Combining evidence (II)
Weight-of-evidence
Line of
evidence
Internal
validity
Relevance
for question
Weight Outcome
in vitro ? ? ? ?
in vivo ? ? ? ?
epidemiology ? ? ? ?

OUTBREAK INVESTIGATIONS AND
DOSE-RESPONSE MODELLING
Alternative weight-of-evidence approaches

Health Canada, 2011 (clipping of p. 12)
Grading
of the
evidence

Binomial dose-response model
Dose
– Number of bacterial cells ingested: n
Response
– Probability of infection at dose n: P(n)
Model
P(n) = 1 – (1 – p)n
p is the shape parameter (probability that a single cell causes infection)
Assumptions
– r is constant in the population of bacterial cells
Use
– If the dose n is known exactly
Using
Evidence in
modelling

Extensions of the binomial model
Exponential
– Use if n is not known precisely; n ~ Poisson(●)
Beta-binomial
– p is not known precisely; p ~ beta(●,●)
Beta-Poisson
– use if p and n
are not known precisely;
p ~ beta(●,●)
n ~ Poisson(●)
● = distribution
parameter
for simulation

Evidence-based choice or combination of models
Classical approach
– Using biological reasoning
about underlying assumptions
– Asymptotic behaviour
– Choosing the model with the
best fit
Binomial Beta-Poisson
WoE approach
– Use all models and combine
results mathematically
– “Evaluate the relative
plausibility of each fitted model
by a weight of evidence
relative to the selected best
model” (Moon et al., 2005.
Model Averaging Using the
Kullback Information Criterion
in Estimating Effective Doses
for Microbial Infection and
Illness. Risk Analysis 25/5,
1147-1159)
0 500 1000 1500 2000
0.00.20.40.60.81.0
n
P(n)
r=0.003
r=0.002
r=0.001
d
P(d)
10
2
1 10
2
10
4
10
6
10
8
10
10
10
12
0
0.2
0.4
0.6
0.8
1
0.2 0.4
0.2 40
0.8 4000

Conclusions
Internal validity and relevance for the question (external validity)
– Can be part of a uncertainty assessment
– Can be used to derived weights in a WoE approach
`The two methodologies weighing of evidence and uncertainty
assessment are complementary and have a common overall goal,
which is to provide the best possible basis for science-based
decision making
Line of
evidence
Internal validity
Relevance for
question
Weight Outcome
in vitro ? ? ? ?
in vivo ? ? ? ?
epidemiology ? ? ? ?

References
Bogdanos DP et al. (2013). Infectome: A platform to trace infectious triggers of autoimmunity. Autoimmun Rev. 12:726-40.
EFSA BIOHAZ Panel (2015). Scientific Opinion on the development of a risk ranking toolbox for the EFSA BIOHAZ Panel.
EFSA Journal 2015;13: 3939, 131 pp.
Fazil A et al. (2008). Choices, choices: the application of multi-criteria decision analysis to a food safety decision-making
problem. Journal of Food Protection 71, 2323-2333.
Health Canada (2011). Public Health Agency of Canada; Canadian Food Inspection Agency. Weight of evidence: factors to
consider for appropriate and timely action in a foodborne illness outbreak investigation. Ottawa, ON: Minister of Health.
Moon H et al. (2005). Model averaging using the Kullback information criterion in estimating effective doses for microbial
infection and illness. Risk Analysis 25/5, 1147-1159.
Olivieri AW et al. (2014). Risk-Based Review of California’s Water-Recycling Criteria for Agricultural Irrigation. Journal of
Environmental Engineering, 2014.140.
Vik J et al. (2014). Summary: weight of evidence - factors to consider when investigating a food-borne illness outbreak.
Canada Communicable Disease Report 40, 303.
WHO (2003). Hazard characterization for pathogens in food and water: guidelines. Microbiological Risk Assessment Series,
No. 3. WHO, Geneva, 76 pp.
WHO (2008). Exposure assessment of microbiological hazards in food. Microbiological Risk Assessment Series, No. 7.
WHO, Geneva, 102 pp.
WHO (2009). Risk Characterization of microbiological hazards in food. Microbiological Risk Assessment Series, No. 17
WHO, Geneva, 135 pp.
WHO/IPCS (2009). EHC 240: Principles and methods for the risk assessment of chemicals in food. Annex 1, 45 pp.

FEDERALINSTITUTE
FORRISKASSESSMENT
FEDERALINSTITUTE
FORRISKASSESSMENT
Thank you
Matthias Greiner
Federal Institute for Risk Assessment, and
University of Veterinary Medicine Hannover (TiHo),
Foundation
Tel. +49 30 18412 3297  Fax +49 30 18412 2958
matthias.greiner @bfr.bund.de  www.bfr.bund.de

Weighing evidence and assessing uncertainty in microbiological risk assessment

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Weighing evidence and assessing uncertainty in microbiological risk assessment

Similar to Weighing evidence and assessing uncertainty in microbiological risk assessment (20)

More from EFSA EU

More from EFSA EU (20)

Recently uploaded

Recently uploaded (20)

Weighing evidence and assessing uncertainty in microbiological risk assessment