Margherita Harris Visiting fellow in the Department of Philosophy, Logic and Scientific Method at the London School of Economics and Political Science. ABSTRACT: According to the severe tester, one is justified in declaring to have evidence in support of a hypothesis just in case the hypothesis in question has passed a severe test, one that it would be very unlikely to pass so well if the hypothesis were false. Deborah Mayo (2018) calls this the strong severity principle. The Bayesian, however, can declare to have evidence for a hypothesis despite not having done anything to test it severely. The core reason for this has to do with the (infamous) likelihood principle, whose violation is not an option for anyone who subscribes to the Bayesian paradigm. Although the Bayesian is largely unmoved by the incompatibility between the strong severity principle and the likelihood principle, I will argue that the Bayesian’s never-ending quest to account for yet an other notion, one that is often attributed to Keynes (1921) and that is usually referred to as the weight of evidence, betrays the Bayesian’s confidence in the likelihood principle after all. Indeed, I will argue that the weight of evidence and severity may be thought of as two (very different) sides of the same coin: they are two unrelated notions, but what brings them together is the fact that they both make trouble for the likelihood principle, a principle at the core of Bayesian inference. I will relate this conclusion to current debates on how to best conceptualise uncertainty by the IPCC in particular. I will argue that failure to fully grasp the limitations of an epistemology that envisions the role of probability to be that of quantifying the degree of belief to assign to a hypothesis given the available evidence can be (and has been) detrimental to an adequate communication of uncertainty.

1. On Severity, the Weight of
Evidence and the Relationship
Between the Two

2. Two competing principles
Strong severity principle: We have evidence for a claim C just to the extent it survives a stringent
scrutiny. If C passes a test that was highly capable of finding flaws or discrepancies from C, and yet
none or few are found, then the passing result, x, is evidence for C. (Mayo, 2018)

3. Two competing principles
Strong severity principle: We have evidence for a claim C just to the extent it survives a stringent
scrutiny. If C passes a test that was highly capable of finding flaws or discrepancies from C, and yet
none or few are found, then the passing result, x, is evidence for C. (Mayo, 2018)
The likelihood principle: All the information about ϴ obtainable from an experiment is contained in the
likelihood function for ϴ given x. Two likelihood functions for ϴ (from the same or different
experiments) contain the same information about ϴ if they are proportional to one another. (Berger
and Wolpert, 1988)

4. The likelihood principle: All the information about ϴ obtainable from an
experiment is contained in the likelihood function for ϴ given x. Two
likelihood functions for ϴ (from the same or different experiments) contain
the same information about ϴ if they are proportional to one another.
What exactly does it mean to say that all of the information that data
provides about a parameter ϴ is contained in the likelihood function?

5. The likelihood principle: All the information about ϴ obtainable from an experiment is
contained in the likelihood function for ϴ given x. Two likelihood functions for ϴ (from the
same or different experiments) contain the same information about ϴ if they are proportional
to one another.
Bayesian Confirmation Theory
Let S be a set of mutually exclusive, collectively exhaustive hypotheses, and let H be a
variable ranging over members of S. Let E be a set of observed data. The likelihood function
is L(H,E)=Pr(E|H).
Let c(H, E) indicate the incremental confirmation that E provides for H.
The likelihood principle: If the likelihood functions L(H,E) and L(H, E*) are proportional
(i.e. if Pr(E|H)=kPr(E*|H)), then for all H in S, c(H, E) = c(H, E*).

6. Does Bayes’ theorem entail the likelihood principle?
No! Bayes' theorem says something about P(H|E) but nothing about c(H, E).

7. Does Bayes’ theorem entail the likelihood principle?
No! Bayes' theorem says something about P(H|E) but nothing about c(H, E).
• A consequence from Bayes’ theorem:
If the likelihood functions L(H, E) and L(H, E*) are proportional, then Pr(H|E) = Pr(H|E*) for every H.
• Additional Assumption:
If P(H|E) = P(H|E*), then c(H, E) = c(H, E*)
⇒ Likelihood principle: If the likelihood functions L(H,E) and L(H, E*) are proportional,
then for all H in S, c(H, E) = c(H, E*).

8. The likelihood principle: If the likelihood functions L(H,E) and L(H, E*) are proportional (i.e. if
Pr(E|H)=kPr(E*|H)), then for all H in S, c(H, E) = c(H, E*)
Strong severity principle: We have evidence for a claim C just to the extent it survives a
stringent scrutiny. If C passes a test that was highly capable of finding flaws or discrepancies
from C, and yet none or few are found, then the passing result, x, is evidence for C.

9. ▪
▪

10. The Weight of Evidence

11. Can we directly measure the weight of evidence?
Can we compare the weight of two distinct evidential sets?
Only under limited conditions
E&E1
E
The Weight of Evidence

12. The paradox of ideal evidence (Popper,1959)
The Bayesian response: The weight of evidence is reflected in the resiliency of one’s credence in H

13. The Bayesian response: The weight of evidence is reflected in the resiliency of one’s credence in H

14. The Bayesian response: The weight of evidence is reflected in the resiliency of one’s credence in H
Despite this difference, however, both John’s credence in H
is ½ in both cases since John’s expectation of the chance of
the coin landing heads is 1/2 in both cases

16. The likelihood function of p
6 heads in 10 flips
300 heads in 500 flips
60 heads in 100 flips

17. If the Bayesian thinks, as they seem to think, that there is something to the notion of the weight of
evidence, and yet the weight of evidence is not always reflected in the likelihood function of a
hypothesis, then this should be enough of a reason for a Bayesian to reject the likelihood principle.

18. If the Bayesian thinks, as they seem to think, that there is something to the notion of the weight of
evidence, and yet the weight of evidence is not always reflected in the likelihood function of a
hypothesis, then this should be enough of a reason for a Bayesian to reject the likelihood principle.
The Weight of evidence and Severity
The weight will increase as the data increases, regardless of how that data was acquired. But to the
severe tester what matters in the assessment of whether one has evidence in support of a
hypothesis is not how much relevant evidence one has gathered per se, rather it is whether or not
the process of generating that evidence has been able to severely test the hypothesis in question.

19. If the Bayesian thinks, as they seem to think, that there is something to the notion of the weight of
evidence, and yet the weight of evidence is not always reflected in the likelihood function of a
hypothesis, then this should be enough of a reason for a Bayesian to reject the likelihood principle.
The Weight of evidence and Severity
The weight will increase as the data increases, regardless of how that data was acquired. But to the
severe tester what matters in the assessment of whether one has evidence in support of a
hypothesis is not how much relevant evidence one has gathered per se, rather it is whether or not
the process of generating that evidence has been able to severely test the hypothesis in question.
The Bayesian's recognition that the posterior probability that one assigns to a hypothesis
conditional on the available evidence is unable to reflect something that to the Bayesian herself
seems important about the nature of that evidence (i.e. its weight) is merely a symptom of the
Bayesian's own dissatisfaction with the likelihood principle.
A mere symptom…

20. The weight of evidence and the burden of proof
A general discomfort with the idea that all that is required for a guilty verdict is that the available
evidence makes it highly probable that the accused is guilty:
“ [the probabilistic measure] may say something about the evidence to hand, but it says nothing about
the completeness or weight of that evidence. A slight body of evidence may give rise to a reasonable
degree of probabilistic certainty, but what of the fragility of that assessment, given the many items of
further evidence that have not yet been considered?” (Hamer, 2012).

21. Davidson and Pargetter (1987):
“Cases can be dismissed on the grounds of insufficient evidence, or the guilty verdict resisted if
there is insufficient evidence, even if the evidence that is available is reliable and does make it
highly probable that the accused is guilty. This suggests that guilt beyond reasonable doubt is
only established if the jury believes that they have all the relevant evidence, in the sense that any
further evidence which probably obtains would not change the probability of guilt; or
alternatively, that any evidence which would lower the probability of guilt has a low probability.
Our third requirement for guilt beyond reasonable doubt is thus
(c) the probability of guilt must have high resilience relative to all bodies of evidence which are
probable.
We shall call strength of evidence in this sense ‘weight', and so our third requirement is that the
probability of guilty should have high weight. We follow J. M. Keynes use of the term 'weight' to
refer to this feature of a probability”
▪

22. “It is a mistake, [….] to identify the idea of Keynesian
weight with the idea of resilience…. The two ideas are
distinct, and though there are causal connections between
them, the dependence relationships are complex. We
should not, therefore, embrace the suggestion by
Davidson and Pargetter that Keynesian weight be
identified with resilience of probability […]”

23. “Despite all this, the concept of resilience does have some
use… the harder it is to shift the probability of interest, that
is, the more resilient that probability is, the less is the
expected gain in utility of determining which of various
possible states of an evidenced fact obtains before acting,
and so the more rational it is to act on the probability one
has at the time. Resilience, therefore, is an ingredient in the
rationality of acting on a probability by being an ingredient
in deciding whether one should augment Keynesian weight
by getting more information before acting on one’s
assessed probability. This connection places resilience in its
proper place; in the legal context, it is a component part of
the decision by the court whether the case is ripe for
submission of the dispute for decision by the fact-finder….
Other factors include the cost of the acquisition of further
information […] the extent to which the probability falls
below or exceeds some threshold of decision [….]”

24. Dale’s attempt to account for the weight of evidence,
together with all other attempts, are bound to be
unsatisfactory from an epistemic point of view. In Dale’s
case, his requirement of practical optimisation of Keynesian
weight is clearly epistemically unsatisfactory, because it
suggests that whether or not one is justified in declaring to
have sufficient evidence for a claim depends on pragmatic
factors that have nothing to do with the nature of the
evidence at one’s disposal nor with how severely that claim
has been put to test.

25. Intergovernmental Panel on Climate Change (IPCC)

26. Intergovernmental Panel on Climate Change (IPCC)

27. Intergovernmental Panel on Climate Change (IPCC)
“most authors agreed that the confidence/likelihood distinction was confusing” (Janzhood, 2020).