Phil 6334, D. Mayo: Duhem's Problem and more...(April 3 2014 slides)

1. 1
April 3, 2014 Phil 6334
The Howson (1997) paper was invited as a discussion on the general
topic of a paper of mine (and one other paper): “Duhem's Problem, the
Bayesian Way, and Error Statistics or “What’s Belief Got to Do With It?”
The physicist can never submit an isolated hypothesis to the control
of experiment, but only a whole group of hypotheses. When
experiment is in disagreement with his predictions, it teaches him
that one at least of the hypotheses that constitute this group is wrong
and must be modified. But experiment does not show him the one
that must be changed (Duhem 1954, p.185).

2. 2
“...is to point out that the Bayesian personalist approach to
scientific inference provides a ...solution to this [Duhem] puzzle by
telling us exactly when [disregarding unsuccessful predictions] can
be reconstructed as rational and when it has to be deemed
irrational. Rationality here, for the Bayesian, simply means
conformity with [Bayes'] theorem” (Dorling 1979, p. 177).
Bayes' Theorem (one form):
P(H | e) = P(e | H) P(H)
P(e | H) P(H) + P(e | not-H) P(not-H)
The Bayesian “catchall factor”: P(e | not-H).

3. 3
1. Dorling’s Homework Problem:
(I). The components:
Hypothesis H: Newton’s theory of motion and gravitation
e: the predicted secular acceleration of the moon
e’: the observed acceleration of the moon--the anomalous result
Auxiliary hypothesis A: the effects of tidal friction are not of a
sufficient order of magnitude to affect the acceleration of the
moon.
H and A entail e, but e’ is observed.

4. 4
(II) An informal (and neutral) description of a situation where
anomaly e’ indicates (or is best explained by) auxiliary A being in
error:
 there is a great deal of evidence in favor of a theory or
hypothesis H, whereas
 there is little evidence for the truth of auxiliary A, say hardly
more evidence for its truth than for its falsity, and
 unless A is false, there is no other plausible way to explain e’.
A Bayesian rendering may be obtained by inserting
"agent X believes that"
prior to assertions (1), (2), and (3).

5. 5
(III) The Numerical Solution to the Homework Problem:
ASSUME:
(i) P(H) = 0.9 and P(A) = 0.6
(ii) “The agent contemplates auxiliary A being true”.
The probability of e’, given A and not-H is very small. Let this
very small value be .
(Dorling takes  to be 0.001.)
(iii)“The agent contemplates auxiliary A being false”
(a) The probability of e’, given H and not-A, is 50.
(b) The probability of e’, given not-H and not-A, is 50.
(iv) H and A are probabilistically independent:
P(H and A) = P(H)P(A)

6. 6
1.1. THE RESULTS
The Bayesian Catchall factor: P(e’|not-H) = 20.6
(.0206)
The posterior probabilities: P(H|e’) = 0.897
P(A|e’) = 0.003
H hasn’t gone down much, the blame for the anomaly is placed on
auxiliary A.
Of course, the opposite assignment could have been given thus
putting the blame on the theory.

7. 7
Two key features of the error statistical approach (of central relevance
to Duhem's problem):
1. A Piecemeal Approach.

Two contrasts with the Bayesian Way may be noted:
I. Gets beyond a single probability pie
II. Gets beyond a white-glove analysis

8. 8
2. The Fundamental Use of Error Probabilities of Tests
▪ The question of whether data provides good evidence for a
hypothesis is regarded as an objective (though empirical)
one, not a subjective, one.
▪ Data count as good evidence for H just to the extent that H
passes a severe test.

9. 9
Statistical Significance Tests
The null (or “test”) hypothesis: H0, there is no increased risk of
R (in a given population)
[H0 says it is an error to suppose a genuine increased risk is
responsible for any observed difference in risk rates.]
The alternative hypothesis: J, there is an increased risk of R (in a
given population)
Here e is anomalous for J.

10. 10
The significance question: how often would a (positive)
difference in R rates as high as (or even higher than) the one
observed (e) occur, if in fact H0 were true? The answer is called
the statistical significance level of the data.

11. 11
Error Probabilities Register Illegitimate Ways to “Save”
Hypotheses From Anomalies
H0: there is no increased risk of R (in a given population)
J: there is an increased risk of R (in a given population)
e: a 0 (or a statistically insignificant) difference
Way #1: J + “compensation hypothesis” (there really is a risk
but something compensated for it in this data)
Way #2: J’: there is an increased risk of R’ (in a given
population)
(There’s some other risk, found through searching the data)

12. 12
The hypotheses erected to accord with the evidence fail to
pass severe tests.
The probability of erroneously finding some alleged
compensating factor or other, Way #1, and the probability of
erroneously finding one or another excess in risk, Way #2, are
no longer the low .01 level as at the start, but are instead higher,
in extreme cases, maximal (i.e., 1)

13. 13
skip
Two Types of Strategies in the Error Statistical Approach to
Duhem's Problem:
(1) “blocker” strategies:
We criticize attempts to explain away anomalies (e.g., as due to
H-saving factors) on the grounds that
 they fail to pass severe tests
 their denials pass severe tests.
(2) Show anomaly e’ may be blamed on an auxiliary hypothesis A
by showing A’ (the denial of A), passes a severe test.

14. 14
In Dorling's illustration, a result e’ that is anomalous for H is
taken to provide positive grounds for discrediting A and
confirming its denial A'. (The degree of belief in A' went from
0.4 to 0.99—by dint of anomaly e’.)
But the error statistician wants to know if the test is severe!
 this requires positive evidence that the alleged extraneous factor is
responsible for the anomaly.
 strong belief in H together with low enough degree of belief in the
Bayesian catchall factor do not suffice for showing A' has passed a
severe test.
 going from satisfying the Bayesian conditions to declaring strong
evidence for A'--is a very unreliable one (makes it too easy to blame
auxiliary hypothesis A even if A is true).
such an appeal to A’ would thereby be blocked.

15. 15
Mini overview:
Severity Requirement: Data x provides good evidence for
inferring H only if it results from a procedure which, taken as a
whole, constitutes H having passed a severe test — that is, a
procedure which would have (at least with very high probably)
uncovered the falsity of, or errors in H, and yet H emerged
unscathed.
Inductive learning, in this view, proceeds by testing hypotheses and
inferring those which pass probative or severe tests — tests which
very probably would have unearthed some error in the hypothesis H,
were such an error present.

16. 16
A methodology for induction, accordingly, is a methodology for
arriving at severe tests, and for scrutinizing inferences by considering
the severity with which they have passed tests.
Methodological rules and strategies are claims about how to avoid
mistakes and learning from different types of errors; their appraisal
turns on understanding how methods enable avoidance of specific
errors.
Hence an inductive methodology of severe testing will focus on
understanding the properties of tools for generating, modeling and
analyzing data so as to learn about some aspect of the data-
generating mechanism.
These properties, while empirical, are objective.

17. 17
Highly Probable vs. Highly Probed Hypotheses
The Criticism: H may pass with high severity (with data x) even
though the (Bayesian) posterior probability for H (given x) is low.
All such “funny Bayesian examples” need to assume prior
probability assignments to an exhaustive set of hypotheses, while for
a frequentist error statistician, a hypothesis could only be given a
probability assignment if its truth is the outcome of a random trial
(but “events” to not also serve as statistical hypotheses).
 Subjective degree of belief assignments will not ensure the error
probability, and thus the severity, assessments we need.
 Examples with frequentist priors, however, commit the fallacy of
probabilistic instantiation.

18. 18
The Fallacy of Probabilistic Instantiation
Hypothesis H is true of p% of the populations (bags) in this urn of
populations U,
1. P(H is true of a randomly selected bag from an urn of bags U) = p
2. The randomly selected bag that was drawn is the bag used in test
T1 is b1,
Therefore:
(*) P(H is true of b1) = p.
For the frequentist: either H is true of b1 or not — the probability in
(*) is fallacious and results from an unsound instantiation.

19. 19
Students from the Wrong Side of Town
Isaac, has passed comprehensive tests of mastery of high school
subjects regarded as indicating college readiness…
Since such high scores s could rarely result among high school
students who are not sufficiently prepared to be deemed ‘college
ready” we regarded s as good evidence for
H(I): Isaac is college ready.
And let the denial be H’:
H’(I): Isaac is not college ready (i.e., he is deficient).
The probability for such good results, given a student is college ready,
is extremely high:
P(s | H(I)) is practically 1,

20. 20
while very low assuming he is not college ready.
P(s | H’(I)) =.05.
But imagine Isaac was randomly selected from the population of
students in, let us say, Fewready Town—where college readiness is
extremely rare, say one out of one thousand. The critic infers that the
prior probability of Isaac’s college-readiness is therefore .001:
(*) P(H(I)) = .001.
If so, then the posterior probability that Isaac is college ready, given
his high test results, would be very low:
p(H(I)|s) is very low,
even though the posterior probability has increased from the prior in
(*).

21. 21
This is supposedly problematic for testers because we’d say this was
evidence for H(I) (readiness).
Actually I would want degrees of readiness to make my inference, but
these are artificially excluded here.
But, even granting his numbers, the main fallacy here is fallacious
probabilistic instantiation.
Although the probability of a randomly selected student taken from
high schoolers in Fewready Town is .001, it does not follow that
Isaac, the one we happened to select, has a probability of .001 of
being college ready

22. 22
Achinstein says he will grant the fallacy…but only for frequentists:
“My response to the probabilistic fallacy charge is to say that
it would be true if the probabilities in question were construed
as relative frequencies. However, …I am concerned with
epistemic probability.”
He is prepared to grant the following instantiations:
▪P% of the hypotheses in a given pool of hypotheses are true (or a
character holds for p%).
▪The particular hypothesis Hi was randomly selected from this pool.
▪Therefore, the objective epistemic probability P(Hi is true) = p.

23. 23
Of course, epistemic probabilists are free to endorse this road to
posteriors—this just being a matter of analytic definition.
But the consequences speak loudly against the desirability of doing
so.
No Severity. The example considers only two outcomes: reaching the
high scores s, or reaching lower scores, ~s.
Clearly a lower grade gives even less evidence of readiness;
that is, P(H’(I)| ~s) > P(H’(I)|s). Therefore, whether Isaac scored as
high as s or lower, ~s, the epistemic probabilist is justified in having
high belief that Isaac is not ready.
The probability of finding evidence of Isaac’s readiness even if in fact
he is ready (H(I) is true) is low if not zero.

24. 24
Bayesian B-boosters might interpret things differently, noting that
since the posterior for readiness has increased, the test scores provide
at least some evidence for H(I)—but then the invocation of the
example to demonstrate a conflict between a frequentist and Bayesian
assessment would seem to diminish or evaporate.
Reverse Discrimination? To push the problem further, suppose that
the epistemic probabilist receives a report that Isaac was in fact
selected randomly, not from Fewready Town, but from a population
where college readiness is common, Fewdeficient Town.
The same scores s now warrant the assignment of a strong objective
epistemic belief in Isaac’s readiness (i.e., H(I)).
A high-school student from Fewready Town would need to have
scored quite a bit higher on these same tests than a student selected
from Fewdeficient Town for his scores to be considered evidence of
his readiness.

25. 25
When we move from hypotheses like “Isaac is college ready” to
scientific generalizations, the difficulties become even more serious.
We need not preclude that H(I) has a legitimate frequentist prior; the
frequentist probability that Isaac is college ready might refer to
generic and environmental factors that determine the chance of his
deficiency—although I do not have a clue how one might compute it.
The main thing is that this probability is not given by the probabilistic
instantiation above.
These examples, repeatedly used in criticisms, invariably shift the
meaning from one kind of experimental outcome—a randomly
selected student has the property “college ready”—to another—a
genetic and environmental “experiment” concerning Isaac in which
the outcomes are ready or not ready.
This also points out the flaw in trying to glean reasons for epistemic
belief with just any conception of “low frequency of error.”

26. 26
If we declared each student from Fewready to be “unready,” we
would rarely be wrong, but in each case the “test” has failed to
discriminate the particular student’s readiness from his unreadiness.
Moreover, were we really interested in the probability of the event
that a student randomly selected from a town is college ready, and had
the requisite probability model (e.g., Bernouilli), then there would be
nothing to stop the frequentist error statistician from inferring the
conditional probability.
However, there seems to be nothing “Bayesian” in this relative
frequency calculation.
Bayesians scarcely have a monopoly on the use of conditional
probability! But even here it strikes me as a very odd way to talk
about evidence.
(Howson says it shows unsoundness because he identifies a p-value
with a posterior probability in a hypothesis)

27. 27
A Common Variant on the Criticisms: p-values vs. posterior
probabilities):
 Certain choices for prior probabilities in the null and
alternative hypothesis shows that a small p-value is consistent with a
much higher posterior probability in null hypothesis,
 The alternative hypothesis would, in such cases, pass severely,
even though the null hypothesis has a high posterior (Bayesian)
probability.

28. 28
A statistically significant difference from H0 can correspond to large
posteriors in H0
From the Bayesian perspective, it follows that p-values come up short
as a measure of inductive evidence,
 the significance testers balk at the fact that the recommended priors
result in highly significant results being construed as no evidence
against the null — or even evidence for it!

29. 29
The conflict often considers the two sided T(2 test
H0:  =  versus H1:  ≠  .
(The difference between p-values and posteriors are far less marked
with one-sided tests).
“Assuming a prior of .5 to H0, with n = 50 one can classically ‘reject H0
at significance level p = .05,’ although P(H0|x) = .52 (which would
actually indicate that the evidence favors H0).”
This is taken as a criticism of p-values, only because, it is assumed the
.51 posterior is the appropriate measure of the belief worthiness.
As the sample size increases, the conflict becomes more
noteworthy.

30. 30
If n = 1000, a result statistically significant at the .05 level leads
to a posterior to the null of .82!
SEV (H1) = .95 while the corresponding posterior has gone
from .5 to .82. What warrants such a prior?
n (sample size)
______________________________________________________
p t n=10 n=20 n=50 n=100 n=1000
.10 1.645 .47 .56 .65 .72 .89
.05 1.960 .37 .42 .52 .60 .82
.01 2.576 .14 .16 .22 .27 .53
.001 3.291 .024 .026 .034 .045 .124

31. 31
(1) Some claim the prior of .5 is a warranted frequentist assignment:
H0 was randomly selected from an urn in which 50% are true
(*) Therefore P(H0) = p
H0 may be 0 change in extinction rates, 0 lead concentration, etc.
What should go in the urn of hypotheses?
For the frequentist: either H0 is true or false the probability in (*) is
fallacious and results from an unsound instantiation.
We are very interested in how false it might be, which is what we
can do by means of a severity assessment of.

32. 32
(2) Subjective degree of belief assignments will not ensure the error
probability, and thus the severity, assessments we need.
(3) Some suggest an “impartial” or “uninformative” Bayesian prior gives
.5 to H0, the remaining .5 probability being spread out over the alternative
parameter space, Jeffreys.
This “spiked concentration of belief in the null” is at odds with the
prevailing view “we know all nulls are false”.

33. 33
Upshot: However severely I might wish to say that a hypothesis H
has passed a test, the Bayesian critic assigns a sufficiently low prior
probability to H so as to yield a low posterior probability in H.
But this is no argument about why this counts in favor of, rather than
against, their Bayesian computation as an appropriate assessment of
the warrant to be accorded to hypothesis H.
To begin with, in order to use techniques for assigning frequentist
probabilities to events, their examples invariably involve
“hypotheses” that consist of asserting that a sample possesses a
characteristic, such as “having a disease” or “being college ready” or,
for that matter, “being true.”
This would not necessarily be problematic if it were not for the fact
that their criticism requires shifting the probability to the particular
sample selected

34. 34
Bayesians sometimes tell us they will cure the significance tester’s
tendency to exaggerate the evidence against the null (in two-sided
testing) by using some variant on a spiked prior.
But the result of their “cure” is that outcomes may too readily be
taken as no evidence against, or even evidence for, the null
hypothesis, even if it is false.
We actually don’t think we need a cure.
Faced with conflicts between error probabilities and Bayesian
posterior probabilities, the error statistician may well conclude that
the flaw lies with the latter measure.