This document discusses issues with the foundations of statistical analysis and modeling. It argues that statistical analysis often makes the wrong assumption that data is randomly generated by a probabilistic model. Additionally, there is too much focus on technical statistical details rather than providing approximate solutions that are useful to non-statistician users. The document advocates for a more subjective Bayesian approach that embraces uncertainty and variation rather than relying on tests and rigid models. It also calls for statistical analyses to be more transparent by explicitly stating all assumptions and modeling choices.

1. beyond objective and subjective statistics
a discussion
Christian P. Robert
Universit´e Paris-Dauphine (CEREMADE)
& University of Warwick (Dept. of Statistics)
http://xianblog.wordpress.com,seriesblog.net
Royal Statistical Society, April 2017

2. the elephant in the room...
Statistical analysis invariably starts
with the unquestioned premise of the
random nature of the data
which differs from the assumption of a
probabilistic model generating the
data [not much discussed therein]
& connects with repeatability of
observations, which is almost always
wrong

3. the elephant in the room...
Statistical analysis invariably starts
with the unquestioned premise of the
random nature of the data
which differs from the assumption of a
probabilistic model generating the
data [not much discussed therein]
& connects with repeatability of
observations, which is almost always
wrong
How do we address this blatantly wrong start?!
[perspective that led Keynes to abandon statistics and move to
economics!]

4. ...and the tortoise in the next room
Is focus on wrong issue
arguing between ourselves about the best way to solve the
wrong problem,
while users seek approximate solutions with some modicum of
efficiency
i.e., satisfied with imprecise inference

5. ...and the tortoise in the next room
Is focus on wrong issue
arguing between ourselves about the best way to solve the
wrong problem,
while users seek approximate solutions with some modicum of
efficiency
i.e., satisfied with imprecise inference
E.g., how many statistical problems “solved” Amazon in a
day, compared with uncovering new fundamental particles???

6. a forensic illustration
Vote on 10 April by US National Commission on Forensic Science
on how forensic analysts should testify about evidence:
Analysts must
explain
how they examined evidence
what statistical analyses they
chose
inherent uncertainties in their
measurements
[R. Mejia, Nature, 4 April 2017]

7. a forensic illustration
Vote on 10 April by US National Commission on Forensic Science
on how forensic analysts should testify about evidence:
Analysts must
never claim with certainty that
anything on a crime scene is
linked to a suspect
quantify the probability that
observed similarities occurred by
chance
[R. Mejia, Nature, 4 April 2017]

8. the ultimate issue with statistics
“...the ultimate
inaccessibility of a reality
that is truly independent of
observers is a basic human
condition.” A. Gelman
& C. Hennig
Except for the most [basic] scientific
settings, there is not reality behind
statistical models [Box], hence an
inaccessible consensus is the rule

9. ”Keynesians who focus on more subjective factors...”
Read in John Maynard Keynes’s A Treatise on Probability (1921):
“...where general statistics are
available, the numerical
probability which might be
derived from them is inapplicable
because of the presence of
additional knowledge with
regards to the particular case...”

10. ”Keynesians who focus on more subjective factors...”
Read in John Maynard Keynes’s A Treatise on Probability (1921):
“...until a prima facie case has
been established for the existence
of a stable probable frequency,
we have but a flimsy basis for
any statistical induction...”

11. models, models, models
missing: trial-and-error way of
building a statistical model
and/or analysis,
while subjective inputs from
operator(s) and regulators found
at all stages of construction
and should be spelled out rather
than ignored (or rejected)

12. 99% of the Universe
discussion on foundations (§5) thorough [objective judgement!] but
does not expand on issue of “default” or all-inclusive
statistical solutions
used through “point-and-shoot” software by innumerate
practitioners
while under impression of conducting a statistical analysis

13. 99% of the Universe
discussion on foundations (§5) thorough [objective judgement!] but
does not expand on issue of “default” or all-inclusive
statistical solutions
used through “point-and-shoot” software by innumerate
practitioners
while under impression of conducting a statistical analysis
false feeling also occurs in treatment of statistical expertise by
media, courts, and scientific journals

14. 99% of the Universe
discussion on foundations (§5) thorough [objective judgement!] but
does not expand on issue of “default” or all-inclusive
statistical solutions
used through “point-and-shoot” software by innumerate
practitioners
while under impression of conducting a statistical analysis
false feeling also occurs in treatment of statistical expertise by
media, courts, and scientific journals
[We are the 1%!!!]

15. Relativity
Pertains to awareness of multiple perspectives but also to stability
Fundamental dependence of inferential output on statistical
framework(s)
difference in outcomes perfectly acceptable
inner assessment of model feasible by producing pseudo data
comparison of frameworks only to the extent of predictive
characteristics

16. Relativity
Pertains to awareness of multiple perspectives but also to stability
Bayesian analysis [both objective and subjective] well-suited to
this purpose / rescued by relativity
move to quantum theory where objectivity is impossible due
to the presence of the observer/experimenter [??]
reproducibility fraught with danger: production of similar
results depends on rigid framework, relates to fact that
statistics not experimental science

17. rise of the machines
relevance of machine
learning for [model-free]
learning
dismissal of
machine-learning perspective
disappointing
given robustness [against
models] of machine learning
prediction
...if missing in uncertainty
assessment

18. the five pillars of statistical wisdom
frequentism, subjective about the Universe and its probability
distribution, and about the choice of an inference procedure,
with falsifiability only achievable by replicating data via
simulation, but this only rejects poor models
[agreeing with Davies, 2014]

19. the five pillars of statistical wisdom
frequentism, subjective about the Universe and its probability
distribution, and about the choice of an inference procedure,
with falsifiability only achievable by replicating data via
simulation, but this only rejects poor models
[agreeing with Davies, 2014]
subjective Bayesianism is quite objective and open about its
subjectivity or relativity, working in parallel universes that do
not need to intersect, provided they produce such universes
(discussion seems to swerve towards a common prior principle
that I do not understand!)

20. the five pillars of statistical wisdom
subjective Bayesianism is quite objective and open about its
subjectivity or relativity, working in parallel universes that do
not need to intersect, provided they produce such universes
(discussion seems to swerve towards a common prior principle
that I do not understand!)
very subjective concept of objective Bayes when it reduces to
Jaynes’s! and missing the point of “objective” Bayes
principles which is not to be unique but rather define a generic
principle of derivation of a prior from the likelihood function
without a particular justification

21. the five pillars of statistical wisdom
very subjective concept of objective Bayes when it reduces to
Jaynes’s! and missing the point of “objective” Bayes
principles which is not to be unique but rather define a generic
principle of derivation of a prior from the likelihood function
without a particular justification
I do not feel the gap between the above and the falsificationist
Bayesian perspective expressed in Section 5.5, hence sounds
very subjective to me. Unless the school reduces to Gelman
and Shalizi (2013)?!

22. embracing uncertainty with a subjective hug
basic realism and uncertain nature of data call for an absence of
hard decisions like tests and model choices, but rather for
descriptive performances of the suggested procedures, accepting
imperfection and variability in the answer produced locally
“Its not so hard to move away
from hypothesis testing and
toward a Bayesian approach of
embracing variation and
accepting uncertainty.” A.
Gelman

23. conclusion
exposes the need to spell out the various inputs leading to a
statistical analysis
reinforces the call for model awareness:
critical stance on all modelling inputs, including priors!,
disbelief that any model is true
potential if realistic outcome would be to impose not only
production of all conscious choices but also through the
posting of (true or pseudo-) data and of relevant code for all
publications involving a statistical analysis

24. conclusion
proposal too idealistic in that most users (and most makers)
of statistics cannot or would not spell out their assumptions
and choices, being unaware of or unapologetic about those
central difficulty with statistics as a service discipline, namely
that almost anyone anywhere can produce an estimate or a
p-value without ever being proven wrong
how epistemological argument here going to profit statistical
methodology
add layers of warning that the probability behind the model is
not connected with the phenomenon

25. a debate to be continued, hopefully
See you at O’Bayes17:
International Workshop on Objective Bayes Methodology
held in Austin, Texas, Su, December 10 through We, December
13, 2017
[https://sites.google.com/site/obayes2017/]