To infinity and beyond

To Infinity and Beyond
lessons for big data from small
experiments
Stephen Senn, Consultant Statistician, Edinburgh
(c) Stephen Senn 2019 1
Norwich 20 September 2019

Acknowledgements
My thanks for the kind invitation
This work is partly supported by the European Union’s 7th Framework Programme
for research, technological development and demonstration under grant
agreement no. 602552. “IDEAL”.
Work on historical controls is joint with Olivier Collignon, Anna Schritz and Riccardo
Spezia and will appear in Statistical Methods in Medical Research

An anniversary and
an excuse
• RA Fisher 1890-1962
• Statistician at Rothamsted agricultural
station 1919-1933 (Started in October)
• Most influential statistician ever
• Also a major figure in evolutionary biology
• Developed theory of small sample
inference and many modern concepts
• Likelihood, variance, sufficiency, ANOVA
• Developed theory of experimental design
• Blocking, Replication, Randomisation
• Genstat® is the only statistical package, as
far as I know, that reflects block structure
properly
This is the 100th anniversary of Fisher’s
arrival at Rothamsted research station

Outline
• The Rothamsted School
• Why block structure matters
• An example analysed with the help of the Rothamsted School and Genstat®
• The TARGET study – a practical example
• Historical controls – a warning
• Lord’s paradox – a misunderstanding
• Conclusions and lessons

The Rothamsted School
RA Fisher
1890-1962
Variance, ANOVA
Randomisation, design,
significance tests
Frank Yates
1902-1994
Factorials, recovering
Inter-block information
John Nelder
1924-2010
General balance, computing
Genstat®
and Frank Anscombe, David Finney, Rosemary Bailey, Roger Payne etc

A quote from John Nelder 1965
pp 147-148

Some jargon 1
• Outcomes
• What we measure at the end of a trial and regard as being relevant to judging the
effect of treatment
• Treatment
• What the experimenter varies
• Caution: Sometimes we refer to treatment as being a factor that has two or more levels (for
example beta-blocker or placebo) but sometimes, confusingly we may refer to one of the
levels as treatments .(For example, treatment versus placebo.)
• Analogy: geneticists sometimes use gene to mean locus and sometimes to mean allele . (The
gene for earwax or the gene for wet-type earwax.)
• Covariate
• Something else that may predict outcomes and can be measured before the trial
starts

Some jargon 2
• Unit
• That which is treated from the experimental point of view: usually patients,
but it could be centres or it might be episodes in the life of a patient
• Allocation algorithm
• The way that treatments are allocated to units (for example to patients)
• Blocking factor (or sometimes block)
• A particular type of covariate that can be recognised and accounted for in the
allocation process
• For example, centre We can choose to ‘block’ treatments by centre. We try make sure
that (say) equal numbers of patients within a given centre receive two treatments that
are being compared

What does the Rothamsted approach do?
• Matches the allocation procedure to the analysis. You can either
regard this as meaning
• The randomisation you carried out guides the analysis
• The analysis you intend guides the randomisation
• Or both
• Either way, the idea is to avoid inconsistency
• Regarding something as being very important at the allocation stage but not
at the analysis stage is inconsistent
• Permits you not only to take account of things seen but also to make
an appropriate allowance for things unseen
• The way the treatment structure maps onto the block structure is key

Trial in asthma
Basic situation
• Two beta-agonists compared
• Zephyr(Z) and Mistral(M)
• Block structure has several levels
• Different designs will be investigated
• Cluster
• Parallel group
• Cross-over Trial
• Each design will be blocked at a different
level
• NB Each design will collect
6 x 4 x 2 x 7 = 336 measurements of Forced
Expiratory Volume in one second (FEV1)
Block structure
Level Number
within higher
level
Total
Number
Centre 6 6
Patient 4 24
Episodes 2 48
Measurements 7 336

Block structure
• Patients are nested within centres
• Episodes are nested within patients
• Measurements are nested within
episodes
• Centres/Patients/Episodes/Measurements
Measurements not shown

Possible designs
• Cluster randomised
• In any given centre all the patients either receive Zephyr (Z) or Mistral (M) in
both episodes
• Three centres are chosen at random to receive Z and three to receive M
• Parallel group trial
• In each centre half the patients receive Z and half M in both episodes
• Two patients per centre are randomly chosen to receive Z and two receive M
• Cross-over design
• Each patient is given both treatments.
• M is received in one episode and Z in another
• The order of allocation, ZM or MZ is random

Null (skeleton) analysis of variance with Genstat ®
Code Output
BLOCKSTRUCTURE Centre/Patient/Episode/Measurement
ANOVA

Full (skeleton) analysis of variance with Genstat ®
Additional Code Output
TREATMENTSTRUCTURE Design[]
ANOVA
(Here Design[] is a pointer with values corresponding
to each of the three designs.)

The bottom line
• The approach recognises that things vary
• Centres, patients episodes
• It does not require everything to be balanced
• Things that can be eliminated will be eliminated by design
• Cross-over trial eliminates patients and centres
• Parallel group trial eliminates centres
• Cluster randomised eliminates none of these
• The measure of uncertainty produced by the analysis will reflected
what cannot be eliminated
• This requires matching the analysis to the design

A key lesson from the Rothamsted school
• Variances matter
• If you can’t say how precise your estimates are
• You don’t know what to do with them
• You don’t know if you need more information
• You can’t combine them with other information
• You cannot make useful inferences
• For example Bayesians cannot update their prior distributions
• You are in danger of mistaking the best supported for the probable
• To think about ‘point estimates’ only is a fundamental mistake
• Starting from this position leads to error

Variance matters
Points
• Which variances apply depends on
the design
• All three for cluster trial
• Last two for parallel trial
• Third only for cross-over trial
• It is possible for the number of
observations to go to infinity
without the variance going to zero
• There is no ‘design-free’ n
• There is no design-free asymptotic
inference
Variances
 
 
 
2
2
2
, centres
, patients per centre
, episodes per patient
1)
2)
3)
C
P
E
C C
P C P
E C P E
n
n
n
n between centre contribution
n n between patient contribution
n n n within patient contribution




What about the measurement level?
• I put this in to remind us that not
everything you measure brings
exploitable information to the same
degree
• Randomisation between
measurements was not possible in
any of the schemes
• This makes it difficult to exploit
measurements except in a summary
way
• For example, by averaging
• Warning: some repeated measures
analyses are very strongly reliant on
assumed model structure
 
   
1 6
1 12
1 7
1
6 1
7
2 2
1
1
1
, ,
1
, , usually
7 7 7
M
i
i
Y Y Var
Y
Y Var Y Var Y
 
 


 
 
 
 
  
 
 
 
 

Y Y

The TARGET study
• One of the largest studies ever run in osteoarthritis
• 18,000 patients
• Randomisation took place in two sub-studies of equal
size
• Lumiracoxib versus ibuprofen
• Lumiracoxib versus naproxen
• Purpose to investigate CV and GI tolerability of
lumiracoxib

Baseline Demographics
Sub-Study 1 Sub Study 2
Demographic
Characteristic
Lumiracoxib
n = 4376
Ibuprofen
n = 4397
Lumiracoxib
n = 4741
Naproxen
n = 4730
Use of low-dose
aspirin
975 (22.3) 966 (22.0) 1195 (25.1) 1193 (25.2)
History of
vascular disease
393 (9.0) 340 (7.7) 588 (12.4) 559 (11.8)
Cerebro-
vascular disease
69 (1.6) 65 (1.5) 108 (2.3) 107 (2.3)
Dyslipidaemias 1030 (23.5) 1025 (23.3) 799 (16.9) 809 (17.1)
Nitrate use 105 (2.4) 79 (1.8) 181 (3.8) 165 (3.5)

Baseline Chi-square P-values
Model Term
Demographic
Characteristic
Sub-study
(DF=1)
Treatment
given Sub-
study
(DF=2)
Treatment
(DF=2)
Use of low-dose
aspirin
< 0.0001 0.94 0.0012
History of
vascular disease
< 0.0001 0.07 <0.0001
Cerebro-
vascular disease
0.0002 0.93 0.0208
Dyslipidaemias <0.0001 0.92 <0.0001
Nitrate use < 0.0001 0.10 <0.0001

Outcome Variables
Lumiracoxib only
Sub-Study 1 Sub Study 2
Outcome
Variables
Lumiracoxib
n = 4376
Lumiracoxib
n = 4741
Total of
discontinuations
1751
(40.01)
1719
(36.26)
CV events 33
(0.75)
52
(1.10)
At least one AE 699
(15.97)
710
(14.98)
Any GI 1855
(42.39)
1785
(37.65)
Dyspepsia 1230
(28.11)
1037
(21.87)

Deviances and P-Values for Outcomes
Lumiracoxib only fitting Sub-study
Statistic
Outcome
Variables
Deviance Chi-
square
P-Value
Total of
discontinuations
37.43 < 0.0001
CV events 0.92 0.33
At least one AE 0.005 0.94
Any GI 0.004 0.95
Dyspepsia 16.85 < 0.0001

Lessons from TARGET
• If you want to use historical controls you will have to work very hard
• You need at least two components of variation in your model
• Between centre
• Between trial
• And possibly a third
• Between eras
• What seems like a lot of information may not be much

2
2
22
22
2
,
historical studies
patients per historical study
patients in current study
γ between study variance
σ between patient variance
lim
c
h
c
h
c
k n
k
n
n
n
nk




 
          
 

Lord’s Paradox
Lord, F.M. (1967) “ A paradox in the interpretation of
group comparisons”, Psychological Bulletin, 68, 304-
305.
“A large university is interested in investigating the effects on the students
of the diet provided in the university dining halls….Various types of data
are gathered. In particular the weight of each student at the time of his
arrival in September and his weight in the following June are recorded”
We shall consider this in the Wainer and Brown version (also considered
by Pearl & McKenzie) in which there are two halls each assigned a
different one of two diets being compared.

Two Statisticians
Statistician One (Say John)
• Calculates difference in weight
(outcome-baseline) for each hall
• No significant difference
between diets as regards this
‘change score’
• Concludes no evidence of
difference between diets
Statistician Two (Say Jane)
• Adjusts for initial weight as a
covariate
• Finds significant diet effect on
adjusted weight
• Concludes there is a difference
between diets

John’s analysis:
comparing
change-scores)

Jane’s analysis:
Comparing covariate
adjusted scores

Pearl & Mackenzie, 2018
D
(Diet)
WF
W1 “However, for statisticians who
are trained in ‘conventional’
(i.e. model-blind) methodology
and avoid using causal lenses,
it is deeply paradoxical “
The Book of Why p217
“In this diagram, W1, is a
confounder
of D and WF and not a
mediator. Therefore, the
second statistician would
be unambiguously right
here.”
The Book of Why p216
NB This diagram adapted from theirs,
which covers change rather than final
weight.

Start with the randomised equivalent
• We suppose that the diets had been randomised to the two halls
• Le us suppose there are 100 students per hall
• Generate some data
• See what Genstat® says about analysis
• Note that ( as we have seen) it is a particular feature of Genstat® that
it does not have to have outcome data to do this
• Given the block and treatment structure alone it will give us a
skeleton ANOVA
• We start by ignoring the covariate

Skeleton ANOVA
BLOCKSTRUCTURE Hall/Student
TREATMENTSTRUCTURE Diet
ANOVA
Analysis of variance
Source of variation d.f.
Hall stratum
Diet 1
Hall.Student stratum 198
Total 199
Code Output
Gentstat® points out the obvious (which, however, has
been universally overlooked). There are no
degrees of freedom to estimate the variability of the
Diet estimate which appears in the Hall and not the
Hall.Student stratum

Adding initial weight as a covariate
BLOCKSTRUCTURE Hall/Student
TREATMENTSTRUCTURE Diet
COVARIATE Base
ANOVA
Analysis of variance (adjusted for covariate)
Covariate: Base
Hall stratum
Diet 0
Covariate 1
Residual 0
Hall.Student stratum
Covariate 1
Residual 197
Total 199
Code Output
Again Gentstat® points out the obvious (which, however, has
been universally overlooked). There are no degrees of freedom
to estimate the treatment effect because the single degree of
freedom is needed to estimate the between-hall slope.

Conclusion
• The solution of The Book of Why is only correct if the between-hall
regression is the same as the within-hall regression
• Even if that assumption is correct the calculated standard error would
almost certainly be wrong
• Would require the assumption that there is no variance between-halls above
and beyond that predicted by the variance between students
• To treat the n as the number of students rather than the number of
halls is to commit the fallacy of pseudoreplication (Hurlbert, 1984)
• This is precisely what the Rothamsted School approach is designed to
avoid

A simulation to illustrate the problem
• Generate 20 Lord’s paradox cases
• Each of the sort that is addressed in The Book of Why
• Each consists of two halls with diet 1 in one hall and diet 2 in the other
• Impose the strong assumption that The Book of Why implicitly
assumes
• Set between-hall variance to zero
• See if a consistent message is repeated from case to case among the 20 cases
• Repeat the exercise, violating the strong assumption
• Make between-hall variance large and allow for a between-hall covariance
• See if a consistent message is repeated from case to case among the 20 cases

No hall effect Random hall effect

The two cases compared
The implicit assumption is true
Simulation 1
The implicit assumption is false
Simulation 2
Critical values reflect
Bonferroni values (one-
sided) of
1 40 20 = 1 800

Making ‘Hall’ part of the treatment structure
Code Output
BLOCKSTRUCTURE Student
TREATMENTSTRUCTURE Hall+Diet
ANOVA
Null ANOVA with Hall as part of the
treatment structure
Analysis of variance
Student stratum
Hall 1
Residual 198
Total 199
Information summary
Aliased model terms
Diet

A simple way of looking at it
Any effect of diet must be on the final weight
Thus the causal message of the diet is transmitted via 2 1Y Y
This estimates the diet effect. The question is, what else does it estimate?
We correct for anything else by using the counterfactual estimate: “what would this
difference show if there were no effect of diets?”
Statistician Counterfactual Assumption
John 𝑋2 − 𝑋1 Tracking over time
Jane 𝛽 𝑋𝑌 𝑋2 − 𝑋1 between-hall regression = within-hall
regression
Senn 2006 (c) Stephen Senn 2019 44

Conclusions
• Local control is valuable
• Design matters
• Components of variation matter
• The Rothamsted approach brings insight
• Causal analysis needs to be developed further to include components of
variation
• Greenland and Mansournia (2015)
• McLaren and Nicholson (2019)
• Kim and Steiner (2019)
• Just because you are rich in data does not mean you are rich in information
• Be sceptical about “big data”

A warning as regards observational studies
Things that are a problem for controlled clinical trials are very rarely less of a
problem for observational studies.
Propensity score, Mendelian randomisation, causal analysis, blah, blah, blah are
all very well but if you aren’t thinking about components of variation, you
should be.
Variances matter. Assuming that, once confounders are adjusted for, that your
uncertainty is proportional to 1 𝑛 is an all too standard error

Finally, I leave you with this thought
A big data-analyst is an expert at producing misleading
conclusions from huge datasets.
It is much more efficient to use a statistician, who can do
the same with small ones.

References
1. Greenland, S. and M.A. Mansournia, Limitations of individual causal models, causal graphs, and ignorability
assumptions, as illustrated by random confounding and design unfaithfulness. European journal of
Epidemiology, 2015. 30(10): p. 1101-1110.
2. Holland PW, Rubin DB. On Lord's Paradox. In: Wainer H, Messick S, editors. Principles of Modern
Psychological Measurement. Hillsdale, NJ: Lawrence Erlbaum Associates; 1983.
3. Hurlbert, S.H., Pseudoreplication and the design of ecological field experiments. Ecological monographs,
1984. 54(2): p. 187-211
4. Lord FM. A paradox in the interpretation of group comparisons. Psychological Bulletin. 1967;66:304-5.
5. Nelder JA. The analysis of randomised experiments with orthogonal block structure I. Block structure and
the null analysis of variance. Proceedings of the Royal Society of London Series A. 1965;283:147-62.
6. Nelder JA. The analysis of randomised experiments with orthogonal block structure II. Treatment structure
and the general analysis of variance. Proceedings of the Royal Society of London Series A. 1965;283:163-78.
7. Pearl J, Mackenzie D. The Book of Why: Basic Books; 2018
8. Senn SJ. Change from baseline and analysis of covariance revisited. Statistics in Medicine.
2006;25(24):4334–44.
9. .Wainer H, Brown LM. Two statistical paradoxes in the interpretation of group differences: Illustrated with
medical school admission and licensing data. American Statistician. 2004;58(2):117-23.

Blogposts related to the talk
https://errorstatistics.com/2019/03/09/s-senn-to-infinity-and-beyond-how-big-are-your-data-really-guest-post/
To infinity and beyond: how big are your data, really?
https://errorstatistics.com/2018/11/11/stephen-senn-rothamsted-statistics-meets-lords-paradox-guest-post/
Stephen Senn: Rothamsted Statistics meets Lord’s Paradox
On the level. Why block structure matters and its relevance to Lord’s paradox
https://errorstatistics.com/2018/11/22/stephen-senn-on-the-level-why-block-structure-matters-and-its-relevance-to-
lords-paradox-guest-post/
Red herrings and the art of cause fishing: Lord’s Paradox revisited
https://errorstatistics.com/2019/08/02/s-senn-red-herrings-and-the-art-of-cause-fishing-lords-paradox-revisited-guest-
post/

To infinity and beyond

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to To infinity and beyond

Similar to To infinity and beyond (20)

More from Stephen Senn

More from Stephen Senn (8)

Recently uploaded

Recently uploaded (20)

To infinity and beyond