A presentation on the statistical topic Multiple Comparison. Contains the motivation, some commonly used methods. Examples included with the R codes.

1. MULTIPLE COMPARISON
Presented by: Riaz Khan

2. Hypothesis Testing
 An indirect form of statistical inference
 We accept or reject a general hypothesis/statement
 H0 : Null/Original Hypothesis
 H1 : Alternate Hypothesis
 Performed based on a significance level, α
Multiple Comparison: Multiple hypothesis testing
simultaneously
2

3. Motivation
 H0 : 𝜃1 = 𝜃2 = 𝜃3
 H1 : All θ’s are not equal
 Null hypothesis rejected by ANOVA
 Which one different?
 One must compare pairwise 0
0.5
1
1.5
2
2.5
A B C
Independent Variable
DependentVariable
0
0.5
1
1.5
2
2.5
A B C
Independent Variable
DependentVariable
0
0.5
1
1.5
2
2.5
A B C
Independent Variable
DependentVariable
0
0.5
1
1.5
2
2.5
3
3.5
A B C
Independent Variable
DependentVariable
H1
0
0.2
0.4
0.6
0.8
1
1.2
A B C
Independent Variable
DependentVariable
H0
3

4. Multiplicity: Simultaneous Inference Problem
 If 𝛼 = 0.05, each pairwise
comparison has 5% chance of
doing wrong rejection.
 Overall error can be
significantly larger than the
nominal 𝛼
 Type I error chance increases
 For 10 comparisons, chance of type
I error = 1 − 0.9510
= 𝟒𝟎. 𝟏%
Type I error : wrong rejection
Type II error : wrong acceptance
4

5. Fixing Multiplicity
 Fix α very small so that overall type I error rate falls below the
pre-specified value (5%)
 For the 10 comparison test, 𝛼 = 0.005
 Decreasing α increases the chance of Type II error rate and
decreases the power of the test as well (Trade Off)
5

6. Techniques of Multiple Comparison
 Some, of a wide number of methods
 Fisher’s Pairwise t-test
 Fisher’s Least Significant Degree of freedom (LSD)
 Tukey’s Honestly Significant Difference (HSD)
 Generalized Linearized Hypothesis Testing
6

7. Fisher’s Pairwise t-Test
Assumptions
 All data independent and
normally distributed
 Variance homogeneity
Analysis in R
 pairwise.t.test {stats}
 Can be utilized with no adjustment,
Bonferroni adjustment, Holm adjustment
and many…
7

8. Fisher’s Pairwise t-Test (cont’d)
Data: flowers
 A subset taken from iris3 data
 50 samples of Setosa, Versicolor and Viginica
 We are interested in the sepal length of the types
## Type 1 = Setosa
## Type 2 = Versicolor
## Type 3 = Virginica
8

9. Fisher’s Pairwise t-Test (cont’d)
No Adjustment
All pairs are different
Bonferroni Adjustment
 Divides the Type I error rate (α) by the
number of tests (in this case, 3).
 Overly conservative.
All pairs still different, but with
different p values
9

10. Fisher’s Pairwise t-Test (cont’d)
Holm Adjustment
 Sequentially reduce the α value
 If there is k hypotheses, the nth
level is given by
𝛼 𝑛 =
𝛼 𝑛𝑜𝑚𝑖𝑛𝑎𝑙
𝑘 − 𝑛 + 1
 Generally considered superior to
Bonferroni adjustment
All pairs still different, but
again with different p values
10

11. Fisher’s LSD
 Very powerful for 3 treatment groups
 Overall type I error control is poor compared to
the t-Test
## LSD.test{agricolae}
 The R package comes with the correction options
as in t-Test (The example here utilizes Holm
correction)
All pairs still different significantly
11

12. Tukey HSD
 Assumptions
 Independence (within and among the groups)
 Groups are normal
 Within group variance equality
 The HSD are calculated from ANOVA
parameters
 Works good for unequal groups
All pairs still different significantly
12

13. General Linearized Hypothesis Testing
 General approach for null hypotheses on
arbitrary parameters
 Each hypothesis is expressed as a linear
combination of all the group (parameter)
 All the hypotheses expressed as a matrix
 For or case, matrix 𝐾 =
1 0 −1
−1 1 0
0 −1 1
Again, all pairs different significantly
13

14. Comments
 The choice of method of method of the
post hoc test depends on the nature of the
problem
 Simple, k = 3, Fisher’s LSD
 k > 3, variance homogeneity, group
inequality, Tukey HSD
 Trying to decrease type I error always
tends to increase type II error and
decrease the power of comparison
(tradeoff)
14