Bayesian 1-way anova using Bayes factor, compared to frequentist 1-way anova with validation on models and data visualization

1. On construction of frequentist and Bayesian
ANOVA comparison
(STATS/CSE 780 course project)
Kyuson Lim
Department of Mathematics and Statistics
McMaster University
04/05/2022

2. Outline
I Motivation
I Data
I Exploratory Data Analysis (Summary)
I Exploratory Data Analysis (Data Visualization)
I Method
I Methods: Frequentist Anova
I Methods: Bayesian Anova
I Result
I Discussion

3. Motivation (Makowski et al., 2019)
I What is the problem?
I Reliance on the strict assumptions (normality, homogeneity of
variance, independence of samples).
I Frequentist framework: misuse of uniform p-values, changes by
sample size (Chambers et al., 2014).
I What are the two methods? (Andrews & Baguley, 2013)
I Baeysian ANOVA: use Bayes factor, BF12 = p(D|M1)
p(D|M2) from
p(M1|D)
p(M2|D) = p(D|M1)
p(D|M2) × p(M1)
p(M2) to provide the evidence.
I Frequentist ANOVA: use F-statistic given by F0 = MSR
MSE .
MSR = SSR/(k − 1) and MSE = SSE/(n − k) are mean sum
of squares between and within groups.
I Why does this problem important?
I Probability as a degree of belief: given the observed data
computed, posterior distribution is derived.
I Goal of presentation
I Apply Bayesian ANOVA and compare with frequentist ANOVA.
I Understand difference in two sample t-test.

4. Exploratory Data Analysis (Summary)
I Source: Glass Identification Data Set, UCI ML repository
I Observation (Spiehler, 1987):
I Total of 214 glass collected.
I 10 variables, 9 continuous variables, and 1 categorical variable.
I Outliers (82 values) are ommited by RI and Mg composition.
I Two types of 132 observations are compared against RI values.
I 67 building window glass types.
I 65 non-building window glass types.
I Results of exploratory analysis:
I All continuous variables are standardized (scaled).
I Non-window glass types contain considerable number of outliers.
I A classification is inappropriate by data visualization.
I A regression method is rejected by heterogeneity of glass types.
I Building window: correlation between Mg and RI is 0.42.
I Non-building window: correlation between Mg and RI is -0.39.
I Frequentist assumption:
I Shapiro-Wilk test: p-value (1.192e-07) < 0.1, violate normality.
I Bartlett’s test: p-value (0.08) < 0.1, violates homogeneity.

5. Exploratory Data Analysis (Data Visualization)
Figure 1: Source: Glass Identification Data Set, UCI ML repository
I Figure 1. considers classification and regression problems.
I Figure 3. shows the distribution differences for 1-way ANOVA.

6. Methods: Frequentist Anova (Dean et al., 1999)
I Purpose and reason for the comparison (1-way ANOVA)
I Frequentist: robust to 2-sample problem, limited hypothesis
testing, reproducibility issue, estimation for uncertainty.
I Bayesian: flexibility, consideration for prior odds to the posterior
odds, quantify relative support between competing hypothesis.
I Frquentist framework: F statistic compares the variability
between the groups to the variability within the groups.
I The H0 : µ1 = µ2 is rejected if F0 > Fα/2,k−1,n−k or p-value
P(F > F0)
is less than α.
I Welch’s t-test: t = X̄1−X̄2
sp
p 1
n1
+ 1
n2
or t =
¯
X1− ¯
X2
√
s2
1 /n1+s2
2 /n2
.
I Frequentist ANOVA (aov): fit an ANOVA Model
(generalization of two-sample t-test).
I t-test (t.test): compute independent t-test.

7. Methods: Bayesian Anova (Rouder et al., 2009)
I Bayesian framework: Bayes Factor quantifies relative
probability of the observed data under each of the two models,
which is transitive BF12 = p(D|M1)
p(D|M2) = p(D|M1)/p(D|M0)
p(D|M2)/p(D|M0) = BF10
BF20
.
I Model comparison very flexible and robust to choice of prior.
I Bayesian t-test compares models, yi ∼ N(µ ± α
2 , σ2
) with δ ∼
Cauchy and non-informative prior on σ2
, p(σ2
) = 1
σ2 ,p(µ) = 1.
I Bayesian ANOVA (anovaBF): compute Bayes factors for
ANOVA designs (Morey et al., 2015).
I Model: y = µ + x1θ1 + , θ1 ∼ N(0, g1σ2
)
I Jeffrey’s prior places on µ, σ2
, scaled-Inv χ2
(1) prior on g1.
I Bayes factors are computed by integrating the likelihood.
I The integration of g1 is performed through Monte Carlo
sampling (Rouder et al., 2012).
I t-test (ttestBF): allows us to perform model comparison
(same result in two-sample case to anovaBF).
I Inclusion Bayes factors (BMA): examines the term’s inclusion
Bayes factor (Hinne et al., 2019).

8. Results
I Frequentist ANOVA (aov): exists significant difference
between glass types for an equality for means of RI.
I t-test (ttest): agrees with result of 1-way ANOVA.
I Bayesian ANOVA (BF): with uniform (0,1) prior, there is a
posterior probability of 0.99 on statistical difference.
I Bayesian Model Averaging (anovaBF): model with term on
average, 0.143 ( 1
7.01 ) less supported than model without term.
I t-test (ttestBF): strong evidence (1110 times) for alternative
over null (µ1 − µ2 = 0) hypothesis (Makowski et al.,2019).

9. Discussion
I Limitation: performance with number of samples .
I Frequentist approach: robust with violation of assumptions,
changes by sample size from formula (Maarten, 2021).
I Bayesian approach: ANOVA is restricted to classification
difference, accurate with small samples (Kruschke et al., 2012).
I Interpretablity:
I Frequentist approach: overstate the evidence against a null
hypothesis by the p-value.
I Bayesian: simply describe the posterior distribution of the
effect, widely use for Bayesian (Kruschke, 2010).
I Reproducibility:
I Frequentist: Rejection regions are not suitable for
underpowered experiments (Maarten, 2021).
I Stability:
I Frequentist: return a point-estimate, estimate the real effect
(correlation between variables) (Dudek, 2021).
I Bayesian: no assumption, returns posterior of the effect that is
compatible with the observed data (Wagenmakers et al., 2018).

10. References
I Makowski, D., Ben-Shachar, M. S., Lüdecke, D. (2019). bayestestR: Describing effects and their
uncertainty, existence and significance within the Bayesian framework. Journal of Open Source Software,
4(40), 1541.
I Morey, R. D., Rouder, J. N., Jamil, T., Morey, M. R. D. (2015). Package ‘bayesfactor’. URLh
http://cran/r-projectorg/web/packages/BayesFactor/BayesFactor pdf i (accessed 1006 15).
I Chambers, C. D., Feredoes, E., Muthukumaraswamy, S. D., Etchells, P. (2014). Instead of” playing the
game” it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond. AIMS
Neuroscience, 1(1), 4-17.
I Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., Iverson, G. (2009). Bayesian t tests for accepting
and rejecting the null hypothesis. Psychonomic bulletin review, 16(2), 225-237.
I Rouder, J. N., Morey, R. D., Speckman, P. L., Province, J. M., (2012) Default Bayes Factors for ANOVA
Designs. Journal of Mathematical Psychology. 56. p. 356-374.
I Etz, A., Vandekerckhove, J. (2016). A Bayesian perspective on the reproducibility project: Psychology.
PloS one, 11(2), e0149794.
I Get started with bayesian analysis. bayestestR. (2019). Retrieved April 5, 2022, from
https://easystats.github.io/bayestestR/articles/bayestestR.html
I Andrews, M., Baguley, T. (2013). Prior approval: The growth of bayesian methods in psychology. British
Journal of Mathematical and Statistical Psychology, 66(1), 1–7.
I Dudek, B. (2021). One Way ANOVA with R. Retrieved April 5, 2022, URLh
https://bcdudek.net/anova/bayesian-inference-for-1-way-anovas.html
I Maarten, S. (2021). An R companion to statistics: Data Analysis and modelling. Statistics: Data analysis
and modelling . Retrieved April 5, 2022, from
https://mspeekenbrink.github.io/sdam-r-companion/index.html
I Kruschke, J. K. (2010). What to believe: Bayesian methods for data analysis. Trends in Cognitive Sciences,
14(7), 293–300.
I Kruschke, J. K., Aguinis, H., Joo, H. (2012). The time has come: Bayesian methods for data analysis in
the organizational sciences. Organizational Research Methods, 15(4), 722–752.
I Wagenmakers, E. J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., . . . Morey, R. D. (2018).
Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications. Psychonomic
bulletin review, 25(1), 35-57.
I Hinne, M., Gronau, Q. F., van den Bergh, D., and Wagenmakers, E. (2019). A conceptual introduction to
Bayesian Model Averaging. doi: 10.31234/osf.io/wgb64
I Dean, A., Voss, D. and Draguljic, D. (1999). Design and analysis of experiments. New York: Springer.
I Spiehler, V. (1987). Glass Identification Data Set. UCI Machine Learning Repository: Glass identification
data set. from https://archive.ics.uci.edu/ml/datasets/glass+identification