Priors for BNNs, based on: "Wenzel et al.: What Are Bayesian Neural Network Posteriors Really Like?, 2020", "Noci et al.: Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect, 2021", "Fortuin et al.: Bayesian Neural Network Priors Revisited, 2021" and "Immer et al.: Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning, 2021"

1. Priors in Bayesian Neural
Networks
Tomasz Kuśmierczyk
2022-05-06
Based on:
Wenzel et al.: What Are Bayesian Neural Network Posteriors Really Like?, 2020
Noci et al.: Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect, 2021
Fortuin et al.: Bayesian Neural Network Priors Revisited, 2021
Immer et al.: Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning, 2021

2. DNN vs BNN

3. Reminder: Bayesian inference and learning
network
weights

4. Reminder: Bayesian inference and learning
we can sample from posterior using MCMC or learn it using e.g. VI
likelihood prior

5. How do we learn (=find posteriors) for BNNs?
● (Stochastic Gradient)-MCMCs: Hamiltonian MC, Langevin Dynamics etc.
● Distributional:
○ Laplace approximation
○ VI (via ELBO)
● “Model-specific”
○ MC-Dropout
○ SWAG

6. vs:
Cold posterior effect

7. Cold posterior effect
likelihood prior
vs:

8. trading off the relative influence between the prior term and the likelihood term:
→if the CPE becomes stronger as the relative influence of the prior increases,
this would be an indication that the prior is poor
Bad prior hypothesis
(data size)
Noci et al.: Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect, 2021

9. DNN (SGD trained; no prior) weights
Fortuin et al.: Bayesian Neural Network Priors Revisited, 2021

10. FC BNN: different priors

11. Weights correlations

12. CNN BNN: different priors

13. How can we learn priors?

14. Marginal likelihood optimization / type-II MLE / empirical Bayes
Model selection (e.g. choosing priors) by maximizing log of ML:
● For a fixed model, can be estimated using Laplace approximation with
GGN for Hessian
→ Alternate between model updates and the approximation
Will the approximation capture difference in priors?
Immer et al.: Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning, 2021

15. ● Assume approximation from some (parametric) family of
distributions
● Maximize ELBO wrt its parameters λ
Will posterior capture difference in priors? How to learn so complex priors are
accounted for?
Posterior learning and learning priors: VI

16. ● MCMC e.g. SGHMC:
○ explores the space of parameters and generates set of samples { } from the posterior
○ assumes a fixed energy function, for example,
parametric priors cannot be learned, but, we can think about hierarchical priors:
(Nalisnick et al.: Predictive Complexity Priors, 2020): optimize KL divergence to predictive distribution of a reference model
for hierarchical prior
MCMC?

17. Conclusion
Model selection and learning for BNNs are tied