Explaining "Explaining Variational Approximation" by JT Ormerod and MP Wand (2010). I wanted to learn variational methods since its speed for Bayesian inference is just so fast! Here's my condensed version of the paper without the cool examples...you should really try the examples out if you want a better understanding of this method! This presentation assumes some knowledge or experience with Bayesian methods.

1. Explaining “Explaining Variational
Approximation”
Based on Paper
“Explaining Variational Approximation”
JT Ormerod, MP Wand (2010)
Presentation by Wayne Tai Lee

2. My Goal
● Convert the paper into a short presentation
● Not covering the examples (really helpful!)
● Intuition and motivation only

3. Why do we want to use
variational approximations?
● In Statistics, Bayesian solutions always
involve the posterior:
p(Θ|data) = p(data | Θ) p(Θ) / p(data)

4. Why do we want to use
variational approximations?
● p(Θ|data) = p(data | Θ) p(Θ) / p(data)
p(Θ|data) : posterior, belief after updating with data

5. Why do we want to use
variational approximations?
● p(Θ|data) = p(data | Θ) p(Θ) / p(data)
p(Θ|data) : posterior, belief after updating with data
p(data | Θ): likelihood, data generation

6. Why do we want to use
variational approximations?
● p(Θ|data) = p(data | Θ) p(Θ) / p(data)
p(Θ|data) : posterior, belief after updating with data
p(data | Θ): likelihood, data generation
p(Θ): prior, belief before updating with data

7. Why do we want to use
variational approximations?
● p(Θ|data) = p(data | Θ) p(Θ) / p(data)
p(Θ|data) : posterior, belief after updating with data
p(data | Θ): likelihood, data generation
p(Θ): prior, belief before updating with data
p(data): “normalizing constant” to ensure posterior is
a density function

8. Why do we want to use
variational approximations?
● p(Θ|data) = p(data | Θ) p(Θ) / p(data)
p(data | Θ): likelihood, specified by you
p(Θ): prior, specified by you
p(data): a nasty integral that cannot be calculated
explicitly in general

9. Why do we want to use
variational approximations?
● p(Θ|data) = p(data | Θ) p(Θ) / p(data)
p(data | Θ): likelihood, specified by you
p(Θ): prior, specified by you
p(data): a nasty integral that cannot be calculated
explicitly in general
● Consequence:
– Posterior often has no analytical expression

10. Most popular alternative
● To obtain the posterior or any related statistic
– Sample the posterior via MCMC methods

11. Most popular alternative
● To obtain the posterior or any related statistic
– Sample the posterior via MCMC methods
● Pros
– Can get arbitrarily close to the posterior with enough
samples (resource/time intensive)
● Con:
– Lots of tuning necessary
– Time consuming to run

12. Variational Approximation
● Intuition:
– Approximate the posterior with a class of
functions that are easier to deal with
mathematically
– Find the function that minimizes the KL
divergence between the posterior in this
class

13. Variational Approximation
● Intuition:
– Approximate the posterior with a class of
functions that are easier to deal with
mathematically
– Find the function that minimizes the KL
divergence between the posterior in this class
● Pros:
– Suuuuper fast
● Cons:
– No guarantees on closeness

14. Big Picture
Method to get
Posterior
MCMC Variational Method
Strategy Sampling Optimization
Solution Asymptotically Exact Approximation with no bounds
Speed Often slow Fast
The “catch” Tuning and convergence
assessment require experience
Need tractable mathematical
setup

15. Explaining Variational
Approximation
● Change notation: p(y) = p(data)
● Use q(Θ) to approximate p(Θ|y)
● Will assume family of functions for q(Θ)
– q(Θ) = q1(Θ1)q2(Θ2)...qp(Θp)
– Each qi(Θi) is a density

16. Max Lower Bound =
Min KL-Divergence

17. Sanity Check: Optimal Solution is
THE solution
● Optimal q(Θ) is p(Θ|y) for general form:
● Important: this is a very general solution for
arbitrary dependence/distribution of Θ and y
● Product form of q(Θ) allows us to divide and
conquer!

18. Focus on each Θ separately

19. Focus on each Θ separately
+...

20. Focus on each Θ separately
+...

21. Focus on each Θ separately

22. Our assumptions so far
● Product form of q(Θ) allowed us to optimize
each term separately
● qi(Θi) being densities allow
to integrate out nicely

23. How to convert into an optimization
problem that we can solve?

24. We've only learned one trick...

25. We've only learned one trick...
● Optimal q1(Θ1) is then

26. To get a densities, just normalize

27. Unfold our definitions

28. Focus on Θ1

29. Repeat for Θi
● General Mean Field Variational Approximation
Solution:
The density that is proportional to

30. Similarity to Full Conditional in
Gibb Sampling
● Optimal qi(Θi) is proportional to
● Need to do algebra until this is “tractable”
– i.e. something we recognize as a standard
distribution that is easily normalized
– This is where the “setup” comes in important

31. For example
● If
resembles exp(Θi^2 *c) then we know this
must be the Gaussian density!

32. Final Solution
● Product of all qi(θi) is then approximated to
p(θ|y)
● Naturally doesn't do well when there's strong
dependence between the θi
● You Should try the examples in the paper!

33. First Example
● Data generated as
– Y | μ, σ^2 ~ N(μ,σ^2)
● Priors
– μ ~ N(m,s^2)
– σ^2 ~InvGamma(a, b)

34. Gibbs Sampling vs Variational
Samples
● N=100
● N=20

35. Discussion
● Hard to know when the approximation is poor
relative to the true posterior...