The document discusses the development of the Variational Gaussian Process (VGP) as a powerful variational model capable of capturing any continuous posterior distribution, proven through a universal approximation theorem. It highlights the use of deep generative models and variational inference methods to effectively represent complex data. Additionally, an efficient black box algorithm for VGP is introduced within the context of hierarchical variational models.

1. Variational Gaussian
Process
Tran Quoc Hoan
@k09hthaduonght.wordpress.com/
10 February 2016, Paper Alert, Hasegawa lab., Tokyo
The University of Tokyo
Dustin Tran, Rajesh Ranganath, David M.Blei
ICLR 2016

2. 2
Background
p(x) = p(x|z)p(z)
………
………
Parameters
………
………
Parameters
✓
Inference model Generative model
Observations
x
Hidden
variables z
q (z|x)
p✓(x|z)
z ⇠ p✓(z)
In variational auto encoder (VAE), parameters are displayed as neural networks

3. 3
Summary
• Deep generative models provide complex representation
of data
• Variational inference methods require a rich family of
approximating distribution
• They develop a powerful
variational model - the variational
Gaussian process (VGP)
• They prove a universal approximation theorem: the VGP
can capture any continuous posterior distribution.
• They derive an efficient black box algorithm.

4. 4
Variational Models
• We want to compute posterior p(z|x) (z: latent variables, x: data)
• Variational inference seeks to minimize
for a family q(z; )
KL(q(z; )||p(z|x))
• Maximizing evidence lower bound (ELBO)
log p(x) Eq(z; )[log p(x|z)] KL(q(z; )||p(z))
• (Common) Mean-field distribution q(z; ) =
Y
i
q(zi; i)
• Hierarchical variational models
• (Newer) Interpret the family as a variational model for posterior
latent variables z (introducing new latent variables)[1]
Lawrence, N. (2000). Variational Inference in Probabilistic Models. PhD thesis.

5. 5
Gaussian Processes

6. 6
Gaussian Processes

7. 7
Variational Gaussian Processes

8. 8
Variational Gaussian Processes

9. 9
Variational Gaussian Processes
Mean-fields parameters
Induces correlation btw latent variables of the variational model

10. 10
Universal Approximation Theorem

11. 11
Variational Lower Bound
auxiliary model
Variational latent
variable space
Posterior latent
variable space
Data space

12. 12
Auto-Encoding Variational Models
Take both xn, zn as input

13. 13
Black Box Stochastic Optimization

14. 14
Black Box Stochastic Optimization
???

15. 15
Black box inference

16. 16
Experiments

17. 17
Experiments