Variational inference using implicit distributions aims to relax the strong parametric assumptions of traditional variational inference. Implicit distributions can model more complex distributions and are defined by their ability to sample from and take derivatives of samples. The key challenges with implicit distributions are that the evidence lower bound (ELBO) must be approximated differently, using density log-ratios. The paper proposes methods to optimize the ELBO for implicit distributions using prior-contrastive forms, adversarial training similar to generative adversarial networks, and learning from denoiser gradients.

1. Variational Inference
using Implicit
Distributions
by Ferenc Huszar
MUPI journal club
2018-10-04

2. Sources
● arXiv:
Ferenc Huszár: Variational Inference using Implicit Distributions
https://arxiv.org/abs/1702.08235 (Feb 2017 version)
● blog posts:
https://www.inference.vc/variational-inference-with-implicit-probabilistic-model
s-part-1-2/

3. Reminder: VI & ELBO
approximation to
posterior p(z|x)
David M. Blei, Alp Kucukelbir, Jon D. McAuliffe: Variational Inference: A Review for Statisticians
(https://arxiv.org/abs/1601.00670)

4. Explicit vs implicit
● The parametric assumptions we make in VI are often too strong.
● Implicit models would be one way to relax these.
● We can model more complicated distributions.
vs
https://en.wikipedia.org/wiki/Normal_distribution
Implicit distributions:
● can sample from
● cab take derivatives
of samples w.r.t.
params

5. Explicit vs implicit
● The parametric assumptions we make in VI are often too strong.
● Implicit models would be one way to relax these.
● We can model more complicated distributions.
vs
https://en.wikipedia.org/wiki/Normal_distribution
For example 휺 ~ N(0,1)
휺
Neural
network
Implicit distributions:
● can sample from
● cab take derivatives
of samples w.r.t.
params

6. Notation; what is implicit/explicit
we usually would not
see these parametrized
“If q or p are implicit the ELBO needs
to be approximated differently,
e.g., in terms of density log-ratios”

7. [1] Prior contrastive form of ELBO
[2] Joint contrastive form of ELBO
const
prior

8. [1] Gradient of L (so r and s) w.r.t. 흍
forward model has to
be explicit
p(z) and q
can be implicit
this we can optimize by
reparametrizing
z = g흍
(x, 휺)
reminder:
this ratio we
approximate with
and learn

9. [1] Gradient of L (so r and s) w.r.t. 흍
Observation: gradient at some position 흍0
can be simplified so r/s does not depend on 흍:
reminder:
Optimization:
● Update
● Update ELBO

10. notation:
Update log ratio
reminder:
logistic regression
empirical loss:
y = -1 ⇔ z ~ p
y = +1 ⇔ z ~ q
Learn log ratio by optimizing
logistic regression loss

11. Prior-contrastive adversarial VI
one step of ELBO
optimization using
reparametrization
z = g흍
(x, 휺)
K steps of fitting

12. Translate to GANs
● discriminator
● generator G = g흍
(x, 휺)
● training mode similar to GANs - K steps for D vs 1 step for G
● ⇾ adversarial variational Bayes

13. Denoiser-guided learning
ELBO:
approximate with
samples from q
reparametrization
chain rule
ok
ok ?
derivative

14. Learn
● Trick: denoiser solution contains gradient of generating distribution:
● Find denoiser numerically by optimizing:
● Extract gradients:
denoiser
added noisegenerating distribution

15. Denoiser-guided learning
ELBO:
approximate with
samples from q
reparametrization
chain rule
ok
ok
gradient from
denoiser
derivative

16. Full algorithm
one step of ELBO
optimization using
formula for
K steps of learning
denoisers

17. Results