Probabilistic Programming allows very flexible creation of custom probabilistic models and is mainly concerned with insight and learning from your data. The approach is inherently Bayesian so we can specify priors to inform and constrain our models and get uncertainty estimation in form of a posterior distribution. Using MCMC sampling algorithms we can draw samples from this posterior to very flexibly estimate these models. PyMC3 and Stan are the current state-of-the-art tools to construct and estimate these models. One major drawback of sampling, however, is that it's often very slow, especially for high-dimensional models. That's why more recently, variational inference algorithms have been developed that are almost as flexible as MCMC but much faster. Instead of drawing samples from the posterior, these algorithms instead fit a distribution (e.g. normal) to the posterior turning a sampling problem into and optimization problem. ADVI -- Automatic Differentation Variational Inference -- is implemented in PyMC3 and Stan, as well as a new package called Edward which is mainly concerned with Variational Inference.

1. Don't just sample
optimize
Peadar Coyle

3. Bayesian Neural Networks - Thomas
Wiecki - PyMC3 Docs

4. Challenges in Bayesian
Inference
1. Tradeoffs. How do we formalize statistical and
computational tradeoffs for inference?
2. Software. How do we design efficient and flexible
software for generative models?

5. Why do we need Variational
Inference?
Inferring hidden variables
Unlike MCMC:
​ - Deterministic
- Easy to gauge convergence
- Requires dozens of iterations
Doesn't require conjugacy
Slightly hairier math

6. Background
x
p(x, z)
Given
Data set
Generative model with latent variable
Goal
Infer posterior
z ∈ Rd
p(z∣x)
That is the key problem in Bayesian inference

7. Let's look at the posterior
We can write the conditional or posterior distribution
as
The denominator in the marginal distribution is called
the marginal distribution of observations (also called the
evidence) and it is calculated by marginalizing out the
latent variables from the joint distribution
Often this integral is intractable
p(z∣x) =
p(x)
p(z, x)
p(x) = p(z, x)dz∫z

8. Title Text
Text

9. What do we approximate?
We create a variational distribution over the latent
variables
We want to find settings of
So that q is close to p
When p == q this is plain Expectation Maximization
ν
q(z ∣ν)1:m

10. What does closeness
mean?
We measure the closeness of distributions using Kullback-
Leibler Divergence
If q and p are high we're happy
If KL = 0 , then the distributions are equal
If q is low we don't care. If q isn't high but p isn't we pay a
price
http://bit.ly/2oROYAw​
E [log ]q
p(Z∣x)
q(Z)

11. We can do some
math...
Negative of ELBO (evidence lower
bound) + a constant is equal to KL
divergence
−(E [log p(z∣x)] − E [log q(z)]) + log p(x)q q
Constant
ELBO (in brackets)

12. Key points
Minimizing KL divergence is the same as
maximizing ELBO
This allows us to change a sampling problem into an
optimization problem

13. Whats new in PyMC3
Release of the first stable version in early 2017
Variational Inference
Advanced Hamiltonian Monte Carlo samplers
Easy optimization for finding the MAP point.
Theano support for fast compilation

14. What else is new
Gaussian process kernels
New variants of Variational Inference (including
Operator)
Speed improvements
API and documentation improvements
Bayesian Methods for Hackers - in PyMC3 too

15. First gather data from some real-world
phenomena. Then cycle through :
1. Build a probabilistic model of the
phenomena.
2. Reason about the phenomena given model
and data.
3. Criticize the model, revise and repeat.
Box’s loop