This document discusses different approaches to specifying prior distributions in Bayesian statistics. It begins by introducing the binomial model for coin tossing and how priors and posteriors are calculated. It then describes three categories of Bayesian priors: classical Bayesians use a flat prior, modern parametric Bayesians use a Beta distribution prior, and subjective Bayesians quantify existing knowledge about a process. The document shows that different priors lead to different posteriors. It further explains that any prior density can be approximated by mixtures of Beta densities, and extends this concept to the exponential family. The exponential family conjugate prior is also discussed. Finally, connections are made between the exponential family, Beta density priors, and a generalization about conditional expected posteriors.

1. Discussion about "QUANTIFYING PRIOR OPINION"
paper by P. Diaconis and D.Ylvisaker
Mattia Duma
Collegio Carlo Alberto
April 17, 2015
Professor: Julyan Arbel

2. A simple example: toss a coin. But spinning or
shuffling?
• Let Sn be the number of heads in the first n tosses.
• Let f(p) be the prior distribution
Therefore we will have:
P(Sn = k) =
n
k
1
0
pk
(1 − p)n−k
f(p) dp (1)
While the posterior will be:
f(p|Sn = k) =
pk (i − p)n−k f(p)
1
0 pk (1 − p)n−k f(p) dp
(2)

3. Everything depends on f(p)
Let us distinguish three categories of Bayesians:
• Classical Bayesians: there is a flat prior. Took f(p) = 1.
• Modern Parametric Bayesians: took f(p) as Beta
distribution. We bring back to the binomial model.
• Subjective Bayesians. Take the prior as a quantification
of what is known about the process.
In the last category we may take:
f(p) =
n
i=1
wiβ(ai, bi) (3)
Example: 0.5β(10, 20) + 0.2β(15, 15) + 0.3β(20, 10)

4. With different priors we have different posteriors
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
0.0 0.2 0.4 0.6 0.8 1.0
0.01.02.03.0
Classical Bayesians
x
y
prior
posterior
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
0.0 0.2 0.4 0.6 0.8 1.0
01234
two betas
x
y_1
prior
posterior
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
0.0 0.2 0.4 0.6 0.8 1.0
012345
three betas
x
Y_1
prior
posterior
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
1 2 3
012345
Figure: Let n = 10 and Sn = 3

5. have we gone far enough?
It seems like that in Bayesian statistics you should only slap
down a convenient prior and computing Bayes rule.
But why the prior is Beta distribution? Obviously, is a very
convenient prior but not only:
Theorem
Any f(p), continuos denisity function in [0, 1], can be well
approximated by mixture of Beta densities

6. Extending in a more general case: the Exponential
family
An exponential family model is any model whose densities can
be expressed as
p(y|φ) = h(y)c(φ)eφt(y)
(4)
the conjugate prior distribution for exponential family models is
of the form:
p(φ|n0, n0t0) = κ(n0, t0)c(φ)n0
eφn0t0
(5)
Therefore the following poterior distribution is:
p(φ|y1, . . . , yn) ∝ p(φ|n0 + n, n0t0 + n¯t(y)) (6)

7. Important results about the Exponential family
Theorem
If the prior is of the form whose density is the conjugate prior for
exponential family then:
E[t(Y)] = E[E[t(Y)|φ]]
= E[−c (φ)/c(φ)]
= t0
(7)
Where the result is obtained in the following method:
• to show that [E[t(Y)|φ] = −c (φ)/c(φ) take the derivative
w.r.t. φ of both sides of the equation p(y|φ) = 1
• to show that E[−c (φ)/c(φ)] = t0 integrate by parts and
exploit that p(φ) =1

8. Consequences of the results
If we make an appropriate transformation of the posterior for
the exponential family, we obtain that:
Theorem
E[p(φ|y1, . . . , yn)] =
n0t0 + n¯t(y)
n0 + n
(8)
Theorem
Any density function f(p) can be well approximated by mixture
of prior for exponential family.
An intuition is given thinking about that n0 might be put very
large in order to have the prior concentrates at its mean. And
mixture of point masses might converge to f(p).

9. Bringing back to the original case: from the
Exponential Family to the Beta density
the Exponential Family
p(φ) ∝ c(φ)n0 eφn0t0 E[p(φ)] = t0
p(y|φ) ∝ c(φ)eφt(y)
p(φ|y) ∝ c(φ)n0+n
eφ[n0t0+ t(y)]
E[p(φ)|y] = n0
n0+n
t0 +
¯t(y)
n0+n
the Beta density
p(θ) ∝ θa−1
(1 − θ)b−1
E[p(θ)] = a
a+b
p(y|θ) ∝ θy
θ1−y
p(θ|y) ∝ θa+ y−1
(1 − θ)b+n− y−1
E[p(θ)|y] = a+b
a+b+n
a
a+b
+ n
a+b+n
¯y
Therefore p(y|θ) ∝ exp{ylog θ
1−θ }(1 − θ) where:
φ = log θ
1−θ θ = eφ
1+eφ c(φ) = 1
1+eφ
Finally:
p(θ) = dφ
dθ
1
1+eφ
n0
en0t0log θ
1−θ = θn0t0 (1 − θ)n0(1−t0)−1
where
t0 = a
a+b n0 = a + b
.

10. Conclusions: we are going to another generalization
We note that the conditional expected value of the posterior is
linear with respect to the sample model.
Theorem
Let Y and θ be indipendent d-dimensional random vectors.
Assume E|Yi| < ∞ for i = 1, . . . , d and that the characteristic
function of Y has no zeros. If
E[p(θ|Y)] = aE[p(Y)] + b (9)
Then 0 < a < 1 and the distribution of θ is uniquely determined.

11. Bibliography
• P. Diaconis, D. Ylvisaker, Quantifying Prior Opinion,
Department of Statistics Stanford University, 1983
• P.D. Hoff, A First Course in Bayesian Statistical Methods,
Springer, 2009