This document contains mathematical formulas and derivations related to probabilistic modeling. It begins by defining the Bernoulli distribution and reparameterizing it using a logit transformation. It then generalizes this to the multinomial distribution and derives properties of the Dirichlet distribution, which is the conjugate prior for the multinomial. The final sections discuss maximum likelihood estimation for these models and properties of the normal-inverse-chi-square distribution, which is the conjugate prior for the normal distribution with unknown mean and variance.

1. 2014.07.27
PRML
PRML 2.4

2. (2.194)
(2.195)
2.4
2
1
p(x∣η)=h(x)g(η)exp{η
T
u(x)}
∫ p(x∣η)d x=1
g(η)∫h(x)exp{η
T
u(x)}d x=1

3. (2.196)
(2.197)
(2.194)
(2.198)
2.4
1 :
(2.194)
p(x∣µ)=Bern(x∣µ)=µ
x
(1(µ)
1(x
p(x∣µ)=exp{x ln µ+(1(x)ln(1(µ)}
=exp{ln(1(µ)+x ln
µ
1(µ
}
=(1(µ)exp{ln(
µ
1(µ
)x}
η=ln(
µ
1(µ
)
p(x∣η)=h(x)g(η)exp{η
T
u(x)}

4. (2.198)
(2.199)
2.4
µ
σ(η)
η=ln(
µ
1(µ
)
e
η
=
µ
1(µ
e
η
(µ(1+e
η
)=0
µ=
e
η
1+e
η
=
e
(η
e
η
e
(η
(1+e
η
)
=
1
1+e
(η
=σ (η)
1(µ=1(
1
1+e
(η
=
e
(η
1+e
(η
=
1
1+e
η
=σ ((η)

5. (2.200)
(2.194)
(2.201)
(2.202)
(2.203)
2.4
(2.194)
p(x∣η)=σ ((η)exp(ηx)
p(x∣η)=h(x)g(η)exp{η
T
u(x)}
u(x)=x
h(x)=1
g(η)=σ ((η)

6. (2.204)
(2.205)
(2.194)
(2.206) (2.207) (2.208)
2.4
2 :
(2.194)
p(x∣µ)=
M
∏
k=1
µk
xk
=exp{
M
∑
k=1
xk ln µk }
ηk=ln µk η=(η1,⋯,ηM )
T
p(x∣η)=exp(η
T
x)
p(x∣η)=h(x)g(η)exp{η
T
u(x)}
u(x)=x h(x)=1 g(η)=1

7. (2.209) (2.210)
(2.211)
2.4
∑
k=1
M
µk=1 0≤µk≤1 ∑
k=1
m(1
µk≤1 ∑
k=1
M
xk=1
p(x∣µ)=exp(∑
k=1
M
xk ln µk )=exp{∑
k=1
M (1
xk ln µk+xM ln µM }
=exp{∑
k=1
M (1
xk ln µk+(1( ∑
k=1
M (1
xk)ln(1(∑
k=1
M (1
µk)}
=exp{∑
k=1
M (1
xk ln(
µk
1(∑j=1
M (1
µ j
)+ln(1( ∑
k=1
M (1
µk )}

8. (2.212)
(2.213-1)
(2.213-2)
(2.213)
2.4
µk
t
t µ
(2.213-1) (2.213-2)
ln(
µk
1(∑j=1
M (1
µ j
)=ηk
µk=(1(∑j
µ j)e
ηk
=t e
ηk
t=1(∑j
µ j=1(∑j
t e
ηj
=1(t ∑j
e
ηj
t=1/(1+∑j
e
ηj
)
µk=
e
ηk
1+∑j
e
ηj

9. (2.211)
(2.214)
(2.194)
(2.215) (2.216)
(2.216)
2.4
(2.211)
t
µ
(2.194)
p(x∣µ)=exp{∑
k=1
M (1
xk ln(
µk
1(∑j=1
M (1
µ j
)+ln(1(∑
k=1
M (1
µk)}
=(1(∑k
µk)exp{∑k
xk ln(
µk
1(∑j
µ j
)}
p(x∣η)=1/(1+∑k
e
ηk
)exp(η
T
x)
η=(η1,⋯,ηM (1 ,0)
T
p(x∣η)=h(x)g (η)exp{η
T
u(x)}
u(x)=x h(x)=1 g(η)=1/(1+ ∑
k=1
M (1
e
ηk
)

10. (2.218)
(2.219)
(2.194)
(2.220) (2.221)
(2.222) (2.222)
2.4
3 :
(2.194)
p(x∣µ ,σ
2
)=
1
(2πσ
2
)
1/2
exp{
(1
2σ
2
(x(µ)
2
}
=
1
(2πσ
2
)
1/2
exp{
(1
2σ
2
x
2
+
µ
σ
2
x(
1
2σ
2
µ
2
}
p(x∣η)=h(x)g (η)exp{ηT
u(x)}
η=
( µ/σ
2
(1/2σ
2) u(x)=
(x
x
2)
h(x)=(2π)
(1/2 g (η)=((2η2)
1/2
exp(
η1
2
4η2
)
=
1
(2πσ
2
)
1/2
exp(
(1
2σ
2
µ
2
)exp
{( µ/σ2
(1/2σ
2)
T
(x
x2)}

11. 2.4
2.57
Σ-1
N (x∣µ , Σ)=
1
(2π)
D/2
1
∣Σ∣
1/2
exp{
(1
2
(x(µ)
T
Σ
((((1
(x(µ)}
=
1
(2π)D/2
1
∣Σ∣
1/2
exp{
(1
2
(x
T
Σ
((((1
x((((µ
T
Σ
((((1
x((((x
T
Σ
((((1
µ++++ µΣ
((((1
µ)}
µ
T
Σ
((((1
x====x
T
Σ
((((1
µ
=
1
(2π)
D/2
1
∣Σ∣
1/2
exp{
(1
2
µ
T
Σ
((((1
µ}exp{
(1
2
x
T
Σ
((((1
x++++µ
T
Σ
((((1
x}
=
1
(2π)
D/2
1
∣Σ∣
1/2
exp{
(1
2
µ
T
Σ
((((1
µ}exp
{(µ
T
Σ
((((1
,
(1
2
Σ
((((1
)( x
xx
T)}

12. (2.195)
(2.224)
(2.225)
(2.226)
2.4.1
η
η
1 (2.195) *∇g(η)/g(η)
2
g(η) u(x)
g (η)∫h(x)exp{η
T
u(x)}d x=1
∇ g(η)∫h(x)exp{ηT
u(x)}d x
+g (η)∫h(x)exp{η
T
u(x)}u(x)d x=0
∇ g (η)/g (η)+g(η)∫h(x)exp{η
T
u(x)}u(x)d x=0
(∇ g(η)/ g(η)=g(η)∫h(x)exp{η
T
u(x)}u(x)d x=E[u(x)]
∫ p(x∣η)u((((x))))d x
(ln f (x))'= f ' (x)/ f (x)
(∇ ln g (η)=E[u(x)]

13. (a)
(b)
2.4.1
2.58
(2.226)
(∇ ∇ ln g(η)
=∇(E[u(x)])
T
=∇(g(η)∫h(x)exp{η
T
u(x)}u(x)d x)
T
=∇ g (η)(∫h(x)exp{ηT
u(x)}u(x)d x)
T
+g(η)∫h(x)exp{η
T
u(x)}u(x)u(x)
T
d x
(a)=(∇ g (η)/g (η))E[u(x)
T
]
=∇ ln g (η)E[u(x)
T
]
=(E[u(x)]E[u(x)
T
]
(b)=∫ p(x∣η)u((((x))))u((((x))))
T
d x
=E[u(x)u(x)
T
]
(∇ ∇ ln g(η)=E[u(x)u(x)
T
](E[u(x)]E[u(x)
T
]=cov[u(x)
T
]

14. (2.227)
(2.228)
2.4.1
η 0
N→∞ (2.228)
(2.226)
X={x1,..., xN}
p(X∣η)=
(
N
∏n=1
h(xn)
)g(η)N
exp
{ηT
N
∑n=1
u((((xn))))
}
ln p(X∣η)=∑
n=1
N
ln h(xn)+N ln g(η)+η
T
∑
n=1
N
u((((xn))))
∇ ln p(X∣ηML)=0+N ∇ ln g(ηML)+∑
n=1
N
u((((xn))))=0
(∇ ln g(ηML)=
1
N
∑
n=1
N
u((((xn))))
∑n
u((((xn))))
∑n
u((((xn))))
E[u(x)]
ηML η

15. (2.194)
(2.229)
(2.230)
2.4.2
(2.194)
g (2.194) f
(2.229) (2.227)
(2.229)
ν
p(η∣χ , ν)= f ( χ ,ν)g(η)
ν
exp{ν η
T
χ}
p(x∣η)=h(x)g (η)exp{ηT
u(x)}
p(X∣η) p(η∣χ ,ν)=
(
N
∏n=1
h(xn)
)f ( χ ,ν)g (η)
N +ν
exp
{η
T
(
N
∑n=1
u((((xn))))+ν χ)
}
p(η∣X , χ ,ν)∝ g(η)
N +ν
exp
{η
T
(
N
∑n=1
u((((xn))))+ν χ)
}

16. (2.231)
2.4.3
λ K 1/K
λ
1.λ
2.(1.27)
λ=η^2 λ
η
λ
pη(η)= pλ(λ)∣d λ
d η∣= pλ(η
2
)2η∝ η

17. (2.232)
(2.233)
(2.234)
(2.235)
2.4.3
1
x
µ
A B
p(µ)
p(x∣µ)= f (x(µ)
̂x=x+c
p(̂x∣̂µ)= f ( ̂x( ̂µ) ̂µ=µ+c
A µ B
A(c µ B(c
∫A
B
p(µ)d µ=∫A(c
B(c
p(µ)d µ=∫A
B
p( µ(c)d µ
p(µ(c)= p(µ)

18. (2.141)
(2.142)
2.4.3
µN =
σ2
Nσ0
2
+σ
2
µ0+
Nσ0
2
Nσ0
2
+σ
2
µML
µN =
σ0
2
→0
(
σ2
Nσ0
2
+σ
2
µ0+
Nσ0
2
Nσ0
2
+σ
2
µML)=
σ0
2
→ 0
(
σ
2
/σ0
2
N +σ
2
/σ0
2
µ0+
N
N +σ
2
/σ0
2
µML)=µML
1
σN
2
=
1
σ0
2
+
N
σ
2
1
σN
2
=
σ0
2
→∞ (1
σ0
2
+
N
σ2
)=
N
σ2

19. (2.236)
(2.237)
(2.238)
2.4.3
2
x
σ
A B
∝p(σ) 1/σ σ c p 1/c
p(x∣σ)=
1
σ
f (x
σ ) (σ>0)
̂x=cx
p( ̂x∣̂σ)=
1
̂σ
f (̂x
̂σ ) ̂σ=cσ
A µ B
A/c σ B/c
∫A
B
p(σ)d σ=∫A/c
B/c
p(σ)d σ=∫A
B
p(
1
c
σ)
1
c
d σ
p(σ)= p(1
c
σ)1
c

20. (2.240)
2.4.3
(1.27)
∝p(σ) 1/σ p(ln σ)
λ
p(σ)= p(ln σ)∣d ln σ
d σ ∣= p(ln σ)
1
σ
p(ln σ)= p(σ)σ
N (x∣µ ,σ
2
)∝ σ
(1
exp{(( ̃x/σ)
2
} ( ̃x=x(µ)
λ=1/σ
2
p(λ)=
1
σ
exp{(( ̃x/σ)
2
}∣d a
d x∣=
1
σ
exp{(̃x
2
λ}((σ
3
/2)=
(1
2
λ
(1
exp{(̃x
2
λ}
Gam(λ∣a0,b0)
a0=b0=0

21. (2.236)
2.4.3
2.59
f(x) (2.236)
p(x∣σ)=
1
σ
f (x
σ )
∫(∞
∞
p(x∣σ)d x=∫(∞
∞ 1
σ
f (x
σ )d x=∫(∞
∞ 1
σ
f (y)∣d x
d y∣d y
y=x/σ
=∫(∞
∞ 1
σ
f ( y)σ d y=∫(∞
∞
f ( y)d y=1

22. (2.262)
(2.10)
2.3
(N
m )+( N
m(1 )=(N +1
m )
(N
m )≡
N !
(N (m)! m!
(N
m )+( N
m(1 )=
N !
(N (m)! m!
+
N !
(N+1(m)!(m(1)!
=
(N +1(m)N !
(N +1(m)(N (m)! m!
+
N ! m
(N +1(m)(N (m)! m!
=
N !
(N (m)! m(m(1)!
+
N !
(N +1(m)(N (m)!(m(1)!
=
(N+1) N !
(N +1(m)(N(m)! m!
=
(N +1)!
(N +1(m)! m!
=(N +1
m )

23. (2.263)
(1)N=0
N=0 (2.263)
(2)N=k (2.263)
N=k+1 (2.263)
(1)(2) N(N>0) (2.263)
(1+x)
N
=∑
m=0
N
(N
m)x
m
(1+x)
0
=∑
m=0
0
(0
m)x
m
=1
(1+x)
k+1
=(1+x)
k
+x(1+x)
k
=∑
m=0
k
(k
m)x
m
+∑
m=0
k
(k
m)x
m+1
=∑
m=1
k
(k
m)x
m
+1+∑
m=1
k
( k
m(1)x
m
+x
k+1
=∑
m=1
k
(k+1
m )x
m
+1+x
k+1
=∑
m=0
k+1
(k+1
m )x
m

24. (2.264)
(2.263)
∑
m=0
N
(N
m)µ
m
(1(µ)
N(m
=1
∑
m=0
N
(N
m)µ
m
(1(µ)
N(m
=(1(µ)
N
∑
m=0
N
(N
m)µ
m
(1(µ)
N (m
(1(µ)
(N
=(1(µ)N
∑
m=0
N
(N
m)( µ
1(µ )
m
=(1(µ)N
(1+
µ
1(µ )
N
=
{(1(µ)
(1+
µ
1(µ )}
N
=(1(µ+µ)
N
=1
N
=1

25. (2.285)
2.20
(2.285) Σ
Σ λ u (2.45)
(2.285) a
a u
Σ
(2.285)
a
T
Σa>0
Σui=λi ui
ui
T
Σui=ui
T
λi ui=λi∥ui∥
a= ̂a1 u1+⋯+ ̂aD uD
a
T
Σa=( ̂a1 u1
T
+⋯+ ̂aD uD
T
)Σ( ̂a1 u1+⋯+ ̂aD uD)
=( ̂a1 u1
T
+⋯+ ̂aD uD
T
)( ̂a1 λ1 u1+⋯+ ̂aD λD uD)
= ̂a1
2
λ1∥u1∥+⋯+ ̂aD
2
λD∥uD∥