PRML（パターン認識と機械学習 by Christopher M. Bishop）の2.3節の解説スライド

1. PRML 2.3
Yuki Soma
4/24

2. • 2.3 The Gaussian Distribution . . . . . . . . . . . . . . . . . . . 78
– 2.3.1 Conditional Gaussian distributions . . . . . . . . . . . . . . 85
– 2.3.2 Marginal Gaussian distributions . . . . . . . . . . . . . . . 88
– 2.3.3 Bayes’ theorem for Gaussian variables . . . . . . . . . . . 90
– 2.3.4 Maximum likelihood for the Gaussian . . . . . . . . . . . . 93
– 2.3.5 Sequential estimation . . . . . . . . . . . . . . . . . . . . . 94
– 2.3.6 Bayesian inference for the Gaussian . . . . . . . . . . . . . 97
– 2.3.7 Student’s t-distribution . . . . . . . . . . . . . . . . . . . . 102
– 2.3.8 Periodic variables . . . . . . . . . . . . . . . . . . . . . . . 105
– 2.3.9 Mixtures of Gaussians . . . . . . . . . . . . . . . . . . . . 110

3. 1.
2.
–
–
3.
4.
– This is “ ” MAP !

4. • 1
• D

5. • [0,1] N N→∞

6. PRML
• PRML
1. Σ
2. 1
3. 𝜇, Σ
•
– 4/14 http://research.nii.ac.jp/~satoh/utpr/

7. Mahalanobis
• Δ 𝐱, 𝝁 Mahalanobis
• Σ

8. 1. Σ
•
– Exercise 2.17

9. • (a)→(b)
–

10. (a)→(b)
• Σ
• D …
•
– Exercise 2.18

11. (a)→(b)
• Σ Exercise 2.19
• Σ−1

12. (a)→(b)
•
–
–

13. 2. 1
• 1 = 1
• 𝑝 𝐱 d𝐱 = 1

14. 3. 𝜇, Σ
01

15. 3. 𝜇, Σ

16. 1
• Free parameter
–
•
(
𝐷(𝐷 + 1)
2
+ 𝐷) 2𝐷 (𝐷 + 1)
Σ Σ Σ

17. 2
• Unimodal i.e.
• Multimodal

18. • latent variables
1. Section 2.3.9
2. Chapter 12
• Free parameter
• 2 …
– Markov random field Section 8.3
•
– Linear dynamical system Section 13.3
•

19. 2.3.1 Conditional
•
• 𝐱 𝑎, 𝜇 𝑎 𝑀 Σ 𝑎𝑎 𝑀 × 𝑀
• 𝑝(𝐱 𝑎|𝐱 𝑏)
•

20. Precision matrix
• precision matrix
• Σ Λ
•
– Λ 𝑎𝑎, Λ 𝑏𝑏
– Σ

21. • 𝑝 𝐱 = 𝑝(𝐱 𝑎, 𝐱 𝑏)
• 𝑝 𝐱 𝑎 𝐱 𝑏 =
𝑝(𝐱 𝑎,𝐱 𝑏)
𝑝(𝐱 𝑏)
~ Gaussian
• …

22. • 𝐱 𝑎 2
– …
• Δ2
𝐱 𝑎, 𝝁 𝑎|𝑏 + 𝑐𝑜𝑛𝑠𝑡
𝑒 𝑥+𝑦 = 𝑒 𝑥 𝑒 𝑦

23. • Mahalanobis
•
• …

24. • 2 …
•

25. • 1 …
• …

27. Precision matrix is good
• 𝐱 𝜇 𝑎|𝑏, Σ 𝑎|𝑏 …
• Precision matrix
• Note!: 𝐱 𝑏
– Linear gaussian model
• Section 8.1.4

28. 2.3.2 Marginal
• Section
–
•
– exp 𝑓 + 𝑔 = exp 𝑓 exp{𝑔}
– 𝑝(𝐱) 𝐱 𝑏
ℎ(𝐱 𝑎) exp{ℎ 𝐱 𝑎 }
– 𝐱 𝑏 𝐱 𝑎
ℎ(𝐱 𝑎)

29. 𝐱 𝑏
• 𝐱 𝑏
𝐱 𝑏
Mahalanobis
1
Λ 𝑏𝑏
𝐦
𝐱 𝑎

30. 𝐱 𝑏
• 𝐱 𝑏
– +

31. • Conditional

32. •
•

33. • Conditional

34. •
• precision matrix

35. 2

36. 2.3.3 for
• 𝐱 M 𝐲 D
• 𝑝(𝐲|𝐱) Linear Gaussian model Sec 8.1.4
– 𝐱 𝐱
• 𝑝 𝐲 , 𝑝(𝐱|𝐲)

37. •
• 𝐳
• 𝑝(𝐳) p 𝐲 , 𝑝(𝐱|𝐲)
–

38. 𝐳
•
• 𝐳 2

39. 𝐳
• 2 …

40. 𝐳
• 1 …

41. 𝑝(𝐳) p 𝐲 , 𝑝(𝐱|𝐲)
•

42. 2.3.4 for
• D N ~𝑁(𝝁, Σ)
• log
• 𝐗 2
sufficient statistics

43. •
• Exercise 2.34

44. biased? unbiased?
• biased Exercise 2.35
•

45. 2.3.5 Sequential Estimation
• N-1
•
• N
• …

46. • N 𝝁ML
(𝑁)
𝐱 𝑁
• seqential estimation

47. • 𝜃, 𝑧 𝑝(𝑧, 𝜃)

48. Robbins-Monro
• 1951
•
– z ∞
• 𝑓 𝜃 > 0 ⇒ 𝜃∗
< 𝜃, 𝑓 𝜃 < 0 ⇒ 𝜃 < 𝜃∗
– without loss of generality

49. Robbins-Monro
• 3
{𝑎 𝑁} 2.129 0

52. •
[Blum, 1965]

53. 2.3.6 for
•
•
– exp{𝜇 2 }
⇒
⇒

54. •

55. • prior
• Precision 1
1
𝜎2

56. •

57. • D
– Exercise 2.40

58. Sequential Estimation is easy
•

59. •
– precision 𝜆 =
1
𝜎2
•
– 𝜆 𝑛
𝑒 𝑐𝜆
⇒
⇒

60. •

61. 𝑎 > 0 𝑎 ≥ 1

62. precision
• …
• …

63. • 𝑎 𝑁 N
𝑁
2
– 2𝑎0
• 𝑏 𝑁 N
𝑁
2
𝜎ML
2
– =
𝑏0
𝑎0
2𝑎0

64. •

65. • 𝑝 𝜇, 𝜆 = 𝑝 𝜇 𝜆 𝑝(𝜆)
• 𝜇0 =
𝑐
𝛽
, 𝑎 = 1 +
𝛽
2
, 𝑏 = 𝑑 −
𝑐2
2𝛽
• normal-gamma/Gaussian-gamma

66. Gaussian-gamma

67. D
•
–
•
– Wishart
• Exercise 2.45
• 𝜈

68. D
•
– Gaussian-Wishart

69. Student t
• Gaussian-gamma precision
–
–

70. t
• 𝜈 t
• 𝜆 precision
–

71. t
• 1
• ∞
– Exercise: 2.47:

72. t
•
• Infinite mixtures of Gaussian
• tail
⇒t robustness

73. t robustness

74. • 𝜈 = 2𝑎, 𝜆 =
𝑎
𝑏
, 𝜂 =
𝑏
𝑎
𝜏
•
•

75. t
•
– Exercise 2.49:

76. 2.3.8
•
– 𝑝 𝑥 = 𝑝 𝑥 + 2𝜋
•
–
•

77. • PRML
•
Mardia and Jupp (2000)

78. •
•
– 𝜃 𝑀𝐿 =
𝜃 𝑖
𝑁
•
– 𝐱 𝑖 = (cos 𝜃𝑖 , sin 𝜃𝑖)
– 𝐱 𝑀𝐿 = (
1
𝑁
cos 𝜃𝑖 ,
1
𝑁
sin 𝜃𝑖)

79. von Mises
•
•

80. von Mises
•
Conditional

81. von Mises
•
• 𝑚 =
𝑟0
𝜎2 𝑚 concentration parameter

82. von Mises
• 𝑚
– Exercise 2.52

83. von Mises
•
• 𝜃0, 𝑚
• 𝜃0

84. von Mises
• 𝑚 …
–
𝑑
𝑑𝑚
𝐼0 𝑚 = 𝐼1(𝑚) [Abramowitz and Stegun, 1965]
• 𝑚 𝑀𝐿
•

85. 𝐼0 𝑚 , 𝐴 𝑚

86. •
• Conditional marginal
–
•
• von Mises unimodal

87. 2.3.9 Mixtures of Gaussians
• unimodal
⇒mixture distributions

88. 2.3.9 Mixtures of Gaussians
•
• 1
•

89. • ⇒Chapter 9
•
– ⇒EM