The document discusses linear regression and regularization. It introduces linear regression models using basis functions and describes optimization methods like gradient descent. It then covers regularization, discussing how adding constraints like lasso and ridge regression can improve generalization by reducing overfitting. The document presents evidence selection methods like Bayesian model averaging to integrate over models. Finally, it describes applying empirical Bayes methods to estimate hyperparameters like α and β by maximizing the marginal likelihood.

1. PRML
§3
2010 / 06 / 27
id: taki0313
2010 7 9

2. CONTENTS
● §3.1
● §3.2
● §3.3
● §3.4
● §3.5
● §3.6
2010 7 9

3. Introduction
● →
●
{xn} , {tn}
x t
…
t = y(x) p( t | x ) …
Et[t | x] @ §1.5.5
2010 7 9

4. §3.1
●
● Φj
w
Φj(x) = x^j
2010 7 9

5. 2010 7 9

6. §3.1.1
● +
●
● : y(x,w)
2010 7 9

7. N
2010 7 9

8. ED
: =0
2010 7 9

9. …?
2010 7 9

10. M=3, N=2 ///
2010 7 9

11. 2010 7 9

12. 2010 7 9

13. w0
β
2010 7 9

14. §3.1.2
● N
● M=2, N=3
j
N Φ j
Φ(xn) n
2010 7 9

15. ●N>M M S
●t y(xn,w) ∈ S
=
3.2 …
2010 7 9

16. §3.1.3
●
● E
τ η
(3.12)ED …
LMS ( )
2010 7 9

17. §3.1.4
● λ
…
2010 7 9

18. ●
●
● q=1(lasso) λ →
2010 7 9

19. ● …
λ →
M →λ
…?
? §3.4
● 2
2010 7 9

20. §3.1.5
●
y :K
W : (M,K)
2010 7 9

21. ●
tk n tnk,N
2010 7 9

22. §3.2
● §1.5.5
p(t|x) …
…
●
h(x)
y ( !)
2010 7 9

23. ∞ …
h(x)
y(x,w) …?
=
w
D
2010 7 9

24. ●
D
2*(...)
( )^2
Bias:
Variance:
2010 7 9

25. ● …
= Bias^2 + Variance + Noise
Bias Variance
Bias - Variance
h(x) = sin(2πx) , N=25 , M=25
100
λ
2010 7 9

26. ● …
&
!
2010 7 9

27. §3.3
●
●
w
…
2010 7 9

28. w
2
w ∝ ×
?→2
w mN
2010 7 9

29. ∞
mN WML
…
2010 7 9

30. @ (ry
2010 7 9

31. ●1 x, 1 t, y(x,w) = w0 + x*w1
● f(x, a) = a0 + x*a1 (a0=-0.3, a1=0.5)
xn ∈ U(x| -1, 1) f(xn, a)
σ=0.2 tn
● : a0=-0.3 , a1=0.5
● α=2.0 β=(1 / 0.2)^2 = 25 ( )
● ∝ *
●w p(w|α)=N(w|0,1/αE)
2010 7 9

32. y(x,w)
1 ( )
2 ( )
w …
2010 7 9

33. q=2
q=2 …
w =
2010 7 9

34. §3.3.2
● t →
● ?2 k& &
1/β + w
N→∞ →0 (w )
2010 7 9

35. @
2010 7 9

36. ● sin(2πx) , N = 1 , 2 , 4 , 25. ,
● ±
● 9
●w y(x,w)
2010 7 9

37. §3.3.3
●
2010 7 9

38. …
k(x,x’) :
x’ , x
k(x,x’)
x x’
x‘
2010 7 9

39. y(x) y(x’)
:
k(x,x’)
2010 7 9

40. §3.4
●
L {Mi}
D
P(Mi)
D
2010 7 9

41. p(Mi) :
p(D|Mi) :
?
p(D|Mi) / p(D|Mj) :
( )
2010 7 9

42. ● w
w
1
2010 7 9

43. p(w) ~ 1 / Δwprior
: wMAP
Δwposterior
w
( )
2010 7 9

44. M : +
M
…
M1 M2 M3
2010 7 9

45. D
p(w) w
p(D|w)
M1 :
D
M3 :
D
p(D|Mi) D0
M2
M1 D0
M3 D0
2010 7 9

46. M
D
?
2 M1, M2. M1
D
2010 7 9

47. §3.5
●
α,β
…
w
2 …
2010 7 9

48. :w α β
p(t|w,β) : (3.8) , p(w|t,α,β) : (3.49),(3.53),(3.54)
p(α,β|t)
α β
α β w
2010 7 9

49. α,β
p(t|α,β)
α,β
EM
2010 7 9

50. §3.5.1
● p(t|α,β)
(3.11), (3.12), (3.52) …
2010 7 9

51. 2010 7 9

52. (3.27)
w
A ∇∇E(w) :
A
mN
2010 7 9

53. 2010 7 9

54. ● M
● α = 5 * 10^{-3}
● 3~8 3
● M=3
2010 7 9

55. §3.5.2
● p(t|α,β) α
●
→A α + λi
● ln |A| α
2010 7 9

56. α
2α &Σ M
α …
2010 7 9

57. α → β
γ, mN α
α →γ,mN →α →…
× β → λi
β
λi β
2010 7 9

58. §3.5.3
● α
●
→
● λi
2010 7 9

59. λ
γ γ
well-determined
λ1 < λ2
λ2 λ1
γ←
γ →
γ:
γ → → Wml
2010 7 9

60. β
1
μML
γ
γ → 1 / N-γ
2010 7 9

61. ● +9 M=10
●β 11.1 α
● γ 2αEw(mn)
●
● α
●
2010 7 9

62. ● γ
● 0≦α≦∞ → 0≦γ≦M
{wi}
α→γ
γ→
α
● M << N well-determined
●
●α β …
2010 7 9

63. §3.6
● →
●
●
●
1.
2. →M
●
1.
2.
2010 7 9