Deep Learning

1. Learning
Deep
Learning
M1
Sonse
Shimaoka

2. Neural
Network
(From
the
lecture
slide
of
Nando
de
Freitas
)

3. Machine
Learning

4. Supervised
Learning
Input
output
[0,0,1,0,1,1,0,0,0,1,1]
1
[1,1,1,0,1,1,1,0,0,1,1]
0
[1,1,1,0,1,1,0,0,0,1,1]
0
[0,0,0,0,1,1,1,0,0,0,0]
1
[1,0,1,0,1,1,0,0,0,0,0]
1
[1,0,1,0,0,0,0,0,0,1,1]
0
[0,0,0,0,1,1,0,1,0,1,1]
1
Training
data
Input
output
[1,0,1,0,1,1,0,0,0,1,0]
?
[1,1,1,1,1,1,1,0,0,1,1]
?
[1,0,1,0,1,1,0,1,0,1,1]
?
Test
data
GeneralizaGon

5. Perceptron
∑ sign
x1
x2
x3
w1
w3
w2
b
y
y = sign wj xj
j=1
3
∑ + b
"
#
$$
%
&
''

6. Perceptron
∑ sign
1
3
−2
2
1.5
1
0.5
1
1*2 + 3*1
−2*1.5+ 0.5
= 2.5
y = sign wj xj
j=1
3
∑ + b
"
#
$$
%
&
''

7. Perceptron
(x1, x2, x3 ) = (1,3,−2)
(w1, w2, w3 ) = (2,1,1.5)
b = 0.5
= sign 1*2 + 3*1− 2*1.5+ 0.5( ) = sign(2.5) =1
y = sign wi xi
i=1
3
∑ + b
"
#
$
%
&
' = sign w1x1 + w2 x2 + w3x3 + b( )

8. Perceptron
x1
x2
w1x1 + w2 x2 + b = 0

9. Problem
with
Perceptron
x1
x2
w1x1 + w2 x2 + b = 0
What
is
the
probability
that
this
point
belongs
to
the
posiGve
class?
Perceptron
can’t
answer
this!

10. Problem
with
Perceptron
x1
x2
Impossible
to
separate
linearly
!!

11. LogisGc
Regression
∑ sigmoid
x1
x2
x3
w1
w3
w2
b
y
y = sigmoid wj xj
j=1
3
∑ + b
"
#
$$
%
&
''

12. LogisGc
Regression
sigmoid x( ) =
1
1+ exp(−x)

13. LogisGc
Regression
∑ sigmoid
y = sigmoid wj xj
j=1
3
∑ + b
"
#
$$
%
&
''
1
3
−2
2
1.5
1
0.5
1*2 + 3*1
−2*1.5+ 0.5
= 2.5
0.924
Probability!!

14. Feature
TransformaGon
x1
x2
New
Space
Φ
Non
Linear
TransformaGon
φ1(x1, x2 )
φ2 (x1, x2 )
Original
Space
But,
we
must
sGll
design
the
transformaGon…

15. Feed
Forward
Neural
Network
∑ f
A
neuron
AcGvaGon
funcGon

16. Feed
Forward
Neural
Network
∑ f
∑ f
∑ f
∑ g
x1
x3
x2
h1
h2
h3
y1

17. Feed
Forward
Neural
Network
∑ f
∑ f
∑ f
∑ g
x1
x3
x2
h1
h2
h3
y1
input
layer
hidden
layer
output
layer

18. AbstracGon
by
Layer
Linear
Linear
V
f g
W
x h yWx Vh

19. FFN
can
learn
representaGons!!

20. FFN
can
learn
representaGons!!

21. AcGvaGon
FuncGons
sigmoid x( ) =
1
1+ exp(−x)
=
exp(x)
exp(x)+1

22. AcGvaGon
FuncGons
tanh x( ) =
exp(x)− exp(−x)
exp(x)+ exp(−x)

23. AcGvaGon
FuncGons
rectifier(x) = max(0, x)

24. AcGvaGon
FuncGons
soft max(x1,..., xm )c =
exp(xc )
exp(xk )
k=1
m
∑

25. Loss
FuncGons
• When
you
want
a
model
to
learn
to
do
something,
you
give
it
feedback
on
how
well
it
is
doing.
• This
funcGon
that
computes
an
objecGve
measure
of
the
model's
performance
is
called
a
loss
func1on.
• A
typical
loss
funcGon
takes
in
the
model's
output
and
the
ground
truth
and
computes
a
value
that
quanGfies
the
model's
performance.
• The
model
then
corrects
itself
to
have
a
smaller
loss.

26. L2
norm
(y1,..., yn )
L =
1
n
ti − yi
2
2
i=1
n
∑
(t1,...,tn )
Output:
Target:
Loss:
Task:
Regression

27. Cross
Entropy
(y1,..., yn )
L =
1
n
−ti log yi −(1− ti )log(1− yi )
i=1
n
∑
(t1,...,tn )
Output:
Target:
Loss:
Task:
Binary
ClassificaGon

28. Class
NegaGve
Log
Likelihood
(y1,..., yn )
L = −
1
n
ti,k log yi,k
k
m
∑
i=1
n
∑
(t1,...,tn )
Output:
Target:
Loss:
Task:
MulG
Class
ClassificaGon

29. Output
acGvaGon
funcGons
and
Loss
funcGons
Task
Output
ac1va1on
Loss
func1on
Regression
Linear
L2
norm
Binary
ClassificaGon
Sigmoid
Cross
Entropy
MulG
Class
ClassificaGon
So]max
Class
NLL

30. ProbabilisGc
PerspecGve
• We
can
assume
NNs
are
compuGng
condiGonal
probabiliGes
∑ f
∑ f
∑ f
∑ g
x1
x3
x2
h1
h2
h3
p(t1 | x1, x2, x3 )

31. ProbabilisGc
PerspecGve
• When
NLL = −log p(ti | xi )
i=1
n
∏ = −log
1
2πσ
exp −
ti − yi( )
2
2σ 2
#
$
%
%
&
'
(
(
i=1
n
∏
=
1
2σ 2
ti − yi( )
2
− n 2πσ
i=1
n
∑
p(t | x) =
1
2πσ
exp −
t − y( )
2
2σ 2
"
#
$
$
%
&
'
'
L2
norm

32. ProbabilisGc
PerspecGve
• When
NLL = −log p(ti | xi )
i=1
n
∏ = −log yi
ti
(1−
i=1
n
∏ yi )1−ti
= −ti log yi − (1− ti )log(1− yi )
i=1
n
∑
p(t | x) = yt
(1− y)1−t
Cross
Entropy

33. ProbabilisGc
PerspecGve
• When
NLL = −log p(ti | xi )
i=1
n
∏ = −log y
ti,k
i,k
k=1
m
∏
i=1
n
∏
= − ti,k log yi,k
k=1
m
∑
i=1
n
∑
p(t | x) = yk
tk
k=1
m
∏
Class
NegaGve
Log
Likelihood

34. Gradient
Descent
• Gradient
• Gradient
Descent

35. Gradient
Descent
FuncGon
to
be
minimized
IniGal
point
Learning
rate
Update
rule
L(w)
winit
wnew
← wold
−α
∂L
∂w w=wold
α

36. Gradient
Descent
Big
learning
rate
Small
learning
rate

37. Loss
funcGon
for
LogisGc
regression
L(w,b;D) = log yti
i
i=1
n
∏ (1− yi )1−ti
= ti log yi + (1− ti )log(1− yi )
i=1
n
∑
yi =
1
1+ exp(−wT
xi − b)

38. Gradient
with
respect
to
w
∂L(w,b;D)
∂w
=
∂
∂w
ti log yi + (1− ti )log(1− yi )
i=1
n
∑
=
∂
∂w
ti log yi + (1− ti )log(1− yi )( )
i=1
n
∑
=
∂yi
∂w
∂
∂yi
ti log yi + (1− ti )log(1− yi )( )
i=1
n
∑
=
∂yi
∂w
ti
yi
−
1− ti
1− yi
$
%
&
'
(
)
i=1
n
∑ =
∂yi
∂w
ti − yi
yi (1− yi )
$
%
&
'
(
)
i=1
n
∑
= xi yi (1− yi )
ti − yi
yi (1− yi )
$
%
&
'
(
)
i=1
n
∑
= xi (ti − yi )
i=1
n
∑
∵
∂yi
∂w
=
∂
∂w
1
1+ exp(−wT
xi − b)
#
$
%
&
'
(
=
−
∂
∂w
1+ exp(−wT
xi − b)( )
1+ exp(−wT
xi − b)( )
2
=
xi exp(−wT
xi − b)
1+ exp(−wT
xi − b)( )
2
= xi yi (1− yi )

39. Gradient
with
respect
to
b
∂L(w,b;D)
∂b
=
∂
∂b
ti log yi + (1− ti )log(1− yi )
i=1
n
∑
=
∂
∂b
ti log yi + (1− ti )log(1− yi )( )
i=1
n
∑
=
∂yi
∂b
∂
∂yi
ti log yi + (1− ti )log(1− yi )( )
i=1
n
∑
=
∂yi
∂b
ti
yi
−
1− ti
1− yi
$
%
&
'
(
)
i=1
n
∑ =
∂yi
∂b
ti − yi
yi (1− yi )
$
%
&
'
(
)
i=1
n
∑
= yi (1− yi )
ti − yi
yi (1− yi )
$
%
&
'
(
)
i=1
n
∑
= ti − yi
i=1
n
∑
∵
∂yi
∂b
=
∂
∂b
1
1+ exp(−wT
xi − b)
#
$
%
&
'
(
=
−
∂
∂b
1+ exp(−wT
xi − b)( )
1+ exp(−wT
xi − b)( )
2
=
exp(−wT
xi − b)
1+ exp(−wT
xi − b)( )
2
= yi (1− yi )

40. Gradient
Descent
for
LogisGc
Regression
FuncGon
to
be
minimized
Update
rule
bnew
← bold
−α ti − yi
i=1
n
∑
L(w,b;D)= ti log yi + (1− ti )log(1− yi )
i=1
n
∑
wnew
← wold
−α xi (ti − yi )
i=1
n
∑

41. Exercise:
Gradient
Descent
for
Linear
Regression
L(w,b;D) = ti − yi( )
2
i=1
n
∑
yi = wT
xi + b

42. Answer
FuncGon
to
be
minimized
Update
rule
L(w,b;D) = ti − yi( )
2
i=1
n
∑
bnew
← bold
−α ti − yi
i=1
n
∑
wnew
← wold
−α xi (ti − yi )
i=1
n
∑

43. BackpropagaGon
How
do
we
compute
and
?
∂L
∂W
∂L
∂V
∑ f
∑ f
∑ f
∑ g
x1
x3
x2
h1
h2
h3
y2
L
W2,3
V3,1
∑ g y1
u1
u2
u3
l1
l2

44. BackpropagaGon
Use
the
Chain
Rule!!!
∂
∂x
q s x( )( )=
∂s(x)
∂x
∂q(s(x))
∂s(x)
x
s(x)
qs q(p(x))
∂q(s(x))
∂s(x)
∂s(x)
∂x
∂q(s(x))
∂s(x)

45. BackpropagaGon
∑ f
∑ f
∑ f
∑ g
x1
x3
x2
h1
h2
h3
y2
L
W2,3
V3,1
Start
from
Output
layer:
∂L
∂y1
∑ g y1
u1
u2
u3
l1
l2
∂L
∂y1

46. BackpropagaGon
Apply
Chain
Rule
:
∂L
∂l1
=
∂y1
∂l1
∂L
∂y1
= "g l1( )
∂L
∂y1
∑ f
∑ f
∑ f
∑ g
x1
x3
x2
h1
h2
h3
y2
L
W2,3
V3,1
∑ g y1
u1
u2
u3
l1
l2
∂L
∂y1
∂L
∂l1

47. BackpropagaGon
Apply
Chain
Rule
:
∂L
∂V3,1
=
∂l1
∂V3,1
∂L
∂l1
= h3
∂L
∂l1
∑ f
∑ f
∑ f
∑ g
x1
x3
x2
h1
h2
h3
y2
L
W2,3
V3,1
∑ g y1
u1
u2
u3
l1
l2
∂L
∂y1
∂L
∂l1
∂L
∂V3,1

48. BackpropagaGon
Apply
Chain
Rule
:
∂L
∂h3
=
∂l1
∂h3
∂L
∂l1
+
∂l1
∂h3
∂L
∂l1
= V3,1
∂L
∂l1
+V3,2
∂L
∂l2
∑ f
∑ f
∑ f
∑ g
x1
x3
x2
h1
h2
h3
y2
L
W2,3
V3,1
∑ g y1
u1
u2
u3
l1
l2
∂L
∂y1
∂L
∂l1
∂L
∂V3,1
∂L
∂h3

49. BackpropagaGon
Apply
Chain
Rule
:
∂L
∂u3
=
∂h3
∂u3
∂L
∂h3
= "f u3( )
∂L
∂h3
∑ f
∑ f
∑ f
∑ g
x1
x3
x2
h1
h2
h3
y2
L
W2,3
V3,1
∑ g y1
u1
u2
u3
l1
l2
∂L
∂y1
∂L
∂l1
∂L
∂V3,1
∂L
∂h3
∂L
∂u3

50. BackpropagaGon
Apply
Chain
Rule
:
∂L
∂W2,3
=
∂u3
∂W2,3
∂L
∂u3
= u3
∂L
∂u3
∑ f
∑ f
∑ f
∑ g
x1
x3
x2
h1
h2
h3
y2
L
W2,3
V3,1
∑ g y1
u1
u2
u3
l1
l2
∂L
∂y1
∂L
∂l1
∂L
∂V3,1
∂L
∂h3
∂L
∂u3
∂L
∂W2,3

51. AbstracGon
by
Layer
Linear
Linear
V
f
∂L
∂W
∂L
∂V
g
W
Lx
t
∂L
∂x
h
∂L
∂h
y
∂L
∂y
Wx Vh
∂L
∂ Wx( )
∂L
∂ Vh( )

52. AbstracGon
by
Layer
input output
∂loss
∂input
∂loss
∂outputLayer

53. AbstracGon
by
Layer
input output
Layer
Forward
ComputaGon
output = Layer. forward input( )

54. AbstracGon
by
Layer
∂loss
∂input
∂loss
∂outputLayer
Backward
ComputaGon
∂loss
∂input
= Layer. backward input,
∂loss
∂output
"
#
$
%
&
'
input

55. BackpropagaGon
①
Execute
the
forward
computaGon
Linear
Linear
V
f g
W
Lx
t
h yWx Vh

56. BackpropagaGon
②
Compute
the
derivaGve
of
the
loss
funcGon
with
respect
to
the
output
Linear
Linear
V
f g
W
Lx
t
h y
∂L
∂y
Wx Vh

57. BackpropagaGon
③
StarGng
from
the
final
layer,
backpropagate
derivaGves
through
layers
Linear
Linear
V
f g
W
Lx
t
h y
∂L
∂y
Wx Vh
∂L
∂ Vh( )

58. Classifying
Digits
32×32=1024
pixels
Class:
10
digits
(0~9)
Training:
60000
examples
TesGng:
60000
examples

59. Classifying
Digits
x ∈ R1024
0
0
0
0
1
0
0
0
0
0
!
"
#
#
#
#
#
#
#
#
#
#
#
#
#
#
$
%
&
&
&
&
&
&
&
&
&
&
&
&
&
&
t =

60. Classifying
Digits
Linear
Linear
V,cW,b
x
t
So]max
Tanh
Class
NLL

61. Classifying
Digits
Linear
Linear
V,cW,b
x
t
So]max
Tanh
Class
NLL
u = Wx + b
u

62. Classifying
Digits
Linear
Linear
V,cW,b
x
t
So]max
Tanh
Class
NLL
h = Tanh(u)
hu

63. Classifying
Digits
Linear
Linear
V,cW,b
x
t
So]max
Tanh
Class
NLL
l = Vh + c
h lu

64. Classifying
Digits
Linear
Linear
V,cW,b
x
t
So]max
Tanh
Class
NLL
y = soft max l( )
h ylu

65. Classifying
Digits
Linear
Linear
V,cW,b
x
t
So]max
Tanh
Class
NLL
h y
L = tk log yk
k=1
10
∑ = tT
log y
L
lu

66. Classifying
Digits
Linear
Linear
V,cW,b
x
t
So]max
Tanh
Class
NLL
Wx + b h Vh + c y
∂L
∂y
=
∂
∂y
tT
log y = t1
y1
,...,t10
y10
"
#$
%
&'
T
L
∂L
∂y

67. Classifying
Digits
Linear
Linear
V,cW,b
x
t
So]max
Tanh
Class
NLL
h y
∂L
∂l
= y !(t − y)!
∂L
∂y
= y1 t1 − y1( ),..., y10 t10 − y10( )#$ %&
T
! t1
y1
,...,t10
y10
#
$'
%
&(
T
= t1 t1 − y1( ),...,t10 t10 − y10( )#$ %&
T
L
∂L
∂y
∂L
∂l
lu

68. Classifying
Digits
Linear
Linear
V,cW,b
x
t
So]max
Tanh
Class
NLL
h y
∂L
∂h
= VT ∂L
∂l
L
∂L
∂y
∂L
∂V
=
∂L
∂l
hT ∂L
∂c
=
∂L
∂l
∂L
∂h
∂L
∂V
,
∂L
∂c
∂L
∂l
lu

69. Classifying
Digits
Linear
Linear
V,cW,b
x
t
So]max
Tanh
Class
NLL
h y
∂L
∂u
= 1+ h( )! 1− h( )!
∂L
∂h
L
∂L
∂y
∂L
∂h
∂L
∂V
,
∂L
∂c
∂L
∂u
∂L
∂l
lu

70. Classifying
Digits
Linear
Linear
V,cW,b
x
t
So]max
Tanh
Class
NLL
h y
∂L
∂W
=
∂L
∂u
xT
L
∂L
∂y
∂L
∂h
∂L
∂V
,
∂L
∂c
∂L
∂u
∂L
∂b
=
∂L
∂u
∂L
∂x
= W T ∂L
∂u
∂L
∂W
,
∂L
∂b
∂L
∂x
∂L
∂l
lu

71. Classifying
Digits
bnew
← b −α
∂L
∂b
W new
← W −α
∂L
∂W
V new
← V −α
∂L
∂V
cnew
← c −α
∂L
∂c

72. Torch7

73. Torch7

74. Torch7