Jim O' Donoghue's slides for a primer tutorial on machine and deep learning with theano

1. primer
Overivew and Tutorial
Jim O’ Donoghue
Deep Learning Meetup @Intercom, Stephens Green
7th April 2016

2. my background
machine learning function elements

3. my background
machine learning function elements
hypothesis functions (NN architectures)
objective functions
optimisation

4. my background
machine learning function elements
hypothesis functions (NN architectures)
objective functions
optimisation
linear regression
multi-layer perceptron

5. my background
machine learning function elements
input types
hypothesis functions (NN architectures)
objective functions + optimisation
output types
regression
multi-layer Perceptron

6. Continuous
Discrete
Categorical
Nominal
Ordinal

7. Supervised
Unsupervised
Semi-Supervised
Feature

8. Supervised
Unsupervised
Semi-Supervised
Feature

12. y𝑥

13. y𝑥

14. y𝑥

15. y𝑥 𝑓 𝑥

16. y𝑦+𝜖𝑥 𝑓 𝑥

17. y𝑦+𝜖𝑥 𝑓 𝑥

18. y𝑦+𝜖𝑥
Optimisation
𝑓 𝑥

19. y𝑦+𝜖𝑥
Optimisation + Hyper-Parameters
𝑓 𝑥

20. y𝑦+𝜖𝑥
hypothesis
ℎ 𝑥

21. y𝑦+𝜖𝑥 ℎ 𝑥
output

22. y𝑦+𝜖𝑥 ℎ 𝑥
calculated by ...

23. y𝑦+𝜖𝑥 ℎ 𝜽 𝑥
calculated by ...

24. y𝑦+𝜖𝑥 ℎ 𝜃 𝑥
objective

25. y𝑦+𝜖𝑥 ℎ 𝜽 𝑥
optimise

27. 27
Hypothesis Functions ℎ 𝑥

28. 28
Hypothesis Functions ℎ 𝑥

29. Hypothesis Functions ℎ 𝑥

30. calculate outputs via𝑦

31. 𝜃 = {Weights, bias}
calculate outputs via

32. 𝜃 = {Weights, bias}
calculate outputs via
n activation functions

33. 𝜃 = {Weights, bias}
calculate outputs via
n activation functions
interim

34. 𝜃 = {Weights, bias}
calculate outputs via
interim functions
Linear

35. 𝜃 = {Weights, bias}
calculate outputs via
interim functions
Linear
Tanh
Cosh
Logistic Sigmoid
Recitified Linear
{non
linear

36. 𝜃 = {Weights, bias}
calculate outputs via
ℎ 𝑥 = 𝑔(𝑓 𝑥 )

37. 𝜃 = {Weights, bias}
calculate outputs via
ℎ 𝑥 = 𝑔(𝑓(𝑔(𝑓 𝑥 ))

38. loss/cost/error
y𝑦+𝜖

39. y − 𝑦𝜖
loss/cost/error

40. y − 𝑦𝜖J(𝜃)
loss/cost/error

41. gradient descent
𝜃
𝐽(𝜃)

42. 𝜃
𝐽(𝜃)
gradient descent

43. 𝜃
𝐽(𝜃)
gradient descent

44. 𝜃
𝐽(𝜃)
gradient descent

45. 𝜕𝐽(𝜃)
𝜕𝜃
A. get partial derivative
gradient descent

46. 𝜃
𝐽(𝜃)
gradient descent

47. 𝜃
𝐽(𝜃)
gradient descent

48. 𝜃 ≔ 𝜃 − 𝛼
𝜕𝐽(𝜃)
𝜕𝜃
𝜕𝐽(𝜃)
𝜕𝜃
A. get partial derivative
B. update the parameters
gradient descent

49. 𝜃
𝐽(𝜃)
gradient descent

50. 𝜃
𝐽(𝜃)
gradient descent

51. 𝜃
𝐽(𝜃)
gradient descent

52. 𝜃
𝐽(𝜃)
gradient descent

53. 𝜃
𝐽(𝜃)
𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝑟𝑎𝑡𝑒
𝛼
gradient descent

54. 54𝜃
𝐽(𝜃)
gradient descent
global optimim

55. 55𝜃
𝐽(𝜃)
gradient descent
local optimum

58. first...

59. the activation
Connection
Weights
Class
Input
Features

60. the activation
Connection
Weights
Class
Input
Features

61. the activation
𝑥
𝑦
𝜃 = {Weights, bias}

62. the activation
𝑥
𝑦
𝜃 = {Weights, bias}
𝑦 = 𝑓 𝑥

63. the activation
𝑥
𝑓(𝑥)
𝜃 = {Weights, bias}
𝑦 = 𝑓 𝑥 = 𝑎𝑥 + 𝑏

64. the activation
𝑥
𝑓 𝜃(𝑥)
𝜃 = {Weights, bias}
𝑦 = 𝑓 𝜃 𝑥 = 𝑤𝑥 + 𝑏

65. the activation
𝑥
𝑓 𝜃(𝑥)
𝜃 = {Weights, bias}
𝑦 = 𝑓 𝜃 𝑥 = 𝑤𝑥 + 𝑏
= 𝜃 𝑇
𝑥

66. the activation
𝑥
𝑓 𝜃(𝑥)
𝜃 = {Weights, bias}
𝑦 = 𝑓 𝜃 𝑥 = 𝑤𝑥 + 𝑏
= 𝜃 𝑇 𝑥
=
𝑖=1
𝑛
𝑤𝑖 𝑥𝑖 + 𝑏

67. the activation for
𝑥
𝑧
𝜃 = {Weights, bias}
𝑧 = 𝑓 𝜃 𝑥
= 𝜃 𝑇 𝑥
= 𝑤 𝑇 𝑥 + 𝑏

68. y − 𝑦𝜖J(𝜃)
one sample

69. 1
𝑚
𝑠=1
𝑚
(𝑦 𝑠 − 𝑦 𝑠)2
all samples
J(𝜃)

70. 1
2𝑚
𝑠=1
𝑚
(𝑦 𝑠 − 𝑦 𝑠)2
all samples
J(𝜃)

71. 1
𝑚
𝑠=1
𝑚
(𝑦 𝑠 − 𝑦 𝑠)𝑥 𝑠
partial derivative
𝜕𝐽(𝜃)
𝜕𝜃

72. δ𝑥 𝑠
partial derivative
𝜕𝐽(𝜃)
𝜕𝜃

73. 𝜃 ≔ 𝜃 − 𝛼
1
𝑚
𝑠=1
𝑚
(𝑦 𝑠 − 𝑦 𝑠)𝑥 𝑠
update

74. the activation for
Connection
Weights
Class
Input
Features

75. Connection
Weights
Class
Input
Features

76. the activation
𝑥
𝑔(𝑧)
𝑧 = 𝑓 𝜃 𝑥
𝑎 = 𝑔 𝑧
=
1
1+𝑒−𝑧

77. the activation

78. the activation
𝑥
𝑎
𝑧 = 𝑓 𝜃 𝑥
𝑎 = 𝑔 𝑧
p(a = 1|𝑥, 𝜃) =
1
1+𝑒−𝑧

79. hypothesis
𝑦 = ℎ 𝑥 = 𝑓 𝜃2 𝑎1
𝑥
𝑎1
𝑦

80. hypothesis
𝑦 = ℎ 𝑥 = 𝑓 𝜃2 𝑎1
𝑥
𝑎1
𝑦
𝑎1 = 𝑔 𝑧 =
1
1 + 𝑒−𝑧
𝑧1 = 𝑓 𝜃1 𝑥 = 𝑤1 𝑇 𝑥 + 𝑏

81. hypothesis
𝑦 = ℎ 𝑥 = 𝑓 𝜃2 𝑎1
𝑥
𝑎1
𝑦
𝑎1 = 𝑔 𝑧 =
1
1 + 𝑒−𝑧
𝑧1 = 𝑓 𝜃1 𝑥 = 𝑤1 𝑇 𝑥 + 𝑏

82. hypothesis
𝑥
𝑎1
𝑦
ℎ 𝑥 = 𝑓(𝑔(𝑓 𝑥 )

83. the error function
𝑥
𝑎1
𝑦
1
2𝑚
𝑠=1
𝑚
(𝑦 𝑠 − 𝑦 𝑠)2
J(𝜃)

84. the partial derivative
𝑥
𝑎1
𝑦
𝜕𝐽(𝜃2)
𝜕𝜃2
1
𝑚
𝑠=1
𝑚
(𝑦 𝑠 − 𝑦 𝑠)𝑎1
𝑠

85. the partial derivative
𝑥
𝑎1
𝑦
𝜕𝐽(𝜃2)
𝜕𝜃2
𝜕𝐽(𝜃2)
𝜕 𝑦
𝜕 𝑦
𝜕𝜃2

86. the partial derivative
𝑥
𝑎1
𝑦
𝜕𝐽(𝜃2)
𝜕𝜃2
𝜕𝐽(𝜃2)
𝜕 𝑦
𝑎1
𝑠

87. the partial derivative
𝑥
𝑎1
𝑦
1
𝑚
𝑠=1
𝑚
(𝑦 𝑠 − 𝑦 𝑠)𝑎1
𝑠
𝜕𝐽(𝜃2)
𝜕𝜃2

88. the partial derivative
𝑥
𝑎1
𝛿2 𝑎1
𝛿2
𝜕𝐽(𝜃2)
𝜕𝜃2

89. the partial derivative
𝑥
𝛿1
𝜕𝐽(𝜃)
𝜕𝜃1
𝛿1 𝑥
𝛿2

90. the partial derivative
𝑥
𝛿1
𝛿2
𝜕𝐽(𝜃1)
𝜕𝑎1
𝜕𝑎1
𝜕𝑧1
𝜕z1
𝜕𝜃1
𝜕𝐽(𝜃1)
𝜕𝜃1

91. the partial derivative
𝑥
𝛿1
𝛿2
𝜃2 𝛿2
𝜕𝑎1
𝜕𝑧1
𝜕z1
𝜕𝜃1
𝜕𝐽(𝜃1)
𝜕𝜃1

92. the partial derivative
𝑥
𝛿1
𝜕𝐽(𝜃1)
𝜕𝜃1
𝛿2
𝜃2 𝛿2 𝑎1(1 − 𝑎1)
𝜕z1
𝜕𝜃1

93. the partial derivative
𝑥
𝛿1
𝜕𝐽(𝜃1)
𝜕𝜃1
𝛿2
𝜃2 𝛿2 𝑎1(1 − 𝑎1) 𝑥

94. the partial derivative
𝑥
𝛿1
𝜕𝐽(𝜃1)
𝜕𝜃1
𝛿2
𝜃2 𝛿2 𝑎1(1 − 𝑎1) 𝑥
𝛿1

95. the partial derivative
𝑥
𝛿1
𝜕𝐽(𝜃1)
𝜕𝜃1
𝛿1 𝑥
𝛿2

96. Connection
Weights
Class
Input
Features

97. Class
Connection
Weights
Class
Input
Features
Learned
Features

99. Learning deep architectures for AI
https://deeplearning.net
https://github.com/jimod/deeplearning-
meetup-dublin
http://colah.github.io/
https://www.coursera.org/learn/machine-
learning
https://www.coursera.org/course/neuralnets