Theory for Neural Network.

1. Introduction to
Artificial Intelligence & Deep Learning
Lecture 4: Shallow Neural Network
Dr. Kobkrit Viriyayudhakorn

2. What is a Neural Network?
𝑧 = 𝑤 𝑇
𝑥 + 𝑏𝑤
𝑏
𝑎 = 𝜎 𝑧 ℒ 𝑎, 𝑦
𝑥
Logistic Regression
Neural Network
𝑥
𝑧 1
= 𝑤 1
𝑥 + 𝑏 1𝑤 1
𝑏 1
𝑎 1 = 𝜎 𝑧 1
ℒ 𝑎 2 , 𝑦𝑧 2
= 𝑤 2
𝑎 1
+ 𝑏 2
𝑤 2
𝑏 2
𝑎 2 = 𝜎 𝑧 2

3. Neural Network Representation

4. Neural Network Representation
𝑤 𝑇 𝑥 + 𝑏 𝜎 𝑧
𝑥1
𝑥2
𝑥3
𝑎 = 𝑦

5. Neural Network Representation

6. Neural Network Representation Learning

7. Vectorizing Across Multiple Training Examples

8. Vectorizing Across Multiple Training Examples

9. Explanation for Vectorized Implementation

10. Explanation for Vectorized Implementation

11. Activation Function: Sigmoid

12. Activation Function : Tanh

13. Activation Function : ReLu and Leaky ReLu

14. Pro and cons for Activation functions
• Sigmoid

15. Pro and cons for Activation functions
• Tanh

16. Pro and cons for Activation functions
• ReLu and Leaky ReLu

17. Why Neuron Network need the activation
function?

18. Why Neuron Network need the activation
function?

19. Derivative of Activation function
• Sigmoid

20. Derivative of Activation function
• Tanh

21. Derivative of Activation function
• ReLu and Leaky ReLu

22. Gradient Descent for Neural Network

23. Formulas For Computing Derivatives

24. Backpropagation Intuition
• Logistic Regression
𝑧 = 𝑤 𝑇 𝑥 + 𝑏𝑤
𝑏
𝑎 = 𝜎 𝑧 ℒ 𝑎, 𝑦
𝑥

25. Backpropagation Intuition
• Neural Network
𝑥
𝑧 1 = 𝑤 1 𝑥 + 𝑏 1𝑤 1
𝑏 1
𝑎 1 = 𝜎 𝑧 1
ℒ 𝑎 2 , 𝑦𝑧 2 = 𝑤 2 𝑎 1 + 𝑏 2
𝑤 2
𝑏 2
𝑎 2 = 𝜎 𝑧 2

26. Summary of Gradient Descent
ⅆ𝑧 2
= 𝑎 2
− 𝑦
ⅆ𝑤 2 = 𝑑𝑧 2 𝑎 1 𝑇
ⅆ𝑏 2 = ⅆ𝑧 2
ⅆ𝑧 1 = 𝑤 2 𝑇 ⅆ𝑧 2 ∗ g 1 ′
𝑧 1
ⅆ𝑤 1
= 𝑑𝑧 1
𝑥 𝑇
ⅆ𝑏 1
= ⅆ𝑧 1

27. Random Initialization
• Why?

28. Random Initialization
• Code