Theory for Logistic Regression. Cost Function. Feed Forward. Single neuron neural network. Back propagation.

1. Introduction to
Artificial Intelligence & Deep Learning
Lecture 2: Logistic Regression
Dr. Kobkrit Viriyayudhakorn

2. Following slides are based on
• Andrew Ng’s Coursera Deep Learning course.
• Stanford CS231n: Convolutional Neural Networks for Visual
Recognition course.
• Goodwill’s Deep Learning Books
• Prof. Thanaruk’s Introduction to Concepts and Techniques in Data
Mining and Application to Text Mining Book.
• Andreas’s Introduction to Machine Learning with Python Book.
• Giancarlo Zaccone’s Getting Start with Tensorflow Book.
• Justine Johnson’s Python Numpy Tutorial
• ETC…

3. What is Neural Network (Regression)?
Price
Size of House

4. What is Neural Network (Classification)?Red-dish
Round-dish

5. House Price Prediction with 4-3-1 NN
Size (x1)
#Bed Room (x2)
Wealth (x4)
Zip Code (x3)
Price

6. Supervised Learning in Neural Network
Input (X) Output (Y) Application
House Features Price Real Estate Agent
Patient conditions Disease Physician Assistant
Image 10,000 Objects Photo Recognition
CCTV Camera Footage Person Name / Car License
Number
Security / Robot
Audio Text Transcript Speech Recognition, Subtitle
Generation
Text of Thai Language Text of English Language Machine Translation
Radar Signal, Images Position of Obstacle Autonomous Driving

7. Neural Network Example
Feed Forward Neural Network Convolutional Neural Network (CNN)
Recurrent Neural Network (RNN)

8. Supervised Learning
• Structure Data • Unstructured Data
Sepal
Length
Sepal
Width
… Species
5.1 3.5 I. setosa
6.3 3.3 I. virginica
Iris Flower Dataset

9. Why Deep Learning is so popular?
Labelled data (m)
Performance

10. • Data
• Computation
• Algorithm
Why Deep Learning is taking off?

11. Logistic Regression

12. Binary Classification
1 (Flower) or 0 (Non-Flower)

13. 65 10 14 78
43 21 4 25
5 20 130 60
90 120 90 35
78 28 120 30
35 25 1 18
33 12 24 250
65 0 120 45
85 123 4 50
78 88 123 33
Binary Classification
1 (Flower) or 0 (Non-Flower)
255 245 91 128
123 1 244 255
55 80 120 45
85 123 44 39
88 88 123 33
Red
Blue
Green
64
64

14. Notations

15. Logistic Regression

16. Logistic Regression Cost Function
Loss (error) function:
𝑦 = 𝜎 𝑤 𝑇 𝑥 + 𝑏 𝜎 𝑧 =
1
1 + ⅇ−𝑧
𝑥 ⅈ
, 𝑦 ⅈ
… , 𝑥1
𝑚
𝑦 𝑚 𝑦 ⅈ
≈ 𝑦 ⅈ
, where
Given want

18. Gradient Descent
Want to find W, b that minimize 𝐽 W, 𝑏
𝑦 = 𝜎 𝑊 𝑇 𝑥 + 𝑏 𝜎 𝑧 =
1
1 + ⅇ−𝑧, where
𝐽 W, 𝑏 =
1
𝑚
𝑖=1
𝑚
ℒ 𝑦 𝑖
, 𝑦 𝑖 =−
1
𝑚 𝑖=1
𝑚
𝑦 𝑖 log 𝑦+
𝑖
1 − 𝑦 𝑖 log 1 − 𝑦 𝑖
𝐽 W, 𝑏
𝑊
𝑏

19. Gradient Descent (1D)
w

21. Computation Graph

22. Computing derivatives
b = 3
a = 5
c = 2
u = bc
v = a + u J = 3v
11
6
33

23. Computing derivatives
b = 3
a = 5
c = 2
u = bc
v = a + u J = 3v
11
6
33

24. Logistic Regression Gradient Descent
𝑧 = 𝑤 𝑇
𝑥 + 𝑏
𝑦 = 𝑎 = 𝜎 𝑧
ℒ 𝑎, 𝑦 = − 𝑦 log 𝑎 + 1 − 𝑦 log 𝑎

26. Logistic Regression Gradient Descent
𝑧 = 𝑤1 𝑥1 + 𝑤2 𝑥2 + 𝑏
𝑤1
𝑥1
𝑤2
𝑥2
𝑏
𝑎 = 𝜎 𝑧 ℒ 𝑎, 𝑦

27. Gradient Descent in 𝑚 examples

28. What is Vectorization?

29. Algorithm for Logistic Regression