This document provides an overview of deep learning concepts including neural networks, regression and classification, convolutional neural networks, and applications of deep learning such as housing price prediction. It discusses techniques for training neural networks including feature extraction, cost functions, gradient descent, and regularization. The document also reviews deep learning frameworks and notable deep learning models like AlexNet that have achieved success in tasks such as image classification.

1. Deep Learning
Rouyun Pan

2. Outline
• Neural Networks
• Regression and Classification
• Deep Learning
• Convolution neural network
2

3. The concept of learning
in a ML system
• Learning = Improving with experience at some task
• Improve over task T,
• With respect to performance measure, P
• Based on experience, E.
Deep learning
CNN, RNN, LSTM ...
Machine learning
NN, SVM, DT ...
A.I.
3

4. Case:
Housing Price Prediction
4

5. Housing Price Prediction
5

6. Housing Price Prediction
6

7. Housing Price Prediction
Size
#the rooms
Zip code
View
family size
traffic
life quality
Predicted price
7

8. Basic neuron network
x1
Y'
x3
x4
x2
Input layer Hidden layer Output layer
8

9. Basic neuron network
x1
Y'
x3
x4
x2
Input layer Hidden layer Output layer
Many Weighted Sum
9

10. http://www.asimovinstitute.org/neural-network-zoo/
10

11. Learning Strategy
• Supervised learning
• Unsupervised learning
• Reinforcement learning
11

12. Supervised learning
• These're training data set and already know what correct
output.
• The regression problem:
Predicting results within a continuous output
• The classification problem:
Predicting results in a discrete output
12

13. Application
Input (X) Output (X) Application
House size Prices estate
AD types, User info. Click on AD Online Advertising
Image Object (1,…,1000) Photo tagging
Audio Text transcript Speech recognition
Model
standard
NN
CNN
RNN
English Chinese Machine translation
Image, Radar info Position of the cars Autonomous driving
Customized
hybrid
13

14. Unsupervised learning
• The data have no target attribute.
• Analyze data, look for patterns and clustering
14

15. Reinforcement learning
• The agent take actions in an environment
so as to maximize some notion of cumulative reward.
15

16. The workflow
for Supervised learning
Feature
Extraction
Train
the model
Eval
the model
Feature
Extraction Predict
Model
Label
Label
Model
Data
New data
• Training phase
• predicting phase
16

17. How to train a model
• Training data set.
• The layers and neurons
• Hypothesis / Activation function
• Cost / Loss Function
• Optimization algorithm
17

18. Linear regression
18

19. Training dataset
19

20. How to choose parameters
*Choose so that is close to y for our training example (x, y)
20

21. Cost function
It's to quantify the gap between network outputs and actual values
mean squared error method
•
21

22. Cost function (conti.)
22

23. Calculate the cost
23

24. Calculate the cost
24

25. 25
Calculate the cost

26. Cost function (conti.)
26

27. The plot for cost function
27

28. Find the best weights to
minimize the loss
800
- 0.12
28

29. Find the best weights to
minimize the loss
360
29

30. Find the best weights to
minimize the loss
100
0.12
30

31. Optimization algorithm
Gradient Descent:
A iterative optimization algorithm for finding the minimum of a function
•
* one epoch = one pass of all the training examples
31

32. Gradient Descent
>= 0
< 0
32

33. Learning rate
33

34. Learning rate
• ... , 0.001, 0.003, 0.01, 0.03, 0.1, 0.3. 1...
34

35. Local minimum
Local minimum
Global minimum
35

36. Local minimum
is local minimum
= 0
36

37. Momentum
Momentum
Movement
Movement = + Momentum
Negative of
Negative of
37

38. Mini-Batch optimization
• Mini-batch optimization has the following advantages.
• Reduce the memory usage.
• Avoid being trapped in the local minima with the random m
*Batch size = the number of training examples in one pass
Iterations = number of passes, each pass using [batch size] of examples
38

39. Back propagation (BP)
x1
predicted Y
x3
x4
x2
Input layer Hidden layer Output layer
Y ; Label
update ...
39

40. Feature scaling
40

41. Mean Normalization
• Make sure gradient descent is working properly
41

42. Make sure gradient descent
is working properly
•
•
42

43. Under/Overfitting
Overfitting - high varianceUnderfitting - high bias Sweet spot
Train error
Test error
Train error
Test error
43

44. Avoid Overfitting
• Reduce number of features
• Add more training data.
• Regularization
• Dropout
44

45. Regularization
• Keep all the features, but reduce the magnitude of parameters.
45

46. Dropout
• Instead of using all neurons, "dropout" some randomly
(usually 0.5 probability)
46

47. Classification
•
•
•
47

48. Classification
48

49. Classification
49

50. Classification
50

51. Logistic Regression
Want
Sigmoid Function (Logistic Function)
•
51

52. Logistic Regression
Cost function
non- convex
52

53. Logistic Regression
Cost function
53

54. Cost Function & Gradient Descent
• Cost function - Log loss (Cross-entropy) for sigmoid function
• Gradient Descent
54

55. DL Frameworks
https://en.wikipedia.org/wiki/Comparison_of_deep_learning_software
55

56. Deep learning (DL)
56

57. Why is DL Hot Now?
57

58. ImageNet Challenge
58

59. GPU Usage for ImageNet
59

60. Image Classification Task
60

61. Convolutional
Neural Network (CNN)
61

62. CNN
*Fully connected neural network *Locally connected neural network
62

63. CNN
*Share the weight across hidden units
63

64. CNN
64

65. Convolution
65

66. Visualization of Modulation
Ref: Visualizing Higher-Layer Features of a Deep Network
66

67. Alexnet
• A large, deep convolutional neural network (8 layers) to classify in the
training set into the 1000 different classes.
• On the test data, It achieved top-1 and top-5 error rates of 39.7% and
18.9%
Convolutional layers Fully-connected
CONV Layers: 5
Fully Connected Layers: 3
Weights: 61M
MACs: 724M
67

68. Alexnet
• Trained the network with 2 GPUs on ImageNet data, which contained
over 1.2 million annotated images from a total of over 1000 categories.
• Used ReLU for the nonlinearity functions (Found to decrease training
time as ReLUs are several times faster than the conventional tanh
function).
• Used data augmentation techniques that consisted of image
translations, horizontal reflections, and patch extractions.
• Implemented dropout layers in order to combat the problem of
overfitting to the training data.
• Trained the model using batch stochastic gradient descent, with specific
values for momentum and weight decay.
68

69. GPU & Big data
• Trained on two GTX 580 GPUs for five to six days.
69

70. Data augmentation
• It consisted of image translations, horizontal reflections,
and patch extractions.
70

71. Rectified Linear Unit (Relu)
71

72. Relu function
• The nonlinearity functions that be found to decrease
training time as ReLUs are several times faster than the
conventional tanh function
Relu
tanh
72

73. Polling
• Reduce resolution of each channel independently
• Increase translation-invariance and noise-resilience
73

74. Local response
normalization (LRN)
• Tries to mimic the inhibition scheme in the brain
74

75. Dropout
• Avoid overfitting in FC layer.
75

76. Revolution of Depth
http://icml.cc/2016/tutorials/icml2016_tutorial_deep_residual_networks_kaiminghe.pdf
76

77. CNN comparison
77

78. Demo
• Tensorflow playground
http://playground.tensorflow.org/
• ConvNetJS CIFAR-10 demo
http://cs.stanford.edu/people/karpathy/convnetjs/demo/cifar10.html

79. Resource
• Deep learningon on Coursera, Andrew Ng, Stanford University
https://www.coursera.org/specializations/deep-learning
• Deep Learning on MOOC
https://www.udacity.com/course/deep-learning--ud730
• Machine Learning Foundations, HT Lin, National Taiwan University
https://www.coursera.org/learn/ntumlone-mathematicalfoundations/
• TensorFlow
https://www.tensorflow.org/
• cnn-benchmarks
https://github.com/jcjohnson/cnn-benchmarks