Learn how to adopt pre-trained model and classify the images

1. 國立臺北護理健康大學 NTUNHS
Transfer Learning
Orozco Hsu
2022-04-11
1

2. About me
• Education
• NCU (MIS)、NCCU (CS)
• Work Experience
• Telecom big data Innovation
• AI projects
• Retail marketing technology
• User Group
• TW Spark User Group
• TW Hadoop User Group
• Taiwan Data Engineer Association Director
• Research
• Big Data/ ML/ AIOT/ AI Columnist
2

3. Tutorial
Content
3
Transfer Learning introduction
Homework
Keras pre-trained model
Transfer Learning with the ResNet50 Model

4. Code
• Download code
• https://github.com/orozcohsu/ntunhs_2022_01.git
• Folder/file
• 20220411_inter_master/run.ipynb
4

5. Code
5
Click button
Open it with Colab
Copy it to your
google drive
Check your google
drive

6. Keras pretrained model
6
Ref: https://www.mathworks.com/help/deeplearning/ug/pretrained-convolutional-neural-networks.html

7. Keras pre-trained model
• Keras Applications are deep learning models that are made available
alongside pre-trained weights
• These models can be used for prediction, feature extraction, and fine-
tuning
• Weights are downloaded automatically when instantiating a model.
They are stored at ~/.keras/models/
7

8. Keras pre-trained model
8
Ref: https://keras.io/api/applications/

9. More
• Keras offers two ways to build neural networks
• using Functional API
• It is more powerful than the sequential API in the sense branching or
sharing of layers is allowed here.
• It can have multiple inputs and outputs.
• using Sequential API
• Create the model layer-by-layer
• Sharing of layers or branching of layers is not allowed
• Can’t have multiple inputs or outputs
9

10. More
Using sequential API
10
Ref: https://www.analyticsvidhya.com/blog/2021/07/understanding-sequential-vs-functional-api-in-keras

11. More
Using functional API
11

12. More
12
tensorflow included keras library as a internal deep learning package
always check your tf version, especially in the CoLab

13. More
13

14. Exercise
• Try to run this code
14
pre-trained

15. Transfer Learning introduction
• Transfer Learning is a process where a model built for a problem is
again reused for another problem based on some factors
• Using Transfer Learning you can achieve good results if the data is
kind of similar to the data on which model is already trained
15

16. Transfer Learning introduction
• A CNN can be divided into two main parts
• Feature learning and classification
16
Ref: https://towardsdatascience.com/transfer-learning-with-vgg16-and-keras-50ea161580b4

17. 17
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Transfer Learning introduction
The VGG16 Model has 16 Convolutional and Max Pooling layers, 3 Dense layers
for the Fully-Connected layer, and an output layer of 1,000 classes
Ref: https://www.learndatasci.com/tutorials/hands-on-transfer-learning-keras/

18. Transfer Learning introduction
• Feature Extraction Approach (exclude top)
18
We will bootstrap a Fully-Connected layer to
generate predictions
top

19. Exercise
• Try to run this code
19
transfer_learning_mnist
training data
training data
Model Output
Output
fit
fit

20. Transfer Learning introduction
20
• Fine-Tuning Approach
the final convolutional and pooling layers are
unfrozen to allow training. A Fully-Connected
layer is defined for training and prediction.

21. Different Transfer Learning Strategies
• The training dataset is similar to the base training dataset and is small
in size
• Using a deep net-based ConvNet on such a dataset can result in overfitting of
the model
• The macro features of the pre-trained are likely to be important for this data
also as the data is alike to the base model data
• Freeze the weights of ConvNet and make them in use for extracting the
features
• Detach the fully connected layers from the pre-trained base ConvNet and add
new fully connected layers according to the number of classes in the target
dataset
21
ConvNet: Convolutional Neural Network

22. Different Transfer Learning Strategies
• The training dataset is similar to the base training dataset and is large
in size
• The data we have is large in nature so it is assumed that there won't be any
overfitting of the model here if the fine-tuning is performed in the whole full
network
• Detach the fully connected layers from the pre-trained base ConvNet and add
new fully connected layers according to the number of classes in the target
dataset
• Train the entire network
22

23. Exercise
• Try to run this code
23
fine_tuning
Layer1
Layer2
Layer3
…
Layer148
Layer149
Layer150
Layer151
Layer152
Layer153
Layer154
Set output as an input for next layer
New layer

24. Different Transfer Learning Strategies
• The target dataset is different from the base training dataset and is
small in nature
• Overfitting is again a concern in this type of situation where the data is small
• The higher-level features in ConvNet would not give any appropriate results
on the target dataset as the target dataset is not similar to the base dataset
24

25. Different Transfer Learning Strategies
• The target dataset is different from the base training dataset and is
large in nature
• We can train the ConvNet from scratch, due to the large size of the target
dataset and dissimilarity from the base dataset.
• It is good practice to initialize the weights from the pre-trained model and
fine-tune them that can result in the training faster
25

26. More
• Fast Style Transfer for Arbitrary Styles
• https://www.tensorflow.org/hub/tutorials/tf2_arbitrary_image_stylization
• https://deepai.org/machine-learning-model/fast-style-transfer
• https://www.youtube.com/watch?v=pLWIaQwkJwU
26

27. Transfer Learning with the ResNet50 Model
• It consists of 60,000 (size: 32x32) RGB color images in 10 classes
27
Ref: https://www.cs.toronto.edu/~kriz/cifar.html

28. Transfer Learning with the ResNet50 Model
• ResNet model of Computer Vision that is trained on the ImageNet
dataset having 14 million images that can be used for Image
classification tasks
• imagenet 1,000 class
• https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a
28

29. STAGE 1
STAGE 2
STAGE 3
STAGE 4
STAGE 0
INPUT
OUTPUT
ResNet50
CONV: 7× 7, 64, /2
BN, RELU
MAXPOOL: 3× 3, /2
STAGE 0
(3, 224, 224)
(64, 56, 56)
STAGE 1
BTNK1: 64, 56, 64, 1
(256, 56, 56)
BTNK2: 256, 56
(256, 56, 56)
BTNK2: 256, 56
(256, 56, 56)
STAGE 2
BTNK1: 256, 56, 128, 2
(512, 28, 28)
BTNK2: 512, 28
(512, 28, 28)
BTNK2: 512, 28
(512, 28, 28)
BTNK2: 512, 28
(512, 28, 28)
STAGE 3
BTNK1: 512, 28, 256, 2
(1024, 14, 14)
BTNK2: 1024, 14
(1024, 14, 14)
BTNK2: 1024, 14
(1024, 14, 14)
BTNK2: 1024, 14
(1024, 14, 14)
BTNK2: 1024, 14
(1024, 14, 14)
BTNK2: 1024, 14
(1024, 14, 14)
STAGE 4
BTNK1: 1024, 14, 512, 2
(2048, 7, 7)
BTNK2: 2048, 7
(2048, 7, 7)
BTNK2: 2048, 7
(2048, 7, 7)
BTNK1: C, W, C1, S
CONV: 1× 1, C1, /S
BN, RELU
CONV: 3× 3, C1, /1
BN, RELU
CONV: 1× 1, C1*4, /1
BN
CONV: 1× 1, C1*4, /S
BN
+, RELU
(C, W, W)
(C1*4, W/S, W/S)
BTNK2: C, W
CONV: 1× 1, C/4, /1
BN, RELU
CONV: 3× 3, C/4, /1
BN, RELU
CONV: 1× 1, C, /1
BN
+, RELU
(C, W, W)
(C, W, W)

30. More
30
Make sure GPU runtime
Change to High-RAM if available

31. More
31
Always check your resource if it hits the
maximum usage, it will reset runtime

32. Exercise
• Try to run this code
32
transfer_learning_resnet50

33. Homework
• Try to change other pre-trained models such like VGG16 or
InceptionV3 to rebuild the model and see how its
performance (reference page 14)
• https://www.tensorflow.org/api_docs/python/tf/keras/applications/
vgg16/VGG16
• Try to change other pre-trained models to rebuild a new
transfer learning model and see how its performance
• Or
• Change the trainable layers to compare its performance
(model fine tuning)
33