Plotting the training process Regularization Batch normalization Saving and loading the weights and the architecture of a model Visualize a Deep Learning Neural Network Model in Keras

1. Introduction for Deep Neural
Network DNN with Python
Asst. Prof. Dr.
Noor Dhia Al-Shakarchy
May 2021
Lecture 3

2. Outlines
 Plotting the training process
 Regularization
 Batch normalization
 Saving and loading the weights and the
architecture of a model
 Visualize a Deep Learning Neural Network
Model in Keras
2

3. Plotting the training process
 Matplotlib is a cross-platform, data visualization and
graphical plotting library for Python and its numerical
extension NumPy.
 # Code:
history = model.fit(X, y, epochs=10, batch_size=10,
verbose=2)
print(history.history.keys())
print(history.history['acc'])
3

4. Plotting the training process
# Code:
# summarize history for accuracy
import matplotlib.pyplot
matplotlib.pyplot.plot(history.history['acc'])
matplotlib.pyplot.title('model accuracy')
matplotlib.pyplot.ylabel('accuracy')
matplotlib.pyplot.xlabel('epoch')
matplotlib.pyplot.legend(['train'], loc='upper left')
matplotlib.pyplot.show()
4

5. Plotting the training process
# Code:
import matplotlib.pyplot
# summarize history for loss
matplotlib.pyplot.plot(history.history['loss'])
matplotlib.pyplot.title('model loss')
matplotlib.pyplot.ylabel('loss')
matplotlib.pyplot.xlabel('epoch')
matplotlib.pyplot.legend(['train'], loc='upper left')
matplotlib.pyplot.show()
5

6. normalization layers
Batch normalization layers are added to accelerate
the training process and coordinate the update of
multiple layers in the model.
the general process is sketched in figure below:
6
Batch Normalization sketch for simple network

7. normalization layer Code
7
from keras.layers import BatchNormalization
….
model.add(BatchNormalization())

8. Regularization
 One of the most important problems accrues during
training the model is overfitting, this issue occurs if
the model fits into the training set too well. This
caused the model becomes difficult to generalize to
unseen examples. That means the model accuracy
will be higher in the training set than the
validation/test set. The model can deal with this
problem by adding regularization layers
8

9. Regularization
The list of regularization parameters commonly used
for dense, and convolutional modules:
 kernel_regularizer: Regularizer function applied
to the weight matrix
 bias_regularizer: Regularizer function applied to
the bias vector
 activity_regularizer: Regularizer function applied
to the output of the layer (its activation)
The regularization layers mostly used are:
 Dropout,
 L1/L2 regularization
9

10. Regularization
 dropout layer
The dropout layer is reducing correlation between
neurons
The model present dropout layer after some layers to
avoid the overfitting and to effectively control
noise during the training process.
10
The dropout process

11. dropout layer Code
11
from keras.layers import Dropout
….
model.add(Dropout(.25))

12. Regularization
 L1/L2 regularization
 This layer which also called “Elastic Net
Regularization” tend to decrease overfitting of
deep learning neural network model by
regularization the weight.
12

13. dropout layer Code
13
# in the Densew layer:
from keras.layers import regularizers
from keras.regularizers import l2
from keras.constraints import unit_norm
keras.regularizers.l1(0.01)
keras.regularizers.l2(0.01)
keras.regularizers.l1_l2(l1=0.01, l2=0.01)
model.add(Dense(15, activation='relu', name='fc1',
kernel_constraint=unit_norm(),
kernel_regularizer=l2(0.01),
bias_regularizer=l2(0.01)))

14. Saving and loading the weights
and the architecture of a model
 Model architectures can be easily saved and loaded
as follows:
 # save as JSON json_string = model.to_json()
 # save as YAML yaml_string = model.to_yaml()
14

15. Saving and loading the weights and the
architecture of a model
# save model
model_json = model.to_json()
open('proposed_architecture.json', 'w').write(model_json)
# And the weights learned by our DNN on the training set
model.save_weights('proposed_weights.h5', overwrite=True)
15

16. Saving and loading the weights
and the architecture of a model
# load model
model_architecture = ‘proposed_architecture.json'
model_weights = ‘proposed_weights.h5'
model = model_from_json(open(model_architecture).read())
model.load_weights(model_weights)
print("load model done")
16

17. Visualize a Deep Learning
Neural Network Model in Keras
 They are:
 Summarize Model
 Visualize Model
17

18. Summarize Model
Keras provides a way to summarize a model.
The summary is textual and includes information
about:
 The layers and their order in the model.
 The output shape of each layer.
 The number of parameters (weights) in each
layer.
 The total number of parameters (weights) in the
model.
The summary can be created by calling
the summary() function on the model
Model. summary()
18

19. THANK YOU