The document provides a comprehensive overview of TensorFlow and Keras callbacks, which are utilities called during model training for monitoring and controlling the training process. It details various callback classes such as EarlyStopping, ModelCheckpoint, and ReduceLROnPlateau, along with their functionalities and usage within training methods. Additionally, the document includes practical examples and links to further resources for implementing these callbacks effectively.

1. Alireza Akhavanpour
Tensorflow & Keras callbacks
TensorFlow Callbacks (Part I)
Alireza Akhavanpour
Akhavanpour.ir
CLASS.VISION
https://github.com/Alireza-Akhavan/tf2-tutorial

2. Alireza Akhavanpour
Tensorflow & Keras callbacks
What is Callback?
 Callbacks: utilities called at certain points during model training.
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks

3. Alireza Akhavanpour
Tensorflow & Keras callbacks
What is Callback?
 Callbacks: utilities called at certain points during model training.
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks
 set of methods called at various stages of training, testing, and
predicting
o batch/epoch
o start and ends
 useful to get a view on internal states and statistics of the model
during training.
 You can pass a list of callbacks to any of these methods:
o tf.keras.Model.fit()
o tf.keras.Model.evaluate()
o tf.keras.Model.predict().

4. Alireza Akhavanpour
Tensorflow & Keras callbacks
Model methods that take callbacks

5. Alireza Akhavanpour
Tensorflow & Keras callbacks
Classes
 class BaseLogger: Callback that accumulates epoch averages of metrics.
 class CSVLogger: Callback that streams epoch results to a csv file.
 class Callback: Abstract base class used to build new callbacks.
 class EarlyStopping: Stop training when a monitored quantity has stopped improving.
 class History: Callback that records events into a History object.
 class LambdaCallback: Callback for creating simple, custom callbacks on-the-fly.
 class LearningRateScheduler: Learning rate scheduler.
 class ModelCheckpoint: Save the model after every epoch.
 class ProgbarLogger: Callback that prints metrics to stdout.
 class ReduceLROnPlateau: Reduce learning rate when a metric has stopped improving.
 class RemoteMonitor: Callback used to stream events to a server.
 class TensorBoard: Enable visualizations for TensorBoard.
 class TerminateOnNaN: Callback that terminates training when a NaN loss is encountered.
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks

6. Alireza Akhavanpour
Tensorflow & Keras callbacks
Classes
 class BaseLogger: Callback that accumulates epoch averages of metrics.
 class CSVLogger: Callback that streams epoch results to a csv file.
 class Callback: Abstract base class used to build new callbacks.
 class EarlyStopping: Stop training when a monitored quantity has stopped improving.
 class History: Callback that records events into a History object.
 class LambdaCallback: Callback for creating simple, custom callbacks on-the-fly.
 class LearningRateScheduler: Learning rate scheduler.
 class ModelCheckpoint: Save the model after every epoch.
 class ProgbarLogger: Callback that prints metrics to stdout.
 class ReduceLROnPlateau: Reduce learning rate when a metric has stopped improving.
 class RemoteMonitor: Callback used to stream events to a server.
 class TensorBoard: Enable visualizations for TensorBoard.
 class TerminateOnNaN: Callback that terminates training when a NaN loss is encountered.
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks

7. Alireza Akhavanpour
Tensorflow & Keras callbacks
Methods
 on_batch_begin
 on_batch_end
 on_epoch_begin
 on_epoch_end
 on_predict_batch_begin
 on_predict_batch_end
 on_predict_begin
 on_predict_end
 on_test_batch_begin
 on_test_batch_end
 on_test_begin
 on_train_batch_begin
 on_train_batch_end
 on_train_begin
 on_train_end

8. Alireza Akhavanpour
Tensorflow & Keras callbacks
Usage
One parameter
two parameters
two parameters

9. Alireza Akhavanpour
Tensorflow & Keras callbacks
CSVLogger
from tensorflow.keras.callbacks import CSVLogger
csv_logger = CSVLogger('training.log')
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/CSVLogger

10. Alireza Akhavanpour
Tensorflow & Keras callbacks
BaseLogger
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/BaseLogger

11. Alireza Akhavanpour
Tensorflow & Keras callbacks
What was it?!

12. Alireza Akhavanpour
Tensorflow & Keras callbacks
What was it?!

13. Alireza Akhavanpour
Tensorflow & Keras callbacks
EarlyStopping
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping
val_loss: 0.4012
0.4012

14. Alireza Akhavanpour
Tensorflow & Keras callbacks
EarlyStopping
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping
val_loss: 0.4012
0.4012
val_loss: 0.3580

15. Alireza Akhavanpour
Tensorflow & Keras callbacks
EarlyStopping
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping
val_loss: 0.4012
val_loss: 0.3580
0.3580

16. Alireza Akhavanpour
Tensorflow & Keras callbacks
EarlyStopping
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping
val_loss: 0.4012
val_loss: 0.3580
0.3580
val_loss: 0.3371

17. Alireza Akhavanpour
Tensorflow & Keras callbacks
EarlyStopping
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping
val_loss: 0.4012
val_loss: 0.3580
0.3580
val_loss: 0.3371

18. Alireza Akhavanpour
Tensorflow & Keras callbacks
EarlyStopping
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping
val_loss: 0.4012
val_loss: 0.3580
0.3371
val_loss: 0.3371

19. Alireza Akhavanpour
Tensorflow & Keras callbacks
EarlyStopping
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping
val_loss: 0.4012
val_loss: 0.3580
0.3371
val_loss: 0.3371
val_loss: 0.3118

20. Alireza Akhavanpour
Tensorflow & Keras callbacks
EarlyStopping
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping
val_loss: 0.4012
val_loss: 0.3580
0.3371
val_loss: 0.3371
val_loss: 0.3118

21. Alireza Akhavanpour
Tensorflow & Keras callbacks
EarlyStopping
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping
val_loss: 0.4012
val_loss: 0.3580
0.3118
val_loss: 0.3371
val_loss: 0.3118
val_loss: 0.3606

22. Alireza Akhavanpour
Tensorflow & Keras callbacks
EarlyStopping
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping
val_loss: 0.4012
val_loss: 0.3580
0.3118
val_loss: 0.3371
val_loss: 0.3118
val_loss: 0.3606 0.3606>0.3118

23. Alireza Akhavanpour
Tensorflow & Keras callbacks
EarlyStopping
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping
val_loss: 0.4012
val_loss: 0.3580
0.3118
val_loss: 0.3371
val_loss: 0.3118
val_loss: 0.3606
P=1

24. Alireza Akhavanpour
Tensorflow & Keras callbacks
EarlyStopping
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping
val_loss: 0.4012
val_loss: 0.3580
0.3118
val_loss: 0.3371
val_loss: 0.3118
val_loss: 0.3606
val_loss: 0.3322
P=1

25. Alireza Akhavanpour
Tensorflow & Keras callbacks
EarlyStopping
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping
val_loss: 0.4012
val_loss: 0.3580
0.3118
val_loss: 0.3371
val_loss: 0.3118
val_loss: 0.3606
P=2
val_loss: 0.3322
0.3322>0.3118
Stop
train

26. Alireza Akhavanpour
Tensorflow & Keras callbacks
EarlyStopping
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping

27. Alireza Akhavanpour
Tensorflow & Keras callbacks
EarlyStopping – Quiz 1 …
 Epoch 1/10: val_loss: 0.4343
 Epoch 2/10: val_loss: 0.3553
 Epoch 3/10: val_loss: 0.3347
 Epoch 4/10: val_loss: 0.3448
 Epoch 5/10: val_loss: 0.3170
 Epoch 6/10: val_loss: 0.3568
 Epoch 7/10: val_loss: 0.3328
 Epoch 8/10: val_loss: 0.3197
 Epoch 9/10: val_loss: 0.3171
 Epoch 10/10: val_loss: 0.3400
1 3 5 7 9
Suppose you are using early stopping mechanism with patience
as 3, at which point will the neural network model stop training?

28. Alireza Akhavanpour
Tensorflow & Keras callbacks
EarlyStopping – Quiz2 …
Suppose you are using early stopping mechanism with patience
as 2, at which point will the neural network model stop training?
A) 2 B) 3
C) 4 D) 5
https://www.analyticsvidhya.com/blog/2017/08/skilltest-deep-learning/

29. Alireza Akhavanpour
Tensorflow & Keras callbacks
History
• This callback is automatically applied to every Keras model.
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/History

30. Alireza Akhavanpour
Tensorflow & Keras callbacks
LambdaCallback

31. Alireza Akhavanpour
Tensorflow & Keras callbacks
LearningRateScheduler
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/LearningRateScheduler
• Arguments:
• schedule:
• A function that takes an epoch index as input (integer, indexed from 0)
and returns a new learning rate as output (float).
• verbose:
• 0: quiet
• 1: update messages

32. Alireza Akhavanpour
Tensorflow & Keras callbacks
LearningRateScheduler
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/LearningRateScheduler

33. Alireza Akhavanpour
Tensorflow & Keras callbacks
ModelCheckpoint
• Filepath can contain named formatting options
o will be filled the value of epoch and keys in logs
(passed in on_epoch_end).
o Example: weights.{epoch:02d}-{val_loss:.2f}.hdf5
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint

34. Alireza Akhavanpour
Tensorflow & Keras callbacks
ModelCheckpoint
• filepath:
o String
o path to save the model file.
• monitor:
o quantity to monitor.
• verbose:
o verbosity mode, 0 or 1.
• save_best_only:
o if save_best_only=True
o the latest best model according to the quantity monitored will not be overwritten.
o If filepath doesn't contain formatting options like {epoch} then filepath
o will be overwritten by each new better model.

35. Alireza Akhavanpour
Tensorflow & Keras callbacks
ModelCheckpoint
• mode:
• one of {auto, min, max}
• If save_best_only=True
• the decision to overwrite the current save file is made based on either the maximization or the minimization of
the monitored quantity.
• For val_acc, this should be max,
• for val_loss this should be min, etc.
• In auto mode, the direction is automatically inferred from the name of the monitored quantity.
• save_weights_only:
• If True
• then only the model's weights will be saved
• model.save_weights(filepath)
• Else
• the full model is saved
• model.save(filepath)

36. Alireza Akhavanpour
Tensorflow & Keras callbacks
ModelCheckpoint
• save_freq:
• 'epoch' or integer.
• When using 'epoch‘
• the callback saves the model after each epoch.
• When using integer,
• the callback saves the model at end of a batch at which this many samples have been seen since last saving.
• Note that if the saving isn't aligned to epochs, the monitored metric may potentially be less
reliable (it could reflect as little as 1 batch, since the metrics get reset every epoch). Defaults
to 'epoch'

37. Alireza Akhavanpour
Tensorflow & Keras callbacks
ModelCheckpoint
• you can continually save the model during and at the end of training
• If you are using custom callbacks which have internal variables that change
during a training process, you need to address this when resuming by
initializing these callbacks differently.
• According to the documentation of Keras, a saved model (saved with
model.save(filepath)) contains the following:
 The architecture of the model, allowing to re-create the model
 The weights of the model
 The training configuration (loss, optimizer)
 The state of the optimizer, allowing to resume training exactly where you left off.

38. Alireza Akhavanpour
Tensorflow & Keras callbacks
resuming training

39. Alireza Akhavanpour
Tensorflow & Keras callbacks
Saving custom objects
• HDF5 VS SavedModel
HDF5 uses object configs to save the model architecture
SavedModel saves the execution graph
 SavedModels are able to save custom objects like subclassed models and custom layers
without requiring the orginal code.
https://www.tensorflow.org/tutorials/keras/save_and_load

40. Alireza Akhavanpour
Tensorflow & Keras callbacks
ProgbarLogger

41. Alireza Akhavanpour
Tensorflow & Keras callbacks
ReduceLROnPlateau

42. Alireza Akhavanpour
Tensorflow & Keras callbacks
TerminateOnNaN
tf.keras.callbacks.TerminateOnNaN()
• Callback that terminates training when a NaN loss is encountered.

43. Alireza Akhavanpour
Tensorflow & Keras callbacks
TerminateOnNaN
https://github.com/keras-team/keras/blob/master/keras/callbacks/callbacks.py#L524

44. Alireza Akhavanpour
Tensorflow & Keras callbacks
Quiz!? Stop training…
Change the previous code to stop training when
accuracy is higher than a specific value!
Answer: stop_training.ipynb

45. Alireza Akhavanpour
Tensorflow & Keras callbacks
‫کنید‬ ‫دنبال‬ ‫را‬ ‫ما‬...
https://t.me/cvision
https://www.aparat.com/cvision
https://www.linkedin.com/company/class-vision/
http://class.vision
http://github.com/alireza-akhavan/

46. Alireza Akhavanpour
Tensorflow & Keras callbacks
Sources
o https://www.tensorflow.org/guide/keras/custom_callback
o https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/CSVLogger
o https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/BaseLogger
o https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping
o https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/History
o https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/LearningRateScheduler
o https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint
o https://www.tensorflow.org/tutorials/keras/save_and_load
o https://www.coursera.org/lecture/introduction-tensorflow/using-callbacks-to-control-training-AIkt8
o https://medium.com/analytics-vidhya/groom-your-model-using-keras-callbacks-e97b4fa1b21c
o https://medium.com/analytics-vidhya/tensorflow-2-0-save-and-restore-models-4708ed3f0d8
o https://towardsdatascience.com/resuming-a-training-process-with-keras-3e93152ee11a
o https://www.tensorflow.org/tutorials/keras/save_and_load