Model Drift Monitoring using Tensorflow Model Analysis

1. Model Drift Monitoring using
Tensorflow Model Analysis (TFMA)
Vivek Raja P S

2. A little about myself...
● From Madurai, Tamil Nadu
● Bachelor’s in Computer Science and Engineering (2020)
● Founder & Organizer of Tamil FOSS Community
● 14x Multi-cloud certified (Azure, Oracle & Alibaba)
● Microsoft Certified Data Scientist Associate, AI Engineer
Associate, Data Engineer Associate
● OCI Certified Solution Architect Professional and
Developer Associate
● 15x Hackathon Winner
● Published 3 AI & IoT research papers, 1 patent
● Loves to play guitar, Learning French, Binge watching
Netflix

3. Session’s agenda
● Introduction to ML in Production
● Model Drift Monitoring (MDM) – What definition states
● MDM from production point of view
● How Model Drift can be monitored using TFMA
● Demo
● Queries

4. ML In Production

6. We all know to train, test,
deploy ML Models
But what really happens
after that?

7. Credits: Laurence Moroney

8. What do we
need to care
about?
● Packaging the model
● Finding right
infrastructure
● Deploying model
● High availability API
● Versioning the model
● Monitoring the model

9. Model Drift Monitoring
- What definition states

10. Understanding Model Drift
Many machine learning models tend to be black boxes, where
explainability is very limited, which can make it difficult to
understand why a model is not performing as expected. This is
especially true with regard to how a model performs over time with
new training data.

11. What is concept drift?
A model that was initially working pretty well could later degrade due
to a concept called data drift or concept drift. Data drift occurs when
the underlying statistical structure of your data changes over time.

12. Lets understand with an example
Technically, predictive modeling is the problem of approximating a mapping function (f)
given input data (X) to predict an output value (y).
Often, this mapping is assumed to be static, meaning that the mapping learned from
historical data is just as valid in the future on new data and that the relationships between
input and output data do not change.

13. Lets understand with an example
The change to the data could take any form
● A gradual change over time.
● A recurring or cyclical change.
● A sudden or abrupt change.
There are domains where predictions are ordered by time, such as time series
forecasting and predictions on streaming data where the problem of concept
drift is more likely and should be explicitly tested for and addressed.

14. Lets understand with an example
Forecasting COVID-19 cases in India -
bit.ly/vivekarticle
Website: bit.ly/covidtrack

15. Model Drift Monitoring
-production point of view

16. From Production POV
Indre Zliobaite in the 2010 paper titled “Learning under Concept Drift: An Overview”
● Future assumption: a designer needs to make an assumption about the future data
source.
● Change type: a designer needs to identify possible change patterns.
● Learner adaptivity: based on the change type and the future assumption, a designer
chooses the mechanisms which make the learner adaptive.
● Model selection: a designer needs a criterion to choose a particular parametrization of
the selected learner at every time step (e.g. the weights for ensemble members, the
window size for variable window method).

17. From Production POV
● Defining critical thresholds, detecting violations of these thresholds, and safeguarding the
machine learning production system from degradation is the main goal of model
monitoring.
● Model monitoring is very similar to continuous integration/continuous deployment
(CI/CD) in traditional software development. In CI/CD systems, you monitor the whole
software development and deployment lifecycle using automated tools and alerting. The
goal of model monitoring is to bring some of these well-established rules and systems
into modern production systems that utilize machine learning.

18. From Production POV
● Data drift:
● Model learning capacity:
● Performance degrading change:
● Maintenance and manual deployment cycles:

19. Tensorflow Model Analysis

20. Overview
TensorFlow Model Analysis (TFMA) is a library for performing model
evaluation.
● For: Machine Learning Engineers or Data Scientists
● who: want to analyze and understand their TensorFlow models
● it is: a standalone library or component of a TFX pipeline
● that: evaluates models on large amounts of data in a distributed manner on
the same metrics defined in training. These metrics are compared over
slices of data, and visualized in Jupyter or Colab notebooks.
● unlike: some model introspection tools like tensorboard that offer model
introspection

21. Overview
TensorFlow Model Analysis allows you to perform model evaluations in the TFX
pipeline, and view resultant metrics and plots in a Jupyter notebook. Specifically, it
can provide:
● Metrics computed on entire training and holdout dataset, as well as next-day
evaluations
● Tracking metrics over time
● Model quality performance on different feature slices
● Model validation for ensuring that model's maintain consistent performance

22. Time to code

23. Open for Q&A

25. Let’s connect
Email ID: vivekraja98@gmail.com
Linkedin: @Vivek Raja P S GitHub: @Vivek0712 Twitter: @vivekraja007