MLOps (Machine Learning Operations) is a recent buzzword, that trends a lot. Let's figure out together how maintaining applications with machine learning components is significantly different from maintaining applications without them. We will look into MLOps best practices and typical problems and their implementations/solutions in real world production.

3. Why maintaining ml in production is so hard?
● Models start to constantly degrade the moment they are used in production
● Model errors become clear only in the future, often after decision was made based on
prediction

4. Agenda
● MLOps engineering, monitoring
● Real world example, some good
practices

5. MLOps engineering

6. MLOps engineering
● Infrastructure
● CI/CD/ML Pipelines
● Automation
● Monitoring
● Trusting your predictions

7. Metric is the key
● Finding good metric is hard
● Some baseline is required
● Measure what matters

8. Accuracy
● Depends on the use case
● Typical metrics: RMSE, LogLoss, Confusion matrix metrics and many others
● Sometimes we might not care about accuracy that much
● Can only be measured after actual value is known

9. Data Drift
● Can be measured even before predictions are made
○ Pro tip: mark outliers for investigation
● Typical metrics: PSI, Kolmogorov-Smirnov, Jensen-Shannon
● It might be hard to monitor all features
○ It is a good idea to limit only to “important” features

10. Covid 19

11. Covid 19

12. Solving covid 19 is extremely hard
● ~3000 different “independent” geographical locations in US
● Forecasts need to go into the far future (>12 weeks)
● World changes daily
○ New policies, new strains, vaccines, population behavior, …

13. Operational complexity
● ~30 data sources collected daily
○ new cases/deaths, population mobility, vaccine distribution, etc
● Results are required daily
● Full pipeline run ~8 hours

14. Data itself is a part of the software

15. Treat data same way you treat code
● Tests for the data
○ Number of new cases > 0
● Versioning
○ semantic versions, every job knows which versions it required and produces
● Backups
○ Delete nothing, just offload to storage

16. What else helped us
● “Trust” checks
○ data/model/sanity/metric checks
● Constant backtesting
● Fine tuning and manual approvals
● Daily retraining
○ If we have all the monitoring and checks, why not?

17. Thanks for attention