Concept Drift: Monitoring Model Quality In Streaming ML Applications
•
4 likes•3,802 views
Most machine learning algorithms are designed to work with stationary data. Yet, real-life streaming data is rarely stationary. Machine learned models built on data observed within a fixed time period usually suffer loss of prediction quality due to what is known as concept drift. The most common method to deal with concept drift is periodically retraining the models with new data. The length of the period is usually determined based on cost of retraining. The changes in the input data and the quality of predictions are not monitored, and the cost of inaccurate predictions is not included in these calculations. A better alternative is monitoring the model quality by testing the inputs and predictions for changes over time, and using change points in retraining decisions. There has been significant development in this area within the last two decades. In this webinar, Emre Velipasaoglu, Principal Data Scientist at Lightbend, Inc., will review the successful methods of machine learned model quality monitoring.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Concept Drift: Monitoring Model Quality In Streaming ML Applications
Similar to Concept Drift: Monitoring Model Quality In Streaming ML Applications (20)
More from Lightbend
More from Lightbend (20)
Recently uploaded
Recently uploaded (20)
Concept Drift: Monitoring Model Quality In Streaming ML Applications
- 4. * same data generating distribution (Some algorithms tolerate violation of this to a certain degree.) training set operation=* core problem
- 11. classifiermargin can active learning help? color size
- 12. classifiermargin can active learning help? color size
- 13. classifiermargin can active learning help? color size
- 14. classifiermargin can active learning help? color size
- 15. classifiermargin can active learning help? color size
- 16. classifiermargin can active learning help? color size
- 17. a better solution data classifier feature extraction predictions
- 18. a better solution data classifier feature extraction predictions monitoring
- 19. a better solution data classifier feature extraction predictions monitoring
- 20. a better solution data classifier feature extraction predictions monitoring
- 21. a better solution data classifier feature extraction predictions monitoring
- 22. a better solution data classifier labeling - feature extraction predictions monitoring
- 23. a better solution data classifier labeling -change detection adaptation feature extraction predictions
- 24. monitor how? supervised unsupervised statistical process control sequential analysis error distribution monitoring clustering / novelty detection feature distribution monitoring model-dependent monitoring
- 25. adapt how? explicit mechanisms implicit mechanisms windowing weighting sampling pure methods ensemble methods
- 26. which method?
- 27. monitor how? supervised unsupervised statistical process control sequential analysis error distribution monitoring clustering / novelty detection feature distribution monitoring model-dependent monitoring
- 28. ML theory: samples errors
- 29. statistical process control • Drift Detection Method [DDM] • # of errors is Binomial: • alert:
- 30. statistical process control • Early Drift Detection Method [EDDM] • distance between errors better for gradual drift • warn & start caching: • alert and reset max: • Drift Detection Method [DDM] • # of errors is Binomial: • alert:
- 31. monitor how? supervised unsupervised statistical process control sequential analysis error distribution monitoring clustering / novelty detection feature distribution monitoring model-dependent monitoring
- 32. sequential analysis • Linear Four Rates [LFR] • stationary data => constant contingency table 0 1 0 TN FN 1 FP TP Predicted True
- 33. sequential analysis • Linear Four Rates [LFR] • stationary data => constant contingency table • calculate four rates 0 1 0 TN FN 1 FP TP Predicted True
- 34. sequential analysis • Linear Four Rates [LFR] • stationary data => constant contingency table • calculate four rates • incremental updates 0 1 0 TN FN 1 FP TP Predicted True
- 35. sequential analysis • Linear Four Rates [LFR] • stationary data => constant contingency table • calculate four rates • incremental updates • test for change • Monte Carlo sampling for significance level • Bonferoni correction for correlated tests • O(1) • Better than (E)DDM for class imbalance 0 1 0 TN FN 1 FP TP Predicted True
- 36. monitor how? supervised unsupervised statistical process control sequential analysis error distribution monitoring clustering / novelty detection feature distribution monitoring model-dependent monitoring
- 37. error distribution monitoring • ADaptive WINdowing [ADWIN] • Consider all partitions of a window • Drop the last element if any • Efficient version O(log W) • Data structure for windows ~ exponential histograms • Drop last window rather than last element w0 w1 prediction errors
- 38. resampling • Prediction loss over random permutations vs. ordered training data • Parallel permutation test version available • Still expensive • Only method directly applicable to regression setting • Side note: Even with finite training set, drift could be problematic if model is developed naively.
- 39. monitor how? supervised unsupervised statistical process control sequential analysis error distribution monitoring clustering / novelty detection feature distribution monitoring model-dependent monitoring
- 40. clustering / novelty detection • OLINDDA: K-means, periodically merge unknown to known or flag • MINAS: micro-clusters, incremental stream clustering • DETECTNOD: Discrete Cosine Transform to estimate distances efficiently • Woo-ensemble: Treat outliers as potential emerging class centroids • ECSMiner: Store and use cluster summary efficiently • GC3: Grid based clustering size color
- 41. Curse of Dimensionality clustering / novelty detection size color • OLINDDA: K-means, periodically merge unknown to known or flag • MINAS: micro-clusters, incremental stream clustering • DETECTNOD: Discrete Cosine Transform to estimate distances efficiently • Woo-ensemble: Treat outliers as potential emerging class centroids • ECSMiner: Store and use cluster summary efficiently • GC3: Grid based clustering
- 42. monitor how? supervised unsupervised statistical process control sequential analysis error distribution monitoring clustering / novelty detection feature distribution monitoring model-dependent monitoring
- 43. feature distribution monitoring • Monitor individual features • Many ways to compare: • Pearson correlation [Change of Concept - CoC] • Hellinger distance [HDDDM] ~ O(DB) • PCA to reduce the number of features to track (top [PCA-1] or bottom [PCA-2] n%) w0 w1 color size samples
- 44. monitor how? supervised unsupervised statistical process control sequential analysis error distribution monitoring clustering / novelty detection feature distribution monitoring model-dependent monitoring
- 45. model-dependent monitoring • Not all changes matter • Posterior probability estimate • Use [A-distance] ~ generalized Kolmogorov-Simirnov distance • designed to be less sensitive to irrelevant changes L1-distance KS-distance A-distance
- 46. model-dependent monitoring • [Margin] distribution • rank statistic on density estimates for a binary representation of the data, • compare average margins of a linear classifier induced by the 1-norm SVM • based on the average zero-one or sigmoid error rate of an SVM classifier • Generalized margin [MD3]: • Embed base classifier in a Random Feature Bagged Ensemble • Margin == high disagreement region of the ensemble m argin “margin” “margin”
- 47. adapt how? explicit mechanisms implicit mechanisms windowing weighting sampling pure methods ensemble methods
- 48. explicit mechanisms for adaptation W stationary W drift ADWIN Drop the last sub-window if threshold is exceeded. = Adaptively shrink window during drift.
- 49. explicit mechanisms for adaptation * Adaptation goes through a similar refinement process. JIT w 0 m 0 m 1 m 2 m 3 m 4 I 0 I 1 I 2 I 3 I 4 change detected * w 1 w 2 w 3 w 4
- 50. adapt how? explicit mechanisms implicit mechanisms windowing weighting sampling pure methods ensemble methods
- 51. explicit mechanisms for adaptation Biased Reservoir Sampling bias: capacity: overwrite / exchange randomly w/ Prob{ %full } or append
- 52. adapt how? explicit mechanisms implicit mechanisms windowing weighting sampling pure methods ensemble methods
- 53. implicit mechanisms for adaptation Ensemble Based Adaptation ensemble 1 ensemble (N-1) ensemble N train new member
- 54. implicit mechanisms for adaptation retire / decay train new member ensemble 1 ensemble (N-1) ensemble N Ensemble Based Adaptation
- 55. implicit mechanisms for adaptation retire / decay recurring train new member ensemble 1 ensemble (N-1) ensemble N Ensemble Based Adaptation
- 56. implicit mechanisms for adaptation • Online NonStationary boosting [ONSboost] • NonStationary Random Forests [NSRF] • Dynamic Weighted Majority [DWM] • Learn++ for NonStationary Environments [Learn++.NSE] retire / decay recurring train new member ensemble 1 ensemble (N-1) ensemble N Ensemble Based Adaptation
- 57. which method? Method Efficiency Pros Cons Notes DDM/EDDM O(1) no data stored label cost false alarms sampling necessary in case of fast data, microservices architecture ideal LFR O(1) class imbalance OK label cost ADWIN O(log W) better change localization label cost JIT O(log W) no labels required only for abrupt changes best localization
- 58. which method? Method Efficiency Pros Cons Notes ECSMiner / GC3 O(W 2 / k) O(G log C) emerging concepts clusterable drift only use if emerging concepts expected HDDDM O(DB) no labels not for population drift or class imbalance better when combined with PCA A-distance O(log W) no labels less false positives compared to HDDDM good choice for unsupervised Margin / MD3 Learning, detection, adaptation bundled reduced false alarms must use feature bagged ensembles best choice but must commit to using the specific machine learning algorithmsEnsemble methods recurring concepts large batches
- 60. Check out these resources: Dean’s book Webinars etc.Fast Data Architectures for Streaming Applications Getting Answers Now from Data Sets that Never End By Dean Wampler, Ph. D., VP of Fast Data Engineering 60 LIGHTBEND.COM/LEARN
- 61. Serving Machine Learning Models A Guide to Architecture, Stream Processing Engines, and Frameworks By Boris Lublinsky, Fast Data Platform Architect 61 LIGHTBEND.COM/LEARN