Proposal for Linking Concept Drift and uncertainty of Machine learning
•
0 likes•157 views
This is my proposal for studying the use of uncertainty of ML predictions as a means of detecting concept drifts.
![PAPER – CONCEPT DRIFT + UNCERTAINTY
Wednesday, November 13, 2019 8:32 AM
BROAD AIM (within PHD):
• Study the relationship between concept drift and uncertainty of
ML
Specific aims:
• Investigate the effectiveness of using ML uncertainties to detect
concept drift
○ Aleatoric (which can be further divided into:)
▪ Homoscedestic [1]
▪ Heteroskedestic [1]
○ Epistemic [1]
• Two types of learning (supervised & unsupervised) will be
employed within families of Bayesian Neural Network and a
combination of both (semi-supervised)
○ Supervised learning (Vanilla BNN)
○ Unsupervised learning (Bayesian Autoencoder)
○ Semi-supervised learning (A combination of both)
Background & Motivation
• In manufacturing context, data of CPMS is characterized as
dynamic, uncertain and evolving distributions. The use of batch
training with the assumption that the data distribution remains
stationary is easily challenged and hence, ML systems are
required to detect these change-points in data distribution also
known as concept drifts and subsequently, adapt the ML models.
• From [2]
• We propose the use of uncertainties of predictions in Bayesian
Neural Networks to detect concept drifts.
• The possible benefits are :
○ No need of external detection methods. The uncertainties
quantification and calibration are incorporated during
training of BNN models and as such, we can readily get CD
detection for 'free' in addition to accurate predictions.
○ Quantify degree of severity of CD and knowing where CD
occurs in the latent space (Concept drift understanding) [2]
• However, the use of BNN in this context is not straightforward as
there are many approximation methods to achieving it and to
date, the best method remains an open question [3]. For
instance, some research emphasized that the uncertainty of BNN
trained using Factorised Gaussian and MC Dropout is unreliable
[4].
• The use of uncertainty is relatively new, and only recently, a
paper proposed the use of uncertainty of SVM for concept drift
[5].
Nomenclature:
• CPMS – Cyberphysical manufacturing
systems
• CD – Concept Drift
• BNN – Bayesian Neural Network](https://image.slidesharecdn.com/paper-concept-drift-uncertainty-191125001729/85/Proposal-for-Linking-Concept-Drift-and-uncertainty-of-Machine-learning-1-320.jpg)
![Experiment setup
• All experiments will be implemented using agentMET4FOF
software package
• Datasets (limited to time-series sensors
○ Synthetic data [2]
▪ SEA concepts
▪ Sine, waveform
○ Real data [2]
• ML Models (to be implemented)
○ BNN
○ Variational Autoencoder
• Variables
○ Types of Bayesian inference
▪ MC Dropout
▪ Ensembles
▪ Variational inference
▪ Etc...
○ Types of uncertainty
▪ Aleatoric uncertainty
▪ Epistemic uncertainty
○ Uncertainty thresholds level
▪ Q: How to determine the threshold level?
• Performance metric of using uncertainties as drift detection [2]
[5]
○ True positive (Sensitivity)
○ True negative (Specificity)
○ Delay in detection
References
1. Kendall, Alex, and Yarin Gal. "What uncertainties do we need in bayesian deep learning for
computer vision?." Advances in neural information processing systems. 2017.
2. Lu, Jie, et al. "Learning under concept drift: A review." IEEE Transactions on Knowledge and
Data Engineering (2018).
3. Yao, Jiayu, et al. "Quality of Uncertainty Quantification for Bayesian Neural Network
Inference." arXiv preprint arXiv:1906.09686 (2019).
4. Foong, Andrew YK, et al. "Pathologies of Factorised Gaussian and MC Dropout Posteriors in
Bayesian Neural Networks." arXiv preprint arXiv:1909.00719 (2019).
5. Yu, Shujian, Xiaoyang Wang, and José C. Príncipe. "Request-and-reverify: hierarchical
hypothesis testing for concept drift detection with expensive labels." arXiv preprint
arXiv:1806.10131 (2018).](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Recommended
More Related Content
What's hot
What's hot (17)
Similar to Proposal for Linking Concept Drift and uncertainty of Machine learning
Similar to Proposal for Linking Concept Drift and uncertainty of Machine learning (20)
More from Bang Xiang Yong
More from Bang Xiang Yong (7)
Recently uploaded
Recently uploaded (20)
Proposal for Linking Concept Drift and uncertainty of Machine learning
- 1. PAPER – CONCEPT DRIFT + UNCERTAINTY Wednesday, November 13, 2019 8:32 AM BROAD AIM (within PHD): • Study the relationship between concept drift and uncertainty of ML Specific aims: • Investigate the effectiveness of using ML uncertainties to detect concept drift ○ Aleatoric (which can be further divided into:) ▪ Homoscedestic [1] ▪ Heteroskedestic [1] ○ Epistemic [1] • Two types of learning (supervised & unsupervised) will be employed within families of Bayesian Neural Network and a combination of both (semi-supervised) ○ Supervised learning (Vanilla BNN) ○ Unsupervised learning (Bayesian Autoencoder) ○ Semi-supervised learning (A combination of both) Background & Motivation • In manufacturing context, data of CPMS is characterized as dynamic, uncertain and evolving distributions. The use of batch training with the assumption that the data distribution remains stationary is easily challenged and hence, ML systems are required to detect these change-points in data distribution also known as concept drifts and subsequently, adapt the ML models. • From [2] • We propose the use of uncertainties of predictions in Bayesian Neural Networks to detect concept drifts. • The possible benefits are : ○ No need of external detection methods. The uncertainties quantification and calibration are incorporated during training of BNN models and as such, we can readily get CD detection for 'free' in addition to accurate predictions. ○ Quantify degree of severity of CD and knowing where CD occurs in the latent space (Concept drift understanding) [2] • However, the use of BNN in this context is not straightforward as there are many approximation methods to achieving it and to date, the best method remains an open question [3]. For instance, some research emphasized that the uncertainty of BNN trained using Factorised Gaussian and MC Dropout is unreliable [4]. • The use of uncertainty is relatively new, and only recently, a paper proposed the use of uncertainty of SVM for concept drift [5]. Nomenclature: • CPMS – Cyberphysical manufacturing systems • CD – Concept Drift • BNN – Bayesian Neural Network
- 2. Experiment setup • All experiments will be implemented using agentMET4FOF software package • Datasets (limited to time-series sensors ○ Synthetic data [2] ▪ SEA concepts ▪ Sine, waveform ○ Real data [2] • ML Models (to be implemented) ○ BNN ○ Variational Autoencoder • Variables ○ Types of Bayesian inference ▪ MC Dropout ▪ Ensembles ▪ Variational inference ▪ Etc... ○ Types of uncertainty ▪ Aleatoric uncertainty ▪ Epistemic uncertainty ○ Uncertainty thresholds level ▪ Q: How to determine the threshold level? • Performance metric of using uncertainties as drift detection [2] [5] ○ True positive (Sensitivity) ○ True negative (Specificity) ○ Delay in detection References 1. Kendall, Alex, and Yarin Gal. "What uncertainties do we need in bayesian deep learning for computer vision?." Advances in neural information processing systems. 2017. 2. Lu, Jie, et al. "Learning under concept drift: A review." IEEE Transactions on Knowledge and Data Engineering (2018). 3. Yao, Jiayu, et al. "Quality of Uncertainty Quantification for Bayesian Neural Network Inference." arXiv preprint arXiv:1906.09686 (2019). 4. Foong, Andrew YK, et al. "Pathologies of Factorised Gaussian and MC Dropout Posteriors in Bayesian Neural Networks." arXiv preprint arXiv:1909.00719 (2019). 5. Yu, Shujian, Xiaoyang Wang, and José C. Príncipe. "Request-and-reverify: hierarchical hypothesis testing for concept drift detection with expensive labels." arXiv preprint arXiv:1806.10131 (2018).