발표자: 이기민(KAIST 박사과정)
발표일: 2018.4.
The predictive uncertainty (e.g., entropy of softmax distribution of a deep classifier) is indispensable as it is useful in many machine learning applications (e.g., active learning and ensemble learning) as well as when deploying the trained model in real-world systems. In order to improve the quality of the predictive uncertainty, we proposed a novel loss function for training deep models (ICLR 2018). We showed that confidence deep models trained by our method can be very useful in various machine learning applications such as novelty detection (CVPR 2018) and ensemble learning (ICML 2017).
Uncertainty in Deep Learning, Gal (2016)
Representing Inferential Uncertainty in Deep Neural Networks Through Sampling, McClure & Kriegeskorte (2017)
Uncertainty-Aware Reinforcement Learning from Collision Avoidance, Khan et al. (2016)
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles, Lakshminarayanan et al. (2017)
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?, Kendal & Gal (2017)
Uncertainty-Aware Learning from Demonstration Using Mixture Density Networks with Sampling-Free Variance Modeling, Choi et al. (2017)
Bayesian Uncertainty Estimation for Batch Normalized Deep Networks, Anonymous (2018)
Uncertainty in Deep Learning, Gal (2016)
Representing Inferential Uncertainty in Deep Neural Networks Through Sampling, McClure & Kriegeskorte (2017)
Uncertainty-Aware Reinforcement Learning from Collision Avoidance, Khan et al. (2016)
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles, Lakshminarayanan et al. (2017)
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?, Kendal & Gal (2017)
Uncertainty-Aware Learning from Demonstration Using Mixture Density Networks with Sampling-Free Variance Modeling, Choi et al. (2017)
Bayesian Uncertainty Estimation for Batch Normalized Deep Networks, Anonymous (2018)
Uncertainty Quantification with Unsupervised Deep learning and Multi Agent Sy...Bang Xiang Yong
Presented at MET4FOF Workshop, JULY 2020
I talk about our recent work of combining Bayesian Deep learning with Explainable Artificial Intelligence (XAI) methods. In particular, we look at Bayesian Autoencoders.
Semi supervised, weakly-supervised, unsupervised, and active learningYusuke Uchida
An overview of semi supervised learning, weakly-supervised learning, unsupervised learning, and active learning.
Focused on recent deep learning-based image recognition approaches.
Localization and classification. Overfeat: class agnostic versu class specific localization, fully convolutional neural networks, greedy merge strategy. Multiobject detection. Region proposal and selective search. R-CNN, Fast R-CNN, Faster R-CNN and YOLO. Image segmentation. Semantic segmentation and transposed convolutions. Instance segmentation and Mask R-CNN. Image captioning. Recurrent Neural Networks (RNNs). Language generation. Long Short Term Memory (LSTMs). DeepImageSent, Show and Tell, and Show, Attend and Tell algorithms.
Uncertainty Quantification with Unsupervised Deep learning and Multi Agent Sy...Bang Xiang Yong
Presented at MET4FOF Workshop, JULY 2020
I talk about our recent work of combining Bayesian Deep learning with Explainable Artificial Intelligence (XAI) methods. In particular, we look at Bayesian Autoencoders.
Semi supervised, weakly-supervised, unsupervised, and active learningYusuke Uchida
An overview of semi supervised learning, weakly-supervised learning, unsupervised learning, and active learning.
Focused on recent deep learning-based image recognition approaches.
Localization and classification. Overfeat: class agnostic versu class specific localization, fully convolutional neural networks, greedy merge strategy. Multiobject detection. Region proposal and selective search. R-CNN, Fast R-CNN, Faster R-CNN and YOLO. Image segmentation. Semantic segmentation and transposed convolutions. Instance segmentation and Mask R-CNN. Image captioning. Recurrent Neural Networks (RNNs). Language generation. Long Short Term Memory (LSTMs). DeepImageSent, Show and Tell, and Show, Attend and Tell algorithms.
We introduce a new algorithm for image segmentation based on crowdsourcing through a game : Ask'nSeek. The game provides information on the objects of an image, under
the form of clicks that are either on the object, or on the background. These logs are then used in order to determine the best segmentation for an object among a set of candidates generated by the state-of-the-art CPMC algorithm. We also introduce a simulator that allows the generation of game logs and therefore gives insight about the number of games needed on an image to perform acceptable segmentation.
Presented by Amaia Salvador in CrowdMM 2013 (http://crowdmm.org/).
More info:
https://imatge.upc.edu/web/publications/crowdsourced-object-segmentation-game-0
A Practical Use of Artificial Intelligence in the Fight Against Cancer by Bri...Data Con LA
Abstract:- Artificial Intelligence is an important topic in the fight against cancer. Clinical Trails are at the frontier of innovation. I will discuss techniques, data sets and platforms we use at Deep 6 to bring patients to clinical trials. The focus will be on practical, repeatable methods I've developed at MySpace, Greenplum, UCLA and the US Intelligence Community.
With the explosive growth of online information, recommender system has been an effective tool to overcome information overload and promote sales. In recent years, deep learning's revolutionary advances in speech recognition, image analysis and natural language processing have gained significant attention. Meanwhile, recent studies also demonstrate its efficacy in coping with information retrieval and recommendation tasks. Applying deep learning techniques into recommender system has been gaining momentum due to its state-of-the-art performance. In this talk, I will present recent development of deep learning based recommender models and highlight some future challenges and open issues of this research field.
Malicious software are categorized into families based on
their static and dynamic characteristics, infection methods, and nature of threat. Visual exploration of malware instances and families in a low dimensional space helps in giving a first overview about dependencies and
relationships among these instances, detecting their groups and isolating outliers. Furthermore, visual exploration of different sets of features is useful in assessing the quality of these sets to carry a valid abstract representation, which can be later used in classification and clustering algorithms to achieve a high accuracy. We investigate one of
the best dimensionality reduction techniques known as t-SNE to reduce the malware representation from a high dimensional space consisting of
thousands of features to a low dimensional space. We experiment with
different feature sets and depict malware clusters in 2-D.
This presentation is for my Seminar Course at the University of Tehran. in this presentation, I will introduce some of the newest and also exciting developments in Generative Adversarial Networks.
الموعد الإثنين 03 يناير 2022
143
مبادرة
#تواصل_تطوير
المحاضرة ال 143 من المبادرة
المهندس / محمد الرافعي طرباي
نقيب المبرمجين بالدقهلية
بعنوان
"IT INDUSTRY"
How To Getting Into IT With Zero Experience
وذلك يوم الإثنين 03 يناير2022
السابعة مساء توقيت القاهرة
الثامنة مساء توقيت مكة المكرمة
و الحضور من تطبيق زووم
https://us02web.zoom.us/meeting/register/tZUpf-GsrD4jH9N9AxO39J013c1D4bqJNTcu
علما ان هناك بث مباشر للمحاضرة على القنوات الخاصة بجمعية المهندسين المصريين
ونأمل أن نوفق في تقديم ما ينفع المهندس ومهمة الهندسة في عالمنا العربي
والله الموفق
للتواصل مع إدارة المبادرة عبر قناة التليجرام
https://t.me/EEAKSA
ومتابعة المبادرة والبث المباشر عبر نوافذنا المختلفة
رابط اللينكدان والمكتبة الالكترونية
https://www.linkedin.com/company/eeaksa-egyptian-engineers-association/
رابط قناة التويتر
https://twitter.com/eeaksa
رابط قناة الفيسبوك
https://www.facebook.com/EEAKSA
رابط قناة اليوتيوب
https://www.youtube.com/user/EEAchannal
رابط التسجيل العام للمحاضرات
https://forms.gle/vVmw7L187tiATRPw9
ملحوظة : توجد شهادات حضور مجانية لمن يسجل فى رابط التقيم اخر المحاضرة
비행기 설계를 왜 통일 해야 할까?
디자인 시스템을 하는 이유
비행기들이 다 용도가 다르다...어떻게 설계하지?
맥락이 다른 페이지와 패턴
경유지까지 아직 멀었다... 언제 수리하지?
디자인 시스템을 적용하는 시점
엔지니어랑 얘기해서 정비해야하는데...어떻게 수리하지?
디자인 시스템을 적용하는 프로세스
비행기 설계가 바뀐걸 어떻게 알리지?
디자인 시스템의 전파
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
2. Algorithmic Intelligence Lab
Outline
• Introduction
• Predictive uncertainty of deep neural networks
• Summary
• How to train confident neural networks
• Training Confidence-Calibrated Classifiers for Detecting Out-of-Distribution
Samples [Lee’ 18a]
• Applications
• Confident Multiple Choice Learning [Lee’ 17]
• Hierarchical novelty detection [Lee’ 18b]
• Conclusion
2
[Lee’ 18a] Lee, K., Lee, H., Lee, K. and Shin, J. Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples. I
n ICLR, 2018.
[Lee’ 17] Lee, K., Hwang, C., Park, K. and Shin, J. Confident Multiple Choice Learning. In ICML, 2017.
[Lee’ 18b] Lee, K., Lee, Min. K, Zhang, Y. Shin. J, Lee, H. Hierarchical Novelty Detection for Visual Object Recognition, In CVPR, 2018.
3. Algorithmic Intelligence Lab
• Supervised learning (e.g., regression and classification)
• Objective: finding an unknown target distribution, i.e., P(Y|X)
• Recent advances in deep learning have dramatically improved accuracy on
several supervised learning tasks
Introduction: Predictive uncertainty of deep neural networks (DNNs)
3
[Amodei’ 16] Amodei, D., Ananthanarayanan, S., Anubhai, R., Bai, J.,
Battenberg, E., Case, C., Casper, J., Catanzaro, B., Cheng, Q., Chen, G. and
Chen, J. Deep speech 2: End-to-end speech recognition in english and
mandarin. In ICML, 2016.
[He’ 16] He, K., Zhang, X., Ren, S. and Sun, J. Deep residual learning for
image recognition. In CVPR, 2016.
[Hershey’ 17] Hershey, S., Chaudhuri, S., Ellis, D.P., Gemmeke, J.F., Jansen, A.,
Moore, R.C., Plakal, M., Platt, D., Saurous, R.A., Seybold, B. and Slaney, M.
CNN architectures for large-scale audio classification. In ICASSP, 2017.
[Girshick’ 15] Girshick, Ross. Fast r-cnn. In ICCV, pp. 1440–1448, 2015
Input space Output space
Objective detection [Girshick’ 15]
Speech recognition
[Amodei’ 16]
Image classification
[He’ 16]
Audio
recognition
[Hershey’ 17]
4. Algorithmic Intelligence Lab
• Uncertainty of predictive distribution is important in DNN’s applications
• What is predictive uncertainty?
• As a example, consider classification task
• It represents a confidence about prediction!
• For example, it can be measured as follows:
• Entropy of predictive distribution [Lakshminarayanan’ 17]
• Maximum value of predictive distribution [Hendrycks’ 17]
Introduction: Predictive uncertainty of deep neural networks (DNNs)
4
[Lakshminarayanan’ 17] Lakshminarayanan, B., Pritzel, A. and Blundell, C., Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS, 2017.
[Henderycks’ 17] Hendrycks, D. and Gimpel, K., A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR 2017.
Persian
cat
tiger
cat
0.12
0.18
Persian
cat
dog
0.99
5. Algorithmic Intelligence Lab
• Predictive uncertainty is related to many machine learning problems:
• Predictive uncertainty is also indispensable when deploying DNNs in
real-world systems [Dario’ 16]
Introduction: Predictive uncertainty of deep neural networks (DNNs)
5
Autonomous drive Secure authentication system
[Dario’ 16] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mane ́. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016.
[Henderycks’ 17] Hendrycks, D. and Gimpel, K., A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR 2017.
[Guo’ 17] Guo, C., Pleiss, G., Sun, Y. and Weinberger, K.Q., 2017. On Calibration of Modern Neural Networks. In ICML 2017.
[Goodfellow’ 14] Goodfellow, I.J., Shlens, J. and Szegedy, C., 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.
[Srivastava’ 14] Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I. and Salakhutdinov, R., Dropout: a simple way to prevent neural networks from overfitting. JMLR. 2014.
Novelty detection
[Hendrycks’ 17]
Adversarial detection
[Song’ 18]
Ensemble learning
[Lee’ 17]
6. Algorithmic Intelligence Lab
• However, DNNs do not capture their predictive uncertainty
• E.g., DNNs trained to classify MNIST images often produce high confident
probability 91% even for random noise [Henderycks’ 17]
• Challenge arises in improving the quality of the predictive uncertainty!
• Main topic of this presentation
• How to train confident neural networks?
• Training confidence-calibrated classifiers for detecting out-of-distribution
samples [Lee’ 18a]
• Applications
• Confident multiple choice learning [Lee’ 17]
• Hierarchical novelty detection [Lee’ 18b]
Introduction: Predictive uncertainty of deep neural networks (DNNs)
6
Unknown image Cat TrainDog
99%
[Henderycks’ 17] Hendrycks, D. and Gimpel, K., A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR 2017.
[Lee’ 18a] Lee, K., Lee, H., Lee, K. and Shin, J. Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples. In ICLR 2018.
[Lee’ 17] Lee, K., Hwang, C., Park, K. and Shin, J. Confident Multiple Choice Learning. In ICML, 2017.
[Lee’ 18b] Lee, K., Lee, Min. K, Zhang, Y. Shin. J, Lee, H. Hierarchical Novelty Detection for Visual Object Recognition, In CVPR, 2018.
7. Algorithmic Intelligence Lab
Outline
• Introduction
• Predictive uncertainty of deep neural networks
• Summary
• How to train confident neural networks
• Training Confidence-Calibrated Classifiers for Detecting Out-of-Distribution
Samples [Lee’ 18a]
• Applications
• Confident Multiple Choice Learning [Lee’ 17]
• Hierarchical novelty detection [Lee’ 18b]
• Conclusion
7
[Lee’ 18a] Lee, K., Lee, H., Lee, K. and Shin, J. Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples. I
n ICLR, 2018.
[Lee’ 17] Lee, K., Hwang, C., Park, K. and Shin, J. Confident Multiple Choice Learning. In ICML, 2017.
[Lee’ 18b] Lee, K., Lee, Min. K, Zhang, Y. Shin. J, Lee, H. Hierarchical Novelty Detection for Visual Object Recognition, In CVPR, 2018.
8. Algorithmic Intelligence Lab
• Related problem
• Detecting out-of-distribution [Hendrycks’ 17]
• Detect whether a test sample is from in-distribution (i.e., training distribution
by classifier) or out-of-distribution
• E.g., image classification
• Assume a classifier trains handwritten digits (denoted as in-distribution)
• Detecting out-of-distribution
• Performance of detector reflects confidence of predictive distribution!
How to Train Confident Neural Networks?
8
In-distribution Out-of-distribution
Predictive dist.
Data
[Henderycks’ 17] Hendrycks, D. and Gimpel, K., A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR 2017.
[Guo’ 17] Guo, C., Pleiss, G., Sun, Y. and Weinberger, K.Q., 2017. On Calibration of Modern Neural Networks. In ICML 2017.
[Liang’ 17] Liang, S., Li, Y. and Srikant, R., 2017. Principled Detection of Out-of-Distribution Examples in Neural Networks. arXiv preprint arXiv:1706.02690.
9. Algorithmic Intelligence Lab
• Threshold-based Detector [Guo’ 17, Hendrycks’17, Liang’ 18]
• How to define the score?
• Baseline detector [Hendrycks’17]
• Confidence score = maximum value of predictive distribution
• Temperature scaling [Guo’ 17]
• Confidence score = maximum value of scaled predictive distribution
• Limitations
• Performance of prior works highly depends on how to train the classifiers
Related Work
9
[Input] [Classifier]
score
10
If score > 𝜖: In-distribution
Else: out-of-distribution
[Henderycks’ 17] Hendrycks, D. and Gimpel, K., A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR 2017.
[Guo’ 17] Guo, C., Pleiss, G., Sun, Y. and Weinberger, K.Q., 2017. On Calibration of Modern Neural Networks. In ICML 2017.
[Liang’ 17] Liang, S., Li, Y. and Srikant, R., 2017. Principled Detection of Out-of-Distribution Examples in Neural Networks. In ICLR, 2018.
Output of neural networks
10. Algorithmic Intelligence Lab
• Main components of our contribution
• New loss
• Confident loss for confident classifier
• New generative adversarial network (GAN)
• GAN for generating out-of-distribution samples
• New training method
• Joint training of classifier and GAN
• Experimental results
• Our method drastically improves the detection performance
• VGGNet trained by our method improves TPR compared to the baseline:
• 14.0%39.1% and 46.3% 98.9% on CIFAR-10 and SVHN, respectively
• Providing visual understandings on the proposed method
Our Contributions
10
11. Algorithmic Intelligence Lab
• Confident loss
• Minimize the KL divergence on data from out-of-distribution
• Interpretation
• Assigning higher maximum prediction values to in-distribution samples than o
ut-of-distribution ones
Contribution 1: Confident Loss
11
Data from in-dist Data from out-of-dist
Data distribution Uniform distribution
“Zero confidence”
12. Algorithmic Intelligence Lab
• Confident loss
• Minimize the KL divergence on data from out-of-distribution
• Interpretation
• Assigning higher maximum prediction values to in-distribution samples than o
ut-of-distribution ones
• Effects of confidence loss
• Fraction of the maximum prediction value from simple CNNs (2 Conv + 3 FC)
• KL divergence term is optimized using CIFAR-10 training data
Contribution 1: Confident Loss
12
Data from in-dist Data from out-of-dist
13. Algorithmic Intelligence Lab
• Main issues of confidence loss
• How to optimize the KL divergence loss?
• The number of out-of-distribution samples might be almost infinite to cover
the entire space
• Our intuition
• Samples close to in-distribution could be more effective in improving the
detection performance
Contribution 2. GAN for Generating Out-of-Distribution Samples
13
14. Algorithmic Intelligence Lab
• New GAN objective
• Term (a) forces the generator to generate low-density samples
• (approximately) minimizing the log negative likelihood of in-distribution
• Term (b) corresponds to the original GAN loss
• Generating out-of-distribution samples close to in-distribution
• Experimental results on toy example and MNIST
Contribution 2. GAN for Generating Out-of-Distribution Samples
14
15. Algorithmic Intelligence Lab
• We suggest training the proposed GAN using a confident classifier
• Converse is also possible
• We propose a joint confidence loss
• Classifier’s confidence loss: (c) + (d)
• GAN loss: (d) + (e)
• Alternating algorithm for optimizing the joint confidence loss
Contribution 3. Joint Confidence Loss
15
Step 2. update classifier
Classifier
GAN
Step 1. update GAN
Classifier
GAN
16. Algorithmic Intelligence Lab
• TP = true positive
• FN = false negative
• TN = true negative
• FP = false positive
• FPR at 95% TPR
• FPR = FP/(FP + TN), TPR = TP/(TP + FN)
• AUROC (Area Under the Receiver Operating Characteristic curve)
• ROC curve = relationship between TPR and FPR
• Detection Error
• Minimum misclassification probability over all thresholds
• AUPR (Area under the Precision-Recall curve)
• PR curve = relationship between precision=TP/(TP+FP) and recall=TP/(TP+FN)
Experimental Results - Metric
16
17. Algorithmic Intelligence Lab
• Measure the detection performance of threshold-based detectors
• Confidence loss with some explicit out-of-distribution dataset
• Classifier trained by our method drastically improves the detection
performance across all out-of-distributions
Experimental Results
17
Realistic images such as TinyImageNet (aqua line) and
LSUN(green line) are more useful than synthetic datasets
(orange line) for improving the detection perfor-mance
18. Algorithmic Intelligence Lab
• Joint confidence loss
• Confidence loss with the original GAN (orange bar) is often useful for
improving the detection performance
• Joint confidence loss (bluebar) still outperforms all baseline it in all cases
Experimental Results
18
19. Algorithmic Intelligence Lab
• Interpretability of trained classifier
• Classifier trained by cross entropy loss shows sharp gradient maps for both
samples from in- and out-of-distributions
• Classifiers trained by the confidence losses do only on samples from in-
distribution.
Experimental Results
19
20. Algorithmic Intelligence Lab
Outline
• Introduction
• Predictive uncertainty of deep neural networks
• Summary
• How to train confident neural networks
• Training Confidence-Calibrated Classifiers for Detecting Out-of-Distribution
Samples [Lee’ 18a]
• Applications
• Confident Multiple Choice Learning [Lee’ 17]
• Hierarchical novelty detection [Lee’ 18b]
• Conclusion
20
[Lee’ 18a] Lee, K., Lee, H., Lee, K. and Shin, J. Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples. I
n ICLR, 2018.
[Lee’ 17] Lee, K., Hwang, C., Park, K. and Shin, J. Confident Multiple Choice Learning. In ICML, 2017.
[Lee’ 18b] Lee, K., Lee, Min. K, Zhang, Y. Shin. J, Lee, H. Hierarchical Novelty Detection for Visual Object Recognition, In CVPR, 2018.
21. Algorithmic Intelligence Lab
• Ensemble learning
• Train multiple models to try and solve the same problem
• Combine the outputs of them to obtain the final decision
• Bagging [Breiman’ 96], boosting [Freund’ 99] and mixture of experts
[Jacobs’ 91]
Application: Ensemble Learning using Deep Neural Networks
21
Final decision
Majority
voting
Test data
[Freund’ 99] Freund, Yoav, Schapire, Robert, and Abe, N. A short introduction to boosting. Journal-Japanese Society For Arti- ficial Intelligence, 14(771-780):1612, 1999.
[Breiman’ 96] Breiman, Leo. Bagging predictors. Machine learning, 24 (2):123–140, 1996.
[Jacobs’ 91] Jacobs, Robert A, Jordan, Michael I, Nowlan, Steven J, and Hinton, Geoffrey E. Adaptive mixtures of local experts. Neural computation, 1991.
22. Algorithmic Intelligence Lab
• Independent Ensemble (IE) [Ciregan’ 12]
• Independently train each model with random initialization
• IE generally improves the performance by reducing the variance
• Multiple choice learning (MCL) [Guzman’ 12]
• Making each model specialized for certain subset of data
• MCL can produce diverse solutions
• Image classification on CIFAR-10 using 5 CNNs
Ensemble Methods for Deep Neural Networks
22
23. Algorithmic Intelligence Lab
• Multiple choice learning (MCL) [Guzman’ 12]
• Making each model specialized for certain subset of data
• Overconfidence issues of MCL
Ensemble Methods for Deep Neural Networks
23
Cat
Cat
Model 1 (specialized in “Cat” image)
Model 2 (specialized in “Dog” image)Dog
Dog
24. Algorithmic Intelligence Lab
• Multiple choice learning (MCL) [Guzman’ 12]
• Making each model specialized for certain subset of data
• Overconfidence issues of MCL
Ensemble Methods for Deep Neural Networks
24
Cat
Cat
Model 1 (specialized in “Cat” image)
Model 2 (specialized in “Dog” image)Dog
Dog
Cat Overconfident
1%
99%
97%
3%
25. Algorithmic Intelligence Lab
• Multiple choice learning (MCL) [Guzman’ 12]
• Making each model specialized for certain subset of data
• Overconfidence issues of MCL
Ensemble Methods for Deep Neural Networks
25
Cat
Cat
Model 1 (specialized in “Cat” image)
Model 2 (specialized in “Dog” image)Dog
Dog
Cat
1%
99%
97%
3%
DogCat
49% 51%
Averaged probability
Average
Voting
26. Algorithmic Intelligence Lab
• Making the specialized models with confident predictions
• Main components of our contributions
• Experiments on CIFAR-10 using 5 CNNs (2 Conv + 2 FC)
Confident Multiple Choice Learning (CMCL)
26
New loss: confident oracle loss
New architecture: feature sharing
New training method: random labeling
27. Algorithmic Intelligence Lab
• Confident oracle loss
• Generating confident predictions by minimizing the KL divergence
Confident Oracle Loss
27
28. Algorithmic Intelligence Lab
• Confident oracle loss
• Generating confident predictions by minimizing the KL divergence
Confident Oracle Loss
28
Model 1 Model 2 Model 3
Data distribution
29. Algorithmic Intelligence Lab
• Confident oracle loss
• Generating confident predictions by minimizing the KL divergence
Confident Oracle Loss
29
Model 1 Model 2 Model 3
Data distribution Uniform distribution
30. Algorithmic Intelligence Lab
• Classification test set error rates on CIFAR-10 and SVHN
• Top-1 error
• Select the class from averaged probability
• Oracle error
• Measuring whether none of the members predict the correct class
• We use both feature sharing and random labeling for all experiments
Experimental Results: Image Classification
30
• 32 × 32 RGB
• 10 classes
• 50,000 training set
• 10,000 test set
• 32 × 32 RGB
• 10 classes
• 73,257 training set
• 26,032 test set
CIFAR-10 [Krizhevsky’ 09] SVHN [Netzer’ 11]
31. Algorithmic Intelligence Lab
• Ensemble of small-scale CNNs (2 Conv + 2 FC)
Experimental Results: Image Classification
31
K=1 K=2
Model 1
Model 2
Model 3
Model 1
Model 2
Model 3
“Picking K specialized models”
33. Algorithmic Intelligence Lab
• iCoseg dataset
Experimental Results: Image Segmentation
33
Fully convolutional neural networks
(FCNs) [Long’ 15]
Pixel-level classification
problem with 2 classes
1(foreground) and 0 (background)
[Long’ 15] Long, J., Shelhamer, E. and Darrell, T. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
34. Algorithmic Intelligence Lab
• Prediction results of segmentation for few sample images
• MCL and CMCL generate high-quality predictions
• CMCL only outperforms IE in terms of the top-1 error
Experimental Results: Image Segmentation
34
- 6.77%
relative
reduction
35. Algorithmic Intelligence Lab
Outline
• Introduction
• Predictive uncertainty of deep neural networks
• Summary
• How to train confident neural networks
• Training Confidence-Calibrated Classifiers for Detecting Out-of-Distribution
Samples [Lee’ 18a]
• Applications
• Confident Multiple Choice Learning [Lee’ 17]
• Hierarchical novelty detection [Lee’ 18b]
• Conclusion
35
[Lee’ 18a] Lee, K., Lee, H., Lee, K. and Shin, J. Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples. I
n ICLR, 2018.
[Lee’ 17] Lee, K., Hwang, C., Park, K. and Shin, J. Confident Multiple Choice Learning. In ICML, 2017.
[Lee’ 18b] Lee, K., Lee, Min. K, Zhang, Y. Shin. J, Lee, H. Hierarchical Novelty Detection for Visual Object Recognition, In CVPR, 2018.
36. Algorithmic Intelligence Lab
• Objective
• 1. Find the closest known (super-)category in taxonomy
• 2. Find fine-grained classification for novel categories (i.e., out-of-
distribution samples)
Hierarchical Novelty Detection
36
Figure 1. An illustration of our hierarchical novelty detection task
37. Algorithmic Intelligence Lab
• Top-down method (TD)
• p(child) = ∑super p(child | super) p(super)
• Objective
• Inference
• Definition of confidence:
Two Main Approaches
37
Novel class
38. Algorithmic Intelligence Lab
• ImageNet dataset
• 22K classes
• Taxonomy
• 396 super classes of 1K known
leaf classes
• Rest of 21K classes can be used
as novel class
• Example
Experimental Results on ImageNet Dataset
38
[Deng’ 12] J. Deng, J. Krause, A. C. Berg, and L. Fei-Fei. Hedging your bets: Optimizing accuracy-specificity trade offs in large scale visual recognition. In
CVPR , pages 3450–3457. IEEE, 2012.
• Hierarchical novelty detection
performance
• Baseline: DARTS [Deng’ 12]
• One can note that our methods
have higher novel class
accuracy than DARTS to have a
same known class accuracy in
most regions
39. Algorithmic Intelligence Lab
• We propose a new method for training confident deep neural networks
• It produce the uniform distribution when the input is not from target
distribution
• We show that it can be applied to many machine learning problems:
• Detecting out-of-distribution problem
• Ensemble learning using deep neural networks
• Hierarchical novelty detection
• We believe that our new approach brings a refreshing angle for
developing confident deep networks in many related applications:
• Network calibration
• Adversarial example detection
• Bayesian probabilistic models
• Semi-supervised learning
Conclusion
39