SlideShare a Scribd company logo
Safety Verification of Deep
Neural Networks
Alexandre Hua - Lotfi Larbaoui
Bruno Roy - Anne Laurence Thoux
5 avril 2018
[2]
Outline
● Introduction
● Literature Review
● Definitions
● Framework verification
● Experimental results
● Comparison
● Conclusion
Abstract
Research Safety artificial intelligence
Machine learning Deep learning
Architecture Deep neural network
Application Self-driving car
Framework Automated verification
Method
Satisfiability modulo theories
(SMT)
Objective Safety of classification decision
Introduction
● Working with classifiers
● Small perturbations can cause the network to misclassify the image
● Framework for automated verification of safety classification decisions
[2]
1993
Extracting Rules from Artificial
Neural Networks
First method to verify the
specification of a neural network.
Verification and validation of
neural networks for safety-critical
applications
Present an analysis techniques that
can be used for verification of
polynomial neural network (PNN).
2002
2010
An Abstraction-Refinement
Approach to Verification of
Artificial Neural Networks
First paper demonstrating that the
output class is constant across a
desired neighborhood.
2016
Safety Verification of Deep Neural
Networks
Present a novel framework that find
a misclassification if found if it
exists, using SMT.
2017
Reluplex: An Efficient SMT Solver
for Verifying Deep Neural
Networks
Suggest a method to extend SMT
solvers, allowing for the verification
of constraints on deep neural
networks.
Literature Review
Definitions
Definitions
The safety of classification decisions
Intuition :
The safety of classification decisions
Intuition :
The safety of classification decisions
Intuition :
The safety of classification decisions
Intuition :
The safety of classification decisions
Intuition :
Adversarial example
The safety of classification decisions
Formally :
Region
Definition of a manipulation
Minimal manipulation and bounded variation
(1)
(2)
(3)
Safety wrt Manipulations
Framework
verification
Boolean satisfiability problem (SAT)
● SAT: given a formula A(x1, x2,..., xn),
are there any Boolean values xi of xi who make A true?
● VALID: given a formula A(x1, x2, …, xn),
A is true for all Boolean values xi of xi?
● VALID(A)⟷ ¬SAT(¬A)
SAT is a fundamental problem of computer science and mathematics, with
applications everywhere It is the prototype of the NP-complete problem to which
many other problems are reduced
Work with formulas mixing logic and theories .
((a = 1)∨(a = 2))∧(a ≥ 3)∧((b ≤ 3)∨(b ≥ 2))
logic + arithmetic
((f (a) = 1)∨(a - 3 = 2))∧(g(a) ≥ 3)∧((B[0] ≤ 3)∨(B[1] ≥ 2))
logic + arithmetic + functions + tables
Satisfiability : there is a model,i.e., a value of unknowns in the theories that makes
the formula true .
Validity : the formula is true for any model ⟺ his negation is not satisfactory.
Satisfiability modulo theories (SMT)
Uninterpreted function
Example : for x,y,z are integers and f is an integer function the following formula
may be true ?
(x = y )∧(x × (f(y)+f(x)) = t)∧(y× (f(x)+f(x)) ≠ t)
No, because the extensional equality is written :
x=y ⇒ f(x) = f(y )
So
(x = y )∧(x × (f (y )+f (x)) = t)⇒(y × (f(x)+f(x)) = t) and the initial formula is false
[5]
[5]
[5]
[5]
Layer-by-layer analysis
Layer-by-Layer Refinement
Figure : Complete refinement in general safety and safety wrt manipulations
Experimental
Results
Experimental Results
● Experimentations on trained classification neural network
● Using well-known image dataset to feed input to classifier such as
○ MNIST
○ CIFAR-10
○ ImageNet
○ GTSRB
Two-Dimensional Point Classification Network
Input Layer First Hidden Layer
[1]
Image Classification Network for the MNIST
Handwritten Image Dataset
[1]
Image Classification Network for the CIFAR-10
Small Image Dataset
Misclassified
as a truck
[1]
Image Classification Network for the ImageNet
Dataset
Adversarial example
found after 6346
dimensional changes
No adversarial example
found after 20 000
dimensional changes
=> report as safe
[1]
[1]
Image Classification Network for the GTSRB
dataset
[1]
[1]
[1]
Comparison
DLV vs FGSM vs JSMA
• FGSM (Fast Gradient Step Method)
calculates the optimal attack for a linear approximation of the
network cost
• JSMA (Jacobian Saliency Map Algorithm)
finds a set of dimensions in the input layer to manipulate,
according to the linear approximation (by computing the
Jacobian matrix) of the model from current output to a
nominated target output
DLV vs FGSM vs JSMA
FGSM
JSMA
DLV Misclassed
[1]
[1]
[1]
DLV vs FGSM vs JSMA
[1]
Conclusion
● Framework for automated verification of safety (for classification decisions)
● Using the Satisfiability Modulo Theory (SMT)
● Framework that finds a misclassification if it exists
● Framework can be generalized to other tasks
References
● [1] : Xiaowei Huang, Marta Kwiatkowska, Sen Wang and Min Wu, "Safety Verification of Deep Neural Networks"
[Online]. Available: http://qav.comlab.ox.ac.uk/papers/hkww17.pdf, 2016.
● [2] : Uber self-driving system should have spotted woman, experts say (22 march 2018) CBC. [Online]. Available:
http://www.cbc.ca/news/world/uber-self-driving-accident-video-1.4587439
● [3] : How Adversarial Attacks Work. (2017) Emil Mikhailov and Roman Trusov. [Online]. Available:
https://blog.ycombinator.com/how-adversarial-attacks-work/
● [4] https://www.pyimagesearch.com/2017/03/20/imagenet-vggnet-resnet-inception-xception-keras/
● [5] http://www.cleverhans.io/security/privacy/ml/2017/06/14/verification.html

More Related Content

What's hot

how to calclute time complexity of algortihm
how to calclute time complexity of algortihmhow to calclute time complexity of algortihm
how to calclute time complexity of algortihmSajid Marwat
 
Algorithm And analysis Lecture 03& 04-time complexity.
 Algorithm And analysis Lecture 03& 04-time complexity. Algorithm And analysis Lecture 03& 04-time complexity.
Algorithm And analysis Lecture 03& 04-time complexity.
Tariq Khan
 
Complexity of Algorithm
Complexity of AlgorithmComplexity of Algorithm
Complexity of Algorithm
Muhammad Muzammal
 
Dynamic Programming - Part II
Dynamic Programming - Part IIDynamic Programming - Part II
Dynamic Programming - Part II
Amrinder Arora
 
Introduction to Algorithms Complexity Analysis
Introduction to Algorithms Complexity Analysis Introduction to Algorithms Complexity Analysis
Introduction to Algorithms Complexity Analysis
Dr. Pankaj Agarwal
 
Predicting organic reaction outcomes with weisfeiler lehman network
Predicting organic reaction outcomes with weisfeiler lehman networkPredicting organic reaction outcomes with weisfeiler lehman network
Predicting organic reaction outcomes with weisfeiler lehman network
Kazuki Fujikawa
 
Deep Learning: R with Keras and TensorFlow
Deep Learning: R with Keras and TensorFlowDeep Learning: R with Keras and TensorFlow
Deep Learning: R with Keras and TensorFlow
Oswald Campesato
 
Deep Learning, Scala, and Spark
Deep Learning, Scala, and SparkDeep Learning, Scala, and Spark
Deep Learning, Scala, and Spark
Oswald Campesato
 
Unit i basic concepts of algorithms
Unit i basic concepts of algorithmsUnit i basic concepts of algorithms
Unit i basic concepts of algorithms
sangeetha s
 
Algorithm analysis
Algorithm analysisAlgorithm analysis
Algorithm analysissumitbardhan
 
Lec7
Lec7Lec7
Algorithem complexity in data sructure
Algorithem complexity in data sructureAlgorithem complexity in data sructure
Algorithem complexity in data sructureKumar
 
pptx - Psuedo Random Generator for Halfspaces
pptx - Psuedo Random Generator for Halfspacespptx - Psuedo Random Generator for Halfspaces
pptx - Psuedo Random Generator for Halfspacesbutest
 
Static Analysis and Verification of C Programs
Static Analysis and Verification of C ProgramsStatic Analysis and Verification of C Programs
Fundamentals of the Analysis of Algorithm Efficiency
Fundamentals of the Analysis of Algorithm EfficiencyFundamentals of the Analysis of Algorithm Efficiency
Fundamentals of the Analysis of Algorithm Efficiency
Saranya Natarajan
 
Asymptotic Notation and Data Structures
Asymptotic Notation and Data StructuresAsymptotic Notation and Data Structures
Asymptotic Notation and Data Structures
Amrinder Arora
 
Scala and Deep Learning
Scala and Deep LearningScala and Deep Learning
Scala and Deep Learning
Oswald Campesato
 
C++ and Deep Learning
C++ and Deep LearningC++ and Deep Learning
C++ and Deep Learning
Oswald Campesato
 
Gaussian processing
Gaussian processingGaussian processing
Gaussian processing
홍배 김
 

What's hot (20)

how to calclute time complexity of algortihm
how to calclute time complexity of algortihmhow to calclute time complexity of algortihm
how to calclute time complexity of algortihm
 
Algorithm And analysis Lecture 03& 04-time complexity.
 Algorithm And analysis Lecture 03& 04-time complexity. Algorithm And analysis Lecture 03& 04-time complexity.
Algorithm And analysis Lecture 03& 04-time complexity.
 
Complexity of Algorithm
Complexity of AlgorithmComplexity of Algorithm
Complexity of Algorithm
 
Dynamic Programming - Part II
Dynamic Programming - Part IIDynamic Programming - Part II
Dynamic Programming - Part II
 
Introduction to Algorithms Complexity Analysis
Introduction to Algorithms Complexity Analysis Introduction to Algorithms Complexity Analysis
Introduction to Algorithms Complexity Analysis
 
Predicting organic reaction outcomes with weisfeiler lehman network
Predicting organic reaction outcomes with weisfeiler lehman networkPredicting organic reaction outcomes with weisfeiler lehman network
Predicting organic reaction outcomes with weisfeiler lehman network
 
Deep Learning: R with Keras and TensorFlow
Deep Learning: R with Keras and TensorFlowDeep Learning: R with Keras and TensorFlow
Deep Learning: R with Keras and TensorFlow
 
Deep Learning, Scala, and Spark
Deep Learning, Scala, and SparkDeep Learning, Scala, and Spark
Deep Learning, Scala, and Spark
 
Unit i basic concepts of algorithms
Unit i basic concepts of algorithmsUnit i basic concepts of algorithms
Unit i basic concepts of algorithms
 
Algorithm analysis
Algorithm analysisAlgorithm analysis
Algorithm analysis
 
Lec7
Lec7Lec7
Lec7
 
Algorithem complexity in data sructure
Algorithem complexity in data sructureAlgorithem complexity in data sructure
Algorithem complexity in data sructure
 
Dsp lab pdf
Dsp lab pdfDsp lab pdf
Dsp lab pdf
 
pptx - Psuedo Random Generator for Halfspaces
pptx - Psuedo Random Generator for Halfspacespptx - Psuedo Random Generator for Halfspaces
pptx - Psuedo Random Generator for Halfspaces
 
Static Analysis and Verification of C Programs
Static Analysis and Verification of C ProgramsStatic Analysis and Verification of C Programs
Static Analysis and Verification of C Programs
 
Fundamentals of the Analysis of Algorithm Efficiency
Fundamentals of the Analysis of Algorithm EfficiencyFundamentals of the Analysis of Algorithm Efficiency
Fundamentals of the Analysis of Algorithm Efficiency
 
Asymptotic Notation and Data Structures
Asymptotic Notation and Data StructuresAsymptotic Notation and Data Structures
Asymptotic Notation and Data Structures
 
Scala and Deep Learning
Scala and Deep LearningScala and Deep Learning
Scala and Deep Learning
 
C++ and Deep Learning
C++ and Deep LearningC++ and Deep Learning
C++ and Deep Learning
 
Gaussian processing
Gaussian processingGaussian processing
Gaussian processing
 

Similar to Safety Verification of Deep Neural Networks_.pdf

Neural networks
Neural networksNeural networks
Neural networks
HarshitGupta367
 
COMPARISON OF WAVELET NETWORK AND LOGISTIC REGRESSION IN PREDICTING ENTERPRIS...
COMPARISON OF WAVELET NETWORK AND LOGISTIC REGRESSION IN PREDICTING ENTERPRIS...COMPARISON OF WAVELET NETWORK AND LOGISTIC REGRESSION IN PREDICTING ENTERPRIS...
COMPARISON OF WAVELET NETWORK AND LOGISTIC REGRESSION IN PREDICTING ENTERPRIS...
ijcsit
 
A simple framework for contrastive learning of visual representations
A simple framework for contrastive learning of visual representationsA simple framework for contrastive learning of visual representations
A simple framework for contrastive learning of visual representations
Devansh16
 
COMPARATIVE PERFORMANCE ANALYSIS OF RNSC AND MCL ALGORITHMS ON POWER-LAW DIST...
COMPARATIVE PERFORMANCE ANALYSIS OF RNSC AND MCL ALGORITHMS ON POWER-LAW DIST...COMPARATIVE PERFORMANCE ANALYSIS OF RNSC AND MCL ALGORITHMS ON POWER-LAW DIST...
COMPARATIVE PERFORMANCE ANALYSIS OF RNSC AND MCL ALGORITHMS ON POWER-LAW DIST...
acijjournal
 
DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...
DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...
DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...
Artem Lutov
 
Towards neuralprocessingofgeneralpurposeapproximateprograms
Towards neuralprocessingofgeneralpurposeapproximateprogramsTowards neuralprocessingofgeneralpurposeapproximateprograms
Towards neuralprocessingofgeneralpurposeapproximateprograms
Paridha Saxena
 
NeuralProcessingofGeneralPurposeApproximatePrograms
NeuralProcessingofGeneralPurposeApproximateProgramsNeuralProcessingofGeneralPurposeApproximatePrograms
NeuralProcessingofGeneralPurposeApproximateProgramsMohid Nabil
 
X-TREPAN: A MULTI CLASS REGRESSION AND ADAPTED EXTRACTION OF COMPREHENSIBLE D...
X-TREPAN: A MULTI CLASS REGRESSION AND ADAPTED EXTRACTION OF COMPREHENSIBLE D...X-TREPAN: A MULTI CLASS REGRESSION AND ADAPTED EXTRACTION OF COMPREHENSIBLE D...
X-TREPAN: A MULTI CLASS REGRESSION AND ADAPTED EXTRACTION OF COMPREHENSIBLE D...
cscpconf
 
X-TREPAN : A Multi Class Regression and Adapted Extraction of Comprehensible ...
X-TREPAN : A Multi Class Regression and Adapted Extraction of Comprehensible ...X-TREPAN : A Multi Class Regression and Adapted Extraction of Comprehensible ...
X-TREPAN : A Multi Class Regression and Adapted Extraction of Comprehensible ...
csandit
 
Keynote at IWLS 2017
Keynote at IWLS 2017Keynote at IWLS 2017
Keynote at IWLS 2017
Manish Pandey
 
Making Robots Learn
Making Robots LearnMaking Robots Learn
Making Robots Learn
inside-BigData.com
 
HW2-1_05.doc
HW2-1_05.docHW2-1_05.doc
HW2-1_05.docbutest
 
00463517b1e90c1e63000000
00463517b1e90c1e6300000000463517b1e90c1e63000000
00463517b1e90c1e63000000Ivonne Liu
 
A detailed analysis of the supervised machine Learning Algorithms
A detailed analysis of the supervised machine Learning AlgorithmsA detailed analysis of the supervised machine Learning Algorithms
A detailed analysis of the supervised machine Learning Algorithms
NIET Journal of Engineering & Technology (NIETJET)
 
AN ANN APPROACH FOR NETWORK INTRUSION DETECTION USING ENTROPY BASED FEATURE S...
AN ANN APPROACH FOR NETWORK INTRUSION DETECTION USING ENTROPY BASED FEATURE S...AN ANN APPROACH FOR NETWORK INTRUSION DETECTION USING ENTROPY BASED FEATURE S...
AN ANN APPROACH FOR NETWORK INTRUSION DETECTION USING ENTROPY BASED FEATURE S...
IJNSA Journal
 
An ann approach for network
An ann approach for networkAn ann approach for network
An ann approach for network
IJNSA Journal
 
All projects
All projectsAll projects
All projects
Karishma Jain
 
Log polar coordinates
Log polar coordinatesLog polar coordinates
Log polar coordinates
Oğul Göçmen
 
Backbone search for object detection for applications in intrusion warning sy...
Backbone search for object detection for applications in intrusion warning sy...Backbone search for object detection for applications in intrusion warning sy...
Backbone search for object detection for applications in intrusion warning sy...
IAESIJAI
 
Introduction to Deep Learning and Tensorflow
Introduction to Deep Learning and TensorflowIntroduction to Deep Learning and Tensorflow
Introduction to Deep Learning and Tensorflow
Oswald Campesato
 

Similar to Safety Verification of Deep Neural Networks_.pdf (20)

Neural networks
Neural networksNeural networks
Neural networks
 
COMPARISON OF WAVELET NETWORK AND LOGISTIC REGRESSION IN PREDICTING ENTERPRIS...
COMPARISON OF WAVELET NETWORK AND LOGISTIC REGRESSION IN PREDICTING ENTERPRIS...COMPARISON OF WAVELET NETWORK AND LOGISTIC REGRESSION IN PREDICTING ENTERPRIS...
COMPARISON OF WAVELET NETWORK AND LOGISTIC REGRESSION IN PREDICTING ENTERPRIS...
 
A simple framework for contrastive learning of visual representations
A simple framework for contrastive learning of visual representationsA simple framework for contrastive learning of visual representations
A simple framework for contrastive learning of visual representations
 
COMPARATIVE PERFORMANCE ANALYSIS OF RNSC AND MCL ALGORITHMS ON POWER-LAW DIST...
COMPARATIVE PERFORMANCE ANALYSIS OF RNSC AND MCL ALGORITHMS ON POWER-LAW DIST...COMPARATIVE PERFORMANCE ANALYSIS OF RNSC AND MCL ALGORITHMS ON POWER-LAW DIST...
COMPARATIVE PERFORMANCE ANALYSIS OF RNSC AND MCL ALGORITHMS ON POWER-LAW DIST...
 
DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...
DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...
DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...
 
Towards neuralprocessingofgeneralpurposeapproximateprograms
Towards neuralprocessingofgeneralpurposeapproximateprogramsTowards neuralprocessingofgeneralpurposeapproximateprograms
Towards neuralprocessingofgeneralpurposeapproximateprograms
 
NeuralProcessingofGeneralPurposeApproximatePrograms
NeuralProcessingofGeneralPurposeApproximateProgramsNeuralProcessingofGeneralPurposeApproximatePrograms
NeuralProcessingofGeneralPurposeApproximatePrograms
 
X-TREPAN: A MULTI CLASS REGRESSION AND ADAPTED EXTRACTION OF COMPREHENSIBLE D...
X-TREPAN: A MULTI CLASS REGRESSION AND ADAPTED EXTRACTION OF COMPREHENSIBLE D...X-TREPAN: A MULTI CLASS REGRESSION AND ADAPTED EXTRACTION OF COMPREHENSIBLE D...
X-TREPAN: A MULTI CLASS REGRESSION AND ADAPTED EXTRACTION OF COMPREHENSIBLE D...
 
X-TREPAN : A Multi Class Regression and Adapted Extraction of Comprehensible ...
X-TREPAN : A Multi Class Regression and Adapted Extraction of Comprehensible ...X-TREPAN : A Multi Class Regression and Adapted Extraction of Comprehensible ...
X-TREPAN : A Multi Class Regression and Adapted Extraction of Comprehensible ...
 
Keynote at IWLS 2017
Keynote at IWLS 2017Keynote at IWLS 2017
Keynote at IWLS 2017
 
Making Robots Learn
Making Robots LearnMaking Robots Learn
Making Robots Learn
 
HW2-1_05.doc
HW2-1_05.docHW2-1_05.doc
HW2-1_05.doc
 
00463517b1e90c1e63000000
00463517b1e90c1e6300000000463517b1e90c1e63000000
00463517b1e90c1e63000000
 
A detailed analysis of the supervised machine Learning Algorithms
A detailed analysis of the supervised machine Learning AlgorithmsA detailed analysis of the supervised machine Learning Algorithms
A detailed analysis of the supervised machine Learning Algorithms
 
AN ANN APPROACH FOR NETWORK INTRUSION DETECTION USING ENTROPY BASED FEATURE S...
AN ANN APPROACH FOR NETWORK INTRUSION DETECTION USING ENTROPY BASED FEATURE S...AN ANN APPROACH FOR NETWORK INTRUSION DETECTION USING ENTROPY BASED FEATURE S...
AN ANN APPROACH FOR NETWORK INTRUSION DETECTION USING ENTROPY BASED FEATURE S...
 
An ann approach for network
An ann approach for networkAn ann approach for network
An ann approach for network
 
All projects
All projectsAll projects
All projects
 
Log polar coordinates
Log polar coordinatesLog polar coordinates
Log polar coordinates
 
Backbone search for object detection for applications in intrusion warning sy...
Backbone search for object detection for applications in intrusion warning sy...Backbone search for object detection for applications in intrusion warning sy...
Backbone search for object detection for applications in intrusion warning sy...
 
Introduction to Deep Learning and Tensorflow
Introduction to Deep Learning and TensorflowIntroduction to Deep Learning and Tensorflow
Introduction to Deep Learning and Tensorflow
 

Recently uploaded

TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
Tier1 app
 
Designing for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web ServicesDesigning for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web Services
KrzysztofKkol1
 
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Globus
 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2
 
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus
 
Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024
Globus
 
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Globus
 
2024 RoOUG Security model for the cloud.pptx
2024 RoOUG Security model for the cloud.pptx2024 RoOUG Security model for the cloud.pptx
2024 RoOUG Security model for the cloud.pptx
Georgi Kodinov
 
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
Juraj Vysvader
 
De mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FMEDe mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FME
Jelle | Nordend
 
Visitor Management System in India- Vizman.app
Visitor Management System in India- Vizman.appVisitor Management System in India- Vizman.app
Visitor Management System in India- Vizman.app
NaapbooksPrivateLimi
 
Into the Box 2024 - Keynote Day 2 Slides.pdf
Into the Box 2024 - Keynote Day 2 Slides.pdfInto the Box 2024 - Keynote Day 2 Slides.pdf
Into the Box 2024 - Keynote Day 2 Slides.pdf
Ortus Solutions, Corp
 
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Hivelance Technology
 
Quarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden ExtensionsQuarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden Extensions
Max Andersen
 
GlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote sessionGlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote session
Globus
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good Practices
Globus
 
Corporate Management | Session 3 of 3 | Tendenci AMS
Corporate Management | Session 3 of 3 | Tendenci AMSCorporate Management | Session 3 of 3 | Tendenci AMS
Corporate Management | Session 3 of 3 | Tendenci AMS
Tendenci - The Open Source AMS (Association Management Software)
 
Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024
Globus
 
Prosigns: Transforming Business with Tailored Technology Solutions
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns: Transforming Business with Tailored Technology Solutions
Prosigns: Transforming Business with Tailored Technology Solutions
Prosigns
 
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamOpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
takuyayamamoto1800
 

Recently uploaded (20)

TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
 
Designing for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web ServicesDesigning for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web Services
 
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...
 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
 
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024
 
Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024
 
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
 
2024 RoOUG Security model for the cloud.pptx
2024 RoOUG Security model for the cloud.pptx2024 RoOUG Security model for the cloud.pptx
2024 RoOUG Security model for the cloud.pptx
 
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
 
De mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FMEDe mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FME
 
Visitor Management System in India- Vizman.app
Visitor Management System in India- Vizman.appVisitor Management System in India- Vizman.app
Visitor Management System in India- Vizman.app
 
Into the Box 2024 - Keynote Day 2 Slides.pdf
Into the Box 2024 - Keynote Day 2 Slides.pdfInto the Box 2024 - Keynote Day 2 Slides.pdf
Into the Box 2024 - Keynote Day 2 Slides.pdf
 
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
 
Quarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden ExtensionsQuarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden Extensions
 
GlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote sessionGlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote session
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good Practices
 
Corporate Management | Session 3 of 3 | Tendenci AMS
Corporate Management | Session 3 of 3 | Tendenci AMSCorporate Management | Session 3 of 3 | Tendenci AMS
Corporate Management | Session 3 of 3 | Tendenci AMS
 
Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024
 
Prosigns: Transforming Business with Tailored Technology Solutions
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns: Transforming Business with Tailored Technology Solutions
Prosigns: Transforming Business with Tailored Technology Solutions
 
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamOpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
 

Safety Verification of Deep Neural Networks_.pdf

  • 1. Safety Verification of Deep Neural Networks Alexandre Hua - Lotfi Larbaoui Bruno Roy - Anne Laurence Thoux 5 avril 2018
  • 2. [2]
  • 3. Outline ● Introduction ● Literature Review ● Definitions ● Framework verification ● Experimental results ● Comparison ● Conclusion
  • 4. Abstract Research Safety artificial intelligence Machine learning Deep learning Architecture Deep neural network Application Self-driving car Framework Automated verification Method Satisfiability modulo theories (SMT) Objective Safety of classification decision
  • 5. Introduction ● Working with classifiers ● Small perturbations can cause the network to misclassify the image ● Framework for automated verification of safety classification decisions [2]
  • 6. 1993 Extracting Rules from Artificial Neural Networks First method to verify the specification of a neural network. Verification and validation of neural networks for safety-critical applications Present an analysis techniques that can be used for verification of polynomial neural network (PNN). 2002 2010 An Abstraction-Refinement Approach to Verification of Artificial Neural Networks First paper demonstrating that the output class is constant across a desired neighborhood. 2016 Safety Verification of Deep Neural Networks Present a novel framework that find a misclassification if found if it exists, using SMT. 2017 Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks Suggest a method to extend SMT solvers, allowing for the verification of constraints on deep neural networks. Literature Review
  • 9. The safety of classification decisions Intuition :
  • 10. The safety of classification decisions Intuition :
  • 11. The safety of classification decisions Intuition :
  • 12. The safety of classification decisions Intuition :
  • 13. The safety of classification decisions Intuition : Adversarial example
  • 14. The safety of classification decisions Formally : Region
  • 15. Definition of a manipulation
  • 16. Minimal manipulation and bounded variation (1) (2) (3)
  • 19. Boolean satisfiability problem (SAT) ● SAT: given a formula A(x1, x2,..., xn), are there any Boolean values xi of xi who make A true? ● VALID: given a formula A(x1, x2, …, xn), A is true for all Boolean values xi of xi? ● VALID(A)⟷ ¬SAT(¬A) SAT is a fundamental problem of computer science and mathematics, with applications everywhere It is the prototype of the NP-complete problem to which many other problems are reduced
  • 20. Work with formulas mixing logic and theories . ((a = 1)∨(a = 2))∧(a ≥ 3)∧((b ≤ 3)∨(b ≥ 2)) logic + arithmetic ((f (a) = 1)∨(a - 3 = 2))∧(g(a) ≥ 3)∧((B[0] ≤ 3)∨(B[1] ≥ 2)) logic + arithmetic + functions + tables Satisfiability : there is a model,i.e., a value of unknowns in the theories that makes the formula true . Validity : the formula is true for any model ⟺ his negation is not satisfactory. Satisfiability modulo theories (SMT)
  • 21. Uninterpreted function Example : for x,y,z are integers and f is an integer function the following formula may be true ? (x = y )∧(x × (f(y)+f(x)) = t)∧(y× (f(x)+f(x)) ≠ t) No, because the extensional equality is written : x=y ⇒ f(x) = f(y ) So (x = y )∧(x × (f (y )+f (x)) = t)⇒(y × (f(x)+f(x)) = t) and the initial formula is false
  • 22. [5]
  • 23. [5]
  • 24. [5]
  • 25. [5]
  • 27. Layer-by-Layer Refinement Figure : Complete refinement in general safety and safety wrt manipulations
  • 28.
  • 30. Experimental Results ● Experimentations on trained classification neural network ● Using well-known image dataset to feed input to classifier such as ○ MNIST ○ CIFAR-10 ○ ImageNet ○ GTSRB
  • 31. Two-Dimensional Point Classification Network Input Layer First Hidden Layer [1]
  • 32. Image Classification Network for the MNIST Handwritten Image Dataset [1]
  • 33. Image Classification Network for the CIFAR-10 Small Image Dataset Misclassified as a truck [1]
  • 34. Image Classification Network for the ImageNet Dataset Adversarial example found after 6346 dimensional changes No adversarial example found after 20 000 dimensional changes => report as safe [1] [1]
  • 35. Image Classification Network for the GTSRB dataset [1] [1] [1]
  • 37. DLV vs FGSM vs JSMA • FGSM (Fast Gradient Step Method) calculates the optimal attack for a linear approximation of the network cost • JSMA (Jacobian Saliency Map Algorithm) finds a set of dimensions in the input layer to manipulate, according to the linear approximation (by computing the Jacobian matrix) of the model from current output to a nominated target output
  • 38. DLV vs FGSM vs JSMA FGSM JSMA DLV Misclassed [1] [1] [1]
  • 39. DLV vs FGSM vs JSMA [1]
  • 40. Conclusion ● Framework for automated verification of safety (for classification decisions) ● Using the Satisfiability Modulo Theory (SMT) ● Framework that finds a misclassification if it exists ● Framework can be generalized to other tasks
  • 41. References ● [1] : Xiaowei Huang, Marta Kwiatkowska, Sen Wang and Min Wu, "Safety Verification of Deep Neural Networks" [Online]. Available: http://qav.comlab.ox.ac.uk/papers/hkww17.pdf, 2016. ● [2] : Uber self-driving system should have spotted woman, experts say (22 march 2018) CBC. [Online]. Available: http://www.cbc.ca/news/world/uber-self-driving-accident-video-1.4587439 ● [3] : How Adversarial Attacks Work. (2017) Emil Mikhailov and Roman Trusov. [Online]. Available: https://blog.ycombinator.com/how-adversarial-attacks-work/ ● [4] https://www.pyimagesearch.com/2017/03/20/imagenet-vggnet-resnet-inception-xception-keras/ ● [5] http://www.cleverhans.io/security/privacy/ml/2017/06/14/verification.html