SlideShare a Scribd company logo
Safety Verification of Deep
Neural Networks
Alexandre Hua - Lotfi Larbaoui
Bruno Roy - Anne Laurence Thoux
5 avril 2018
[2]
Outline
● Introduction
● Literature Review
● Definitions
● Framework verification
● Experimental results
● Comparison
● Conclusion
Abstract
Research Safety artificial intelligence
Machine learning Deep learning
Architecture Deep neural network
Application Self-driving car
Framework Automated verification
Method
Satisfiability modulo theories
(SMT)
Objective Safety of classification decision
Introduction
● Working with classifiers
● Small perturbations can cause the network to misclassify the image
● Framework for automated verification of safety classification decisions
[2]
1993
Extracting Rules from Artificial
Neural Networks
First method to verify the
specification of a neural network.
Verification and validation of
neural networks for safety-critical
applications
Present an analysis techniques that
can be used for verification of
polynomial neural network (PNN).
2002
2010
An Abstraction-Refinement
Approach to Verification of
Artificial Neural Networks
First paper demonstrating that the
output class is constant across a
desired neighborhood.
2016
Safety Verification of Deep Neural
Networks
Present a novel framework that find
a misclassification if found if it
exists, using SMT.
2017
Reluplex: An Efficient SMT Solver
for Verifying Deep Neural
Networks
Suggest a method to extend SMT
solvers, allowing for the verification
of constraints on deep neural
networks.
Literature Review
Definitions
Definitions
The safety of classification decisions
Intuition :
The safety of classification decisions
Intuition :
The safety of classification decisions
Intuition :
The safety of classification decisions
Intuition :
The safety of classification decisions
Intuition :
Adversarial example
The safety of classification decisions
Formally :
Region
Definition of a manipulation
Minimal manipulation and bounded variation
(1)
(2)
(3)
Safety wrt Manipulations
Framework
verification
Boolean satisfiability problem (SAT)
● SAT: given a formula A(x1, x2,..., xn),
are there any Boolean values xi of xi who make A true?
● VALID: given a formula A(x1, x2, …, xn),
A is true for all Boolean values xi of xi?
● VALID(A)⟷ ¬SAT(¬A)
SAT is a fundamental problem of computer science and mathematics, with
applications everywhere It is the prototype of the NP-complete problem to which
many other problems are reduced
Work with formulas mixing logic and theories .
((a = 1)∨(a = 2))∧(a ≥ 3)∧((b ≤ 3)∨(b ≥ 2))
logic + arithmetic
((f (a) = 1)∨(a - 3 = 2))∧(g(a) ≥ 3)∧((B[0] ≤ 3)∨(B[1] ≥ 2))
logic + arithmetic + functions + tables
Satisfiability : there is a model,i.e., a value of unknowns in the theories that makes
the formula true .
Validity : the formula is true for any model ⟺ his negation is not satisfactory.
Satisfiability modulo theories (SMT)
Uninterpreted function
Example : for x,y,z are integers and f is an integer function the following formula
may be true ?
(x = y )∧(x × (f(y)+f(x)) = t)∧(y× (f(x)+f(x)) ≠ t)
No, because the extensional equality is written :
x=y ⇒ f(x) = f(y )
So
(x = y )∧(x × (f (y )+f (x)) = t)⇒(y × (f(x)+f(x)) = t) and the initial formula is false
[5]
[5]
[5]
[5]
Layer-by-layer analysis
Layer-by-Layer Refinement
Figure : Complete refinement in general safety and safety wrt manipulations
Experimental
Results
Experimental Results
● Experimentations on trained classification neural network
● Using well-known image dataset to feed input to classifier such as
○ MNIST
○ CIFAR-10
○ ImageNet
○ GTSRB
Two-Dimensional Point Classification Network
Input Layer First Hidden Layer
[1]
Image Classification Network for the MNIST
Handwritten Image Dataset
[1]
Image Classification Network for the CIFAR-10
Small Image Dataset
Misclassified
as a truck
[1]
Image Classification Network for the ImageNet
Dataset
Adversarial example
found after 6346
dimensional changes
No adversarial example
found after 20 000
dimensional changes
=> report as safe
[1]
[1]
Image Classification Network for the GTSRB
dataset
[1]
[1]
[1]
Comparison
DLV vs FGSM vs JSMA
• FGSM (Fast Gradient Step Method)
calculates the optimal attack for a linear approximation of the
network cost
• JSMA (Jacobian Saliency Map Algorithm)
finds a set of dimensions in the input layer to manipulate,
according to the linear approximation (by computing the
Jacobian matrix) of the model from current output to a
nominated target output
DLV vs FGSM vs JSMA
FGSM
JSMA
DLV Misclassed
[1]
[1]
[1]
DLV vs FGSM vs JSMA
[1]
Conclusion
● Framework for automated verification of safety (for classification decisions)
● Using the Satisfiability Modulo Theory (SMT)
● Framework that finds a misclassification if it exists
● Framework can be generalized to other tasks
References
● [1] : Xiaowei Huang, Marta Kwiatkowska, Sen Wang and Min Wu, "Safety Verification of Deep Neural Networks"
[Online]. Available: http://qav.comlab.ox.ac.uk/papers/hkww17.pdf, 2016.
● [2] : Uber self-driving system should have spotted woman, experts say (22 march 2018) CBC. [Online]. Available:
http://www.cbc.ca/news/world/uber-self-driving-accident-video-1.4587439
● [3] : How Adversarial Attacks Work. (2017) Emil Mikhailov and Roman Trusov. [Online]. Available:
https://blog.ycombinator.com/how-adversarial-attacks-work/
● [4] https://www.pyimagesearch.com/2017/03/20/imagenet-vggnet-resnet-inception-xception-keras/
● [5] http://www.cleverhans.io/security/privacy/ml/2017/06/14/verification.html

More Related Content

What's hot

how to calclute time complexity of algortihm
how to calclute time complexity of algortihmhow to calclute time complexity of algortihm
how to calclute time complexity of algortihm
Sajid Marwat
 
Algorithm And analysis Lecture 03& 04-time complexity.
 Algorithm And analysis Lecture 03& 04-time complexity. Algorithm And analysis Lecture 03& 04-time complexity.
Algorithm And analysis Lecture 03& 04-time complexity.
Tariq Khan
 
Complexity of Algorithm
Complexity of AlgorithmComplexity of Algorithm
Complexity of Algorithm
Muhammad Muzammal
 
Dynamic Programming - Part II
Dynamic Programming - Part IIDynamic Programming - Part II
Dynamic Programming - Part II
Amrinder Arora
 
Introduction to Algorithms Complexity Analysis
Introduction to Algorithms Complexity Analysis Introduction to Algorithms Complexity Analysis
Introduction to Algorithms Complexity Analysis
Dr. Pankaj Agarwal
 
Predicting organic reaction outcomes with weisfeiler lehman network
Predicting organic reaction outcomes with weisfeiler lehman networkPredicting organic reaction outcomes with weisfeiler lehman network
Predicting organic reaction outcomes with weisfeiler lehman network
Kazuki Fujikawa
 
Deep Learning: R with Keras and TensorFlow
Deep Learning: R with Keras and TensorFlowDeep Learning: R with Keras and TensorFlow
Deep Learning: R with Keras and TensorFlow
Oswald Campesato
 
Deep Learning, Scala, and Spark
Deep Learning, Scala, and SparkDeep Learning, Scala, and Spark
Deep Learning, Scala, and Spark
Oswald Campesato
 
Unit i basic concepts of algorithms
Unit i basic concepts of algorithmsUnit i basic concepts of algorithms
Unit i basic concepts of algorithms
sangeetha s
 
Algorithm analysis
Algorithm analysisAlgorithm analysis
Algorithm analysis
sumitbardhan
 
Lec7
Lec7Lec7
Algorithem complexity in data sructure
Algorithem complexity in data sructureAlgorithem complexity in data sructure
Algorithem complexity in data sructure
Kumar
 
Dsp lab pdf
Dsp lab pdfDsp lab pdf
Dsp lab pdf
shareslidesplease
 
pptx - Psuedo Random Generator for Halfspaces
pptx - Psuedo Random Generator for Halfspacespptx - Psuedo Random Generator for Halfspaces
pptx - Psuedo Random Generator for Halfspaces
butest
 
Static Analysis and Verification of C Programs
Static Analysis and Verification of C ProgramsStatic Analysis and Verification of C Programs
Fundamentals of the Analysis of Algorithm Efficiency
Fundamentals of the Analysis of Algorithm EfficiencyFundamentals of the Analysis of Algorithm Efficiency
Fundamentals of the Analysis of Algorithm Efficiency
Saranya Natarajan
 
Asymptotic Notation and Data Structures
Asymptotic Notation and Data StructuresAsymptotic Notation and Data Structures
Asymptotic Notation and Data Structures
Amrinder Arora
 
Scala and Deep Learning
Scala and Deep LearningScala and Deep Learning
Scala and Deep Learning
Oswald Campesato
 
C++ and Deep Learning
C++ and Deep LearningC++ and Deep Learning
C++ and Deep Learning
Oswald Campesato
 
Gaussian processing
Gaussian processingGaussian processing
Gaussian processing
홍배 김
 

What's hot (20)

how to calclute time complexity of algortihm
how to calclute time complexity of algortihmhow to calclute time complexity of algortihm
how to calclute time complexity of algortihm
 
Algorithm And analysis Lecture 03& 04-time complexity.
 Algorithm And analysis Lecture 03& 04-time complexity. Algorithm And analysis Lecture 03& 04-time complexity.
Algorithm And analysis Lecture 03& 04-time complexity.
 
Complexity of Algorithm
Complexity of AlgorithmComplexity of Algorithm
Complexity of Algorithm
 
Dynamic Programming - Part II
Dynamic Programming - Part IIDynamic Programming - Part II
Dynamic Programming - Part II
 
Introduction to Algorithms Complexity Analysis
Introduction to Algorithms Complexity Analysis Introduction to Algorithms Complexity Analysis
Introduction to Algorithms Complexity Analysis
 
Predicting organic reaction outcomes with weisfeiler lehman network
Predicting organic reaction outcomes with weisfeiler lehman networkPredicting organic reaction outcomes with weisfeiler lehman network
Predicting organic reaction outcomes with weisfeiler lehman network
 
Deep Learning: R with Keras and TensorFlow
Deep Learning: R with Keras and TensorFlowDeep Learning: R with Keras and TensorFlow
Deep Learning: R with Keras and TensorFlow
 
Deep Learning, Scala, and Spark
Deep Learning, Scala, and SparkDeep Learning, Scala, and Spark
Deep Learning, Scala, and Spark
 
Unit i basic concepts of algorithms
Unit i basic concepts of algorithmsUnit i basic concepts of algorithms
Unit i basic concepts of algorithms
 
Algorithm analysis
Algorithm analysisAlgorithm analysis
Algorithm analysis
 
Lec7
Lec7Lec7
Lec7
 
Algorithem complexity in data sructure
Algorithem complexity in data sructureAlgorithem complexity in data sructure
Algorithem complexity in data sructure
 
Dsp lab pdf
Dsp lab pdfDsp lab pdf
Dsp lab pdf
 
pptx - Psuedo Random Generator for Halfspaces
pptx - Psuedo Random Generator for Halfspacespptx - Psuedo Random Generator for Halfspaces
pptx - Psuedo Random Generator for Halfspaces
 
Static Analysis and Verification of C Programs
Static Analysis and Verification of C ProgramsStatic Analysis and Verification of C Programs
Static Analysis and Verification of C Programs
 
Fundamentals of the Analysis of Algorithm Efficiency
Fundamentals of the Analysis of Algorithm EfficiencyFundamentals of the Analysis of Algorithm Efficiency
Fundamentals of the Analysis of Algorithm Efficiency
 
Asymptotic Notation and Data Structures
Asymptotic Notation and Data StructuresAsymptotic Notation and Data Structures
Asymptotic Notation and Data Structures
 
Scala and Deep Learning
Scala and Deep LearningScala and Deep Learning
Scala and Deep Learning
 
C++ and Deep Learning
C++ and Deep LearningC++ and Deep Learning
C++ and Deep Learning
 
Gaussian processing
Gaussian processingGaussian processing
Gaussian processing
 

Similar to Safety Verification of Deep Neural Networks_.pdf

Neural networks
Neural networksNeural networks
Neural networks
HarshitGupta367
 
COMPARISON OF WAVELET NETWORK AND LOGISTIC REGRESSION IN PREDICTING ENTERPRIS...
COMPARISON OF WAVELET NETWORK AND LOGISTIC REGRESSION IN PREDICTING ENTERPRIS...COMPARISON OF WAVELET NETWORK AND LOGISTIC REGRESSION IN PREDICTING ENTERPRIS...
COMPARISON OF WAVELET NETWORK AND LOGISTIC REGRESSION IN PREDICTING ENTERPRIS...
ijcsit
 
A simple framework for contrastive learning of visual representations
A simple framework for contrastive learning of visual representationsA simple framework for contrastive learning of visual representations
A simple framework for contrastive learning of visual representations
Devansh16
 
COMPARATIVE PERFORMANCE ANALYSIS OF RNSC AND MCL ALGORITHMS ON POWER-LAW DIST...
COMPARATIVE PERFORMANCE ANALYSIS OF RNSC AND MCL ALGORITHMS ON POWER-LAW DIST...COMPARATIVE PERFORMANCE ANALYSIS OF RNSC AND MCL ALGORITHMS ON POWER-LAW DIST...
COMPARATIVE PERFORMANCE ANALYSIS OF RNSC AND MCL ALGORITHMS ON POWER-LAW DIST...
acijjournal
 
DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...
DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...
DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...
Artem Lutov
 
Towards neuralprocessingofgeneralpurposeapproximateprograms
Towards neuralprocessingofgeneralpurposeapproximateprogramsTowards neuralprocessingofgeneralpurposeapproximateprograms
Towards neuralprocessingofgeneralpurposeapproximateprograms
Paridha Saxena
 
NeuralProcessingofGeneralPurposeApproximatePrograms
NeuralProcessingofGeneralPurposeApproximateProgramsNeuralProcessingofGeneralPurposeApproximatePrograms
NeuralProcessingofGeneralPurposeApproximatePrograms
Mohid Nabil
 
X-TREPAN : A Multi Class Regression and Adapted Extraction of Comprehensible ...
X-TREPAN : A Multi Class Regression and Adapted Extraction of Comprehensible ...X-TREPAN : A Multi Class Regression and Adapted Extraction of Comprehensible ...
X-TREPAN : A Multi Class Regression and Adapted Extraction of Comprehensible ...
csandit
 
X-TREPAN: A MULTI CLASS REGRESSION AND ADAPTED EXTRACTION OF COMPREHENSIBLE D...
X-TREPAN: A MULTI CLASS REGRESSION AND ADAPTED EXTRACTION OF COMPREHENSIBLE D...X-TREPAN: A MULTI CLASS REGRESSION AND ADAPTED EXTRACTION OF COMPREHENSIBLE D...
X-TREPAN: A MULTI CLASS REGRESSION AND ADAPTED EXTRACTION OF COMPREHENSIBLE D...
cscpconf
 
Keynote at IWLS 2017
Keynote at IWLS 2017Keynote at IWLS 2017
Keynote at IWLS 2017
Manish Pandey
 
Making Robots Learn
Making Robots LearnMaking Robots Learn
Making Robots Learn
inside-BigData.com
 
HW2-1_05.doc
HW2-1_05.docHW2-1_05.doc
HW2-1_05.doc
butest
 
00463517b1e90c1e63000000
00463517b1e90c1e6300000000463517b1e90c1e63000000
00463517b1e90c1e63000000
Ivonne Liu
 
A detailed analysis of the supervised machine Learning Algorithms
A detailed analysis of the supervised machine Learning AlgorithmsA detailed analysis of the supervised machine Learning Algorithms
A detailed analysis of the supervised machine Learning Algorithms
NIET Journal of Engineering & Technology (NIETJET)
 
AN ANN APPROACH FOR NETWORK INTRUSION DETECTION USING ENTROPY BASED FEATURE S...
AN ANN APPROACH FOR NETWORK INTRUSION DETECTION USING ENTROPY BASED FEATURE S...AN ANN APPROACH FOR NETWORK INTRUSION DETECTION USING ENTROPY BASED FEATURE S...
AN ANN APPROACH FOR NETWORK INTRUSION DETECTION USING ENTROPY BASED FEATURE S...
IJNSA Journal
 
An ann approach for network
An ann approach for networkAn ann approach for network
An ann approach for network
IJNSA Journal
 
All projects
All projectsAll projects
All projects
Karishma Jain
 
Log polar coordinates
Log polar coordinatesLog polar coordinates
Log polar coordinates
Oğul Göçmen
 
Backbone search for object detection for applications in intrusion warning sy...
Backbone search for object detection for applications in intrusion warning sy...Backbone search for object detection for applications in intrusion warning sy...
Backbone search for object detection for applications in intrusion warning sy...
IAESIJAI
 
Introduction to Deep Learning and Tensorflow
Introduction to Deep Learning and TensorflowIntroduction to Deep Learning and Tensorflow
Introduction to Deep Learning and Tensorflow
Oswald Campesato
 

Similar to Safety Verification of Deep Neural Networks_.pdf (20)

Neural networks
Neural networksNeural networks
Neural networks
 
COMPARISON OF WAVELET NETWORK AND LOGISTIC REGRESSION IN PREDICTING ENTERPRIS...
COMPARISON OF WAVELET NETWORK AND LOGISTIC REGRESSION IN PREDICTING ENTERPRIS...COMPARISON OF WAVELET NETWORK AND LOGISTIC REGRESSION IN PREDICTING ENTERPRIS...
COMPARISON OF WAVELET NETWORK AND LOGISTIC REGRESSION IN PREDICTING ENTERPRIS...
 
A simple framework for contrastive learning of visual representations
A simple framework for contrastive learning of visual representationsA simple framework for contrastive learning of visual representations
A simple framework for contrastive learning of visual representations
 
COMPARATIVE PERFORMANCE ANALYSIS OF RNSC AND MCL ALGORITHMS ON POWER-LAW DIST...
COMPARATIVE PERFORMANCE ANALYSIS OF RNSC AND MCL ALGORITHMS ON POWER-LAW DIST...COMPARATIVE PERFORMANCE ANALYSIS OF RNSC AND MCL ALGORITHMS ON POWER-LAW DIST...
COMPARATIVE PERFORMANCE ANALYSIS OF RNSC AND MCL ALGORITHMS ON POWER-LAW DIST...
 
DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...
DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...
DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...
 
Towards neuralprocessingofgeneralpurposeapproximateprograms
Towards neuralprocessingofgeneralpurposeapproximateprogramsTowards neuralprocessingofgeneralpurposeapproximateprograms
Towards neuralprocessingofgeneralpurposeapproximateprograms
 
NeuralProcessingofGeneralPurposeApproximatePrograms
NeuralProcessingofGeneralPurposeApproximateProgramsNeuralProcessingofGeneralPurposeApproximatePrograms
NeuralProcessingofGeneralPurposeApproximatePrograms
 
X-TREPAN : A Multi Class Regression and Adapted Extraction of Comprehensible ...
X-TREPAN : A Multi Class Regression and Adapted Extraction of Comprehensible ...X-TREPAN : A Multi Class Regression and Adapted Extraction of Comprehensible ...
X-TREPAN : A Multi Class Regression and Adapted Extraction of Comprehensible ...
 
X-TREPAN: A MULTI CLASS REGRESSION AND ADAPTED EXTRACTION OF COMPREHENSIBLE D...
X-TREPAN: A MULTI CLASS REGRESSION AND ADAPTED EXTRACTION OF COMPREHENSIBLE D...X-TREPAN: A MULTI CLASS REGRESSION AND ADAPTED EXTRACTION OF COMPREHENSIBLE D...
X-TREPAN: A MULTI CLASS REGRESSION AND ADAPTED EXTRACTION OF COMPREHENSIBLE D...
 
Keynote at IWLS 2017
Keynote at IWLS 2017Keynote at IWLS 2017
Keynote at IWLS 2017
 
Making Robots Learn
Making Robots LearnMaking Robots Learn
Making Robots Learn
 
HW2-1_05.doc
HW2-1_05.docHW2-1_05.doc
HW2-1_05.doc
 
00463517b1e90c1e63000000
00463517b1e90c1e6300000000463517b1e90c1e63000000
00463517b1e90c1e63000000
 
A detailed analysis of the supervised machine Learning Algorithms
A detailed analysis of the supervised machine Learning AlgorithmsA detailed analysis of the supervised machine Learning Algorithms
A detailed analysis of the supervised machine Learning Algorithms
 
AN ANN APPROACH FOR NETWORK INTRUSION DETECTION USING ENTROPY BASED FEATURE S...
AN ANN APPROACH FOR NETWORK INTRUSION DETECTION USING ENTROPY BASED FEATURE S...AN ANN APPROACH FOR NETWORK INTRUSION DETECTION USING ENTROPY BASED FEATURE S...
AN ANN APPROACH FOR NETWORK INTRUSION DETECTION USING ENTROPY BASED FEATURE S...
 
An ann approach for network
An ann approach for networkAn ann approach for network
An ann approach for network
 
All projects
All projectsAll projects
All projects
 
Log polar coordinates
Log polar coordinatesLog polar coordinates
Log polar coordinates
 
Backbone search for object detection for applications in intrusion warning sy...
Backbone search for object detection for applications in intrusion warning sy...Backbone search for object detection for applications in intrusion warning sy...
Backbone search for object detection for applications in intrusion warning sy...
 
Introduction to Deep Learning and Tensorflow
Introduction to Deep Learning and TensorflowIntroduction to Deep Learning and Tensorflow
Introduction to Deep Learning and Tensorflow
 

Recently uploaded

A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
kalichargn70th171
 
Quarter 3 SLRP grade 9.. gshajsbhhaheabh
Quarter 3 SLRP grade 9.. gshajsbhhaheabhQuarter 3 SLRP grade 9.. gshajsbhhaheabh
Quarter 3 SLRP grade 9.. gshajsbhhaheabh
aisafed42
 
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptxMigration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
ervikas4
 
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom KittEnhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
Peter Caitens
 
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
kalichargn70th171
 
The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024
Yara Milbes
 
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Julian Hyde
 
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CDKuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
rodomar2
 
Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !
Marcin Chrost
 
Webinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for EmbeddedWebinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for Embedded
ICS
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
Green Software Development
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
brainerhub1
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
ToXSL Technologies
 
E-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet DynamicsE-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet Dynamics
Hornet Dynamics
 
UI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design SystemUI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design System
Peter Muessig
 
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
The Third Creative Media
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
gapen1
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
dakas1
 
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesE-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
Quickdice ERP
 
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
kalichargn70th171
 

Recently uploaded (20)

A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
 
Quarter 3 SLRP grade 9.. gshajsbhhaheabh
Quarter 3 SLRP grade 9.. gshajsbhhaheabhQuarter 3 SLRP grade 9.. gshajsbhhaheabh
Quarter 3 SLRP grade 9.. gshajsbhhaheabh
 
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptxMigration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
 
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom KittEnhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
 
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
 
The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024
 
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
 
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CDKuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
 
Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !
 
Webinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for EmbeddedWebinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for Embedded
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
 
E-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet DynamicsE-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet Dynamics
 
UI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design SystemUI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design System
 
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
 
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesE-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
 
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
 

Safety Verification of Deep Neural Networks_.pdf

  • 1. Safety Verification of Deep Neural Networks Alexandre Hua - Lotfi Larbaoui Bruno Roy - Anne Laurence Thoux 5 avril 2018
  • 2. [2]
  • 3. Outline ● Introduction ● Literature Review ● Definitions ● Framework verification ● Experimental results ● Comparison ● Conclusion
  • 4. Abstract Research Safety artificial intelligence Machine learning Deep learning Architecture Deep neural network Application Self-driving car Framework Automated verification Method Satisfiability modulo theories (SMT) Objective Safety of classification decision
  • 5. Introduction ● Working with classifiers ● Small perturbations can cause the network to misclassify the image ● Framework for automated verification of safety classification decisions [2]
  • 6. 1993 Extracting Rules from Artificial Neural Networks First method to verify the specification of a neural network. Verification and validation of neural networks for safety-critical applications Present an analysis techniques that can be used for verification of polynomial neural network (PNN). 2002 2010 An Abstraction-Refinement Approach to Verification of Artificial Neural Networks First paper demonstrating that the output class is constant across a desired neighborhood. 2016 Safety Verification of Deep Neural Networks Present a novel framework that find a misclassification if found if it exists, using SMT. 2017 Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks Suggest a method to extend SMT solvers, allowing for the verification of constraints on deep neural networks. Literature Review
  • 9. The safety of classification decisions Intuition :
  • 10. The safety of classification decisions Intuition :
  • 11. The safety of classification decisions Intuition :
  • 12. The safety of classification decisions Intuition :
  • 13. The safety of classification decisions Intuition : Adversarial example
  • 14. The safety of classification decisions Formally : Region
  • 15. Definition of a manipulation
  • 16. Minimal manipulation and bounded variation (1) (2) (3)
  • 19. Boolean satisfiability problem (SAT) ● SAT: given a formula A(x1, x2,..., xn), are there any Boolean values xi of xi who make A true? ● VALID: given a formula A(x1, x2, …, xn), A is true for all Boolean values xi of xi? ● VALID(A)⟷ ¬SAT(¬A) SAT is a fundamental problem of computer science and mathematics, with applications everywhere It is the prototype of the NP-complete problem to which many other problems are reduced
  • 20. Work with formulas mixing logic and theories . ((a = 1)∨(a = 2))∧(a ≥ 3)∧((b ≤ 3)∨(b ≥ 2)) logic + arithmetic ((f (a) = 1)∨(a - 3 = 2))∧(g(a) ≥ 3)∧((B[0] ≤ 3)∨(B[1] ≥ 2)) logic + arithmetic + functions + tables Satisfiability : there is a model,i.e., a value of unknowns in the theories that makes the formula true . Validity : the formula is true for any model ⟺ his negation is not satisfactory. Satisfiability modulo theories (SMT)
  • 21. Uninterpreted function Example : for x,y,z are integers and f is an integer function the following formula may be true ? (x = y )∧(x × (f(y)+f(x)) = t)∧(y× (f(x)+f(x)) ≠ t) No, because the extensional equality is written : x=y ⇒ f(x) = f(y ) So (x = y )∧(x × (f (y )+f (x)) = t)⇒(y × (f(x)+f(x)) = t) and the initial formula is false
  • 22. [5]
  • 23. [5]
  • 24. [5]
  • 25. [5]
  • 27. Layer-by-Layer Refinement Figure : Complete refinement in general safety and safety wrt manipulations
  • 28.
  • 30. Experimental Results ● Experimentations on trained classification neural network ● Using well-known image dataset to feed input to classifier such as ○ MNIST ○ CIFAR-10 ○ ImageNet ○ GTSRB
  • 31. Two-Dimensional Point Classification Network Input Layer First Hidden Layer [1]
  • 32. Image Classification Network for the MNIST Handwritten Image Dataset [1]
  • 33. Image Classification Network for the CIFAR-10 Small Image Dataset Misclassified as a truck [1]
  • 34. Image Classification Network for the ImageNet Dataset Adversarial example found after 6346 dimensional changes No adversarial example found after 20 000 dimensional changes => report as safe [1] [1]
  • 35. Image Classification Network for the GTSRB dataset [1] [1] [1]
  • 37. DLV vs FGSM vs JSMA • FGSM (Fast Gradient Step Method) calculates the optimal attack for a linear approximation of the network cost • JSMA (Jacobian Saliency Map Algorithm) finds a set of dimensions in the input layer to manipulate, according to the linear approximation (by computing the Jacobian matrix) of the model from current output to a nominated target output
  • 38. DLV vs FGSM vs JSMA FGSM JSMA DLV Misclassed [1] [1] [1]
  • 39. DLV vs FGSM vs JSMA [1]
  • 40. Conclusion ● Framework for automated verification of safety (for classification decisions) ● Using the Satisfiability Modulo Theory (SMT) ● Framework that finds a misclassification if it exists ● Framework can be generalized to other tasks
  • 41. References ● [1] : Xiaowei Huang, Marta Kwiatkowska, Sen Wang and Min Wu, "Safety Verification of Deep Neural Networks" [Online]. Available: http://qav.comlab.ox.ac.uk/papers/hkww17.pdf, 2016. ● [2] : Uber self-driving system should have spotted woman, experts say (22 march 2018) CBC. [Online]. Available: http://www.cbc.ca/news/world/uber-self-driving-accident-video-1.4587439 ● [3] : How Adversarial Attacks Work. (2017) Emil Mikhailov and Roman Trusov. [Online]. Available: https://blog.ycombinator.com/how-adversarial-attacks-work/ ● [4] https://www.pyimagesearch.com/2017/03/20/imagenet-vggnet-resnet-inception-xception-keras/ ● [5] http://www.cleverhans.io/security/privacy/ml/2017/06/14/verification.html