SlideShare a Scribd company logo
Magic Tricks for
Self-driving Cars
Weilin Xu, Zhenyu Zhong, Yunhan Jia
• This is a proof-of-concept.
• We are NOT targeting at any autonomous vehicle vendors.
• Don’t try to fool your neighbor’s car.
AUTHORS’ WARNING
Autonomous Vehicle Framework
Radar
LiDAR
Camera
Image: https://medium.com/toyota-ai-ventures/https-medium-com-toyota-ai-ventures-announcingblackmore-7947eacc9e9e
Camera
Camera-based Obstacle Detection
Our Target: YOLOv3
YOLO v3
Object Detection Model
[147 Layers, 62M Parameters]
Input
[416x416x3]
Output
[3549 Bounding Boxes]
Image: http://media.nj.com/traffic_impact/photo/all-way-stop-sign-that-flashes-in-montclairjpg-30576ab330660eff.jpg
Trained with the COCO Dataset
• Common Objects in Context
• 80 Classes: person, [car, truck, bus], [bicycle, motorcycle], [stop sign,
traffic light], etc.
Source: http://cocodataset.org/
car
0.01%
YOLOv3 Inference 116x9
0
156x
198
373x326Anchor
Boxes
𝑏 𝑥 = 𝜎 𝑡 𝑥 + 𝑐 𝑥
𝑏 𝑦 = 𝜎 𝑡 𝑦 + 𝑐 𝑦
𝑏 𝑤 = 𝑝 𝑤 𝑒 𝑡 𝑤
Center
Point
Object
Size 𝑏ℎ = 𝑝ℎ 𝑒 𝑡ℎ
𝑝ℎ
𝑝 𝑤
13 x 13 Grid
(𝑐 𝑥, 𝑐 𝑦) = (11,2)
Prediction
Vector
Bounding Box 80 Class ConfidenceObjectness
𝑡 𝑥 𝑡 𝑦 𝑡 𝑤 𝑡ℎ 𝑝 𝑜𝑏𝑗 𝑐1 𝑐2 … … 𝑐79 𝑐80
■
stop sign
99%
×
car
0.01%
car
0.01%
Threat Model: Image Patch Attack
Company
Logo
Threat Model: Image Patch Attack
Company
Logo
Threat Model: Image Patch Attack
Company
Logo
Threat Model: Image Patch Attack
Company
Logo
Attack Algorithms
• Input Construction
• Objectives
• Optimization
Resize
Perspective
Transform
Differentiable Input Construction
Company
Logo
Company
Logo
Objectives
• Object Production
• Object Vanish
• Object Transformation
Object Production - Coarse
We want more certain objects on the whole image.
• Easy to implement.
• May be difficult to optimize. Company
Logo
Object Production - Precise
We want a certain object of a specific size in a specific location.
Company
Logo
Object Vanish - Coarse
We want a certain object class to vanish on the whole image.
Object Transformation - Coarse
We want certain object class to transform to other class.
Optimization
• Change of variable
Convert to tanh() space to encode the [0, 1] interval constraint.
Friendly to many off-the-shelf optimizers, e.g. Adam.
• To optimize logits
Skip sigmoid() to avoid vanishing gradient.
Carlini, Nicholas, and David Wagner. “Towards Evaluating the Robustness of Neural Networks.”
IEEE S&P (Oakland) 2016.
But, Image Sensing is not an Identity Function
Model Input
[416x416x3]
▪ Limited Resolution
▪ Distortions
▪ Random Noise
▪ …
Digital
Image
Scene
Towards Robust Physical Adversarial Examples
• [Limited Resolution] Smoother patch with the total variation regularization.
• [Distortions] Color management with the non-printability loss.
• [Inaccurate Patch] Random transformation during optimization iterations.
• ……
Sharif, Mahmood, et al. "Accessorize to a crime: Real and stealthy attacks on state-of-the-art face
recognition." ACM CCS 2016.
Conclusion
• Magicians can fool object detection models, so
can attackers.
• We should be cautious with self-driving cars that
rely on computer vision.

More Related Content

Similar to THE VANISHING TRICK FOR SELF-DRIVING CARS - Weilin Xu - DEF CON 26 CAAD VILLAGE

Dynamic Wounds on Animated Characters in UE4
Dynamic Wounds on Animated Characters in UE4Dynamic Wounds on Animated Characters in UE4
Dynamic Wounds on Animated Characters in UE4
Michał Kłoś
 
Tackling Open Images Challenge (2019)
Tackling Open Images Challenge (2019)Tackling Open Images Challenge (2019)
Tackling Open Images Challenge (2019)
Hiroto Honda
 
Training Drone Image Models with Grand Theft Auto
Training Drone Image Models with Grand Theft AutoTraining Drone Image Models with Grand Theft Auto
Training Drone Image Models with Grand Theft Auto
Commonwealth Computer Research, Inc.
 
How we optimized our Game - Jake & Tess' Finding Monsters Adventure
How we optimized our Game - Jake & Tess' Finding Monsters AdventureHow we optimized our Game - Jake & Tess' Finding Monsters Adventure
How we optimized our Game - Jake & Tess' Finding Monsters Adventure
Felipe Lira
 
Surviving Black Friday - A resilience engineering tale - Omri Fima - Codemoti...
Surviving Black Friday - A resilience engineering tale - Omri Fima - Codemoti...Surviving Black Friday - A resilience engineering tale - Omri Fima - Codemoti...
Surviving Black Friday - A resilience engineering tale - Omri Fima - Codemoti...
Codemotion
 
Getting Intimate with Images on Android with James Halpern
Getting Intimate with Images on Android with James HalpernGetting Intimate with Images on Android with James Halpern
Getting Intimate with Images on Android with James Halpern
FITC
 
An Introduction to Face Detection
An Introduction to Face DetectionAn Introduction to Face Detection
An Introduction to Face Detection
Livares Technologies Pvt Ltd
 
Machine learning for IoT - unpacking the blackbox
Machine learning for IoT - unpacking the blackboxMachine learning for IoT - unpacking the blackbox
Machine learning for IoT - unpacking the blackbox
Ivo Andreev
 
Computer vision-nit-silchar-hackathon
Computer vision-nit-silchar-hackathonComputer vision-nit-silchar-hackathon
Computer vision-nit-silchar-hackathon
Aditya Bhattacharya
 
PG Day Us: Animations for Web & Hybrid
PG Day Us: Animations for Web & HybridPG Day Us: Animations for Web & Hybrid
PG Day Us: Animations for Web & Hybrid
Alex Blom
 
Nakal think test_2015
Nakal think test_2015Nakal think test_2015
Nakal think test_2015
Rajdeep Varma
 
SolidWorks Design Automation Using the SolidWorks API, Microsoft Excel and VBA
SolidWorks Design Automation Using the SolidWorks API, Microsoft Excel and VBASolidWorks Design Automation Using the SolidWorks API, Microsoft Excel and VBA
SolidWorks Design Automation Using the SolidWorks API, Microsoft Excel and VBA
Razorleaf Corporation
 
Integrating multiple CDNs at Etsy
Integrating multiple CDNs at EtsyIntegrating multiple CDNs at Etsy
Integrating multiple CDNs at Etsy
Laurie Denness
 
Seeing is Not Believing: Camouflage Attacks on Image Scaling Algorithms Review
Seeing is Not Believing: Camouflage Attacks on Image Scaling Algorithms ReviewSeeing is Not Believing: Camouflage Attacks on Image Scaling Algorithms Review
Seeing is Not Believing: Camouflage Attacks on Image Scaling Algorithms Review
Junyaup Kim
 
Dynamic Texture Atlas by Epul
Dynamic Texture Atlas by EpulDynamic Texture Atlas by Epul
Dynamic Texture Atlas by Epul
Agate Studio
 
Developing Next-Generation Games with Stage3D (Molehill)
Developing Next-Generation Games with Stage3D (Molehill) Developing Next-Generation Games with Stage3D (Molehill)
Developing Next-Generation Games with Stage3D (Molehill)
Jean-Philippe Doiron
 
Intro to auto_desk_maya2015
Intro to auto_desk_maya2015Intro to auto_desk_maya2015
Intro to auto_desk_maya2015
Naveed Ahmed Hulio
 
Computer vision-must-nit-silchar-ml-hackathon-2019
Computer vision-must-nit-silchar-ml-hackathon-2019Computer vision-must-nit-silchar-ml-hackathon-2019
Computer vision-must-nit-silchar-ml-hackathon-2019
Aditya Bhattacharya
 
Smooth Animations for Web & Hybrid
Smooth Animations for Web & HybridSmooth Animations for Web & Hybrid
Smooth Animations for Web & Hybrid
FITC
 
Threat Modeling Lessons from Star Wars
Threat Modeling Lessons from Star WarsThreat Modeling Lessons from Star Wars
Threat Modeling Lessons from Star Wars
Adam Shostack
 

Similar to THE VANISHING TRICK FOR SELF-DRIVING CARS - Weilin Xu - DEF CON 26 CAAD VILLAGE (20)

Dynamic Wounds on Animated Characters in UE4
Dynamic Wounds on Animated Characters in UE4Dynamic Wounds on Animated Characters in UE4
Dynamic Wounds on Animated Characters in UE4
 
Tackling Open Images Challenge (2019)
Tackling Open Images Challenge (2019)Tackling Open Images Challenge (2019)
Tackling Open Images Challenge (2019)
 
Training Drone Image Models with Grand Theft Auto
Training Drone Image Models with Grand Theft AutoTraining Drone Image Models with Grand Theft Auto
Training Drone Image Models with Grand Theft Auto
 
How we optimized our Game - Jake & Tess' Finding Monsters Adventure
How we optimized our Game - Jake & Tess' Finding Monsters AdventureHow we optimized our Game - Jake & Tess' Finding Monsters Adventure
How we optimized our Game - Jake & Tess' Finding Monsters Adventure
 
Surviving Black Friday - A resilience engineering tale - Omri Fima - Codemoti...
Surviving Black Friday - A resilience engineering tale - Omri Fima - Codemoti...Surviving Black Friday - A resilience engineering tale - Omri Fima - Codemoti...
Surviving Black Friday - A resilience engineering tale - Omri Fima - Codemoti...
 
Getting Intimate with Images on Android with James Halpern
Getting Intimate with Images on Android with James HalpernGetting Intimate with Images on Android with James Halpern
Getting Intimate with Images on Android with James Halpern
 
An Introduction to Face Detection
An Introduction to Face DetectionAn Introduction to Face Detection
An Introduction to Face Detection
 
Machine learning for IoT - unpacking the blackbox
Machine learning for IoT - unpacking the blackboxMachine learning for IoT - unpacking the blackbox
Machine learning for IoT - unpacking the blackbox
 
Computer vision-nit-silchar-hackathon
Computer vision-nit-silchar-hackathonComputer vision-nit-silchar-hackathon
Computer vision-nit-silchar-hackathon
 
PG Day Us: Animations for Web & Hybrid
PG Day Us: Animations for Web & HybridPG Day Us: Animations for Web & Hybrid
PG Day Us: Animations for Web & Hybrid
 
Nakal think test_2015
Nakal think test_2015Nakal think test_2015
Nakal think test_2015
 
SolidWorks Design Automation Using the SolidWorks API, Microsoft Excel and VBA
SolidWorks Design Automation Using the SolidWorks API, Microsoft Excel and VBASolidWorks Design Automation Using the SolidWorks API, Microsoft Excel and VBA
SolidWorks Design Automation Using the SolidWorks API, Microsoft Excel and VBA
 
Integrating multiple CDNs at Etsy
Integrating multiple CDNs at EtsyIntegrating multiple CDNs at Etsy
Integrating multiple CDNs at Etsy
 
Seeing is Not Believing: Camouflage Attacks on Image Scaling Algorithms Review
Seeing is Not Believing: Camouflage Attacks on Image Scaling Algorithms ReviewSeeing is Not Believing: Camouflage Attacks on Image Scaling Algorithms Review
Seeing is Not Believing: Camouflage Attacks on Image Scaling Algorithms Review
 
Dynamic Texture Atlas by Epul
Dynamic Texture Atlas by EpulDynamic Texture Atlas by Epul
Dynamic Texture Atlas by Epul
 
Developing Next-Generation Games with Stage3D (Molehill)
Developing Next-Generation Games with Stage3D (Molehill) Developing Next-Generation Games with Stage3D (Molehill)
Developing Next-Generation Games with Stage3D (Molehill)
 
Intro to auto_desk_maya2015
Intro to auto_desk_maya2015Intro to auto_desk_maya2015
Intro to auto_desk_maya2015
 
Computer vision-must-nit-silchar-ml-hackathon-2019
Computer vision-must-nit-silchar-ml-hackathon-2019Computer vision-must-nit-silchar-ml-hackathon-2019
Computer vision-must-nit-silchar-ml-hackathon-2019
 
Smooth Animations for Web & Hybrid
Smooth Animations for Web & HybridSmooth Animations for Web & Hybrid
Smooth Animations for Web & Hybrid
 
Threat Modeling Lessons from Star Wars
Threat Modeling Lessons from Star WarsThreat Modeling Lessons from Star Wars
Threat Modeling Lessons from Star Wars
 

More from GeekPwn Keen

PRACTICAL ADVERSARIAL ATTACKS AGAINST CHALLENGING MODELS ENVIRONMENTS - Moust...
PRACTICAL ADVERSARIAL ATTACKS AGAINST CHALLENGING MODELS ENVIRONMENTS - Moust...PRACTICAL ADVERSARIAL ATTACKS AGAINST CHALLENGING MODELS ENVIRONMENTS - Moust...
PRACTICAL ADVERSARIAL ATTACKS AGAINST CHALLENGING MODELS ENVIRONMENTS - Moust...
GeekPwn Keen
 
TARGETED ADVERSARIAL EXAMPLES FOR BLACK BOX AUDIO SYSTEMS - Rohan Taori, Amog...
TARGETED ADVERSARIAL EXAMPLES FOR BLACK BOX AUDIO SYSTEMS - Rohan Taori, Amog...TARGETED ADVERSARIAL EXAMPLES FOR BLACK BOX AUDIO SYSTEMS - Rohan Taori, Amog...
TARGETED ADVERSARIAL EXAMPLES FOR BLACK BOX AUDIO SYSTEMS - Rohan Taori, Amog...
GeekPwn Keen
 
TRANSFERABLE ADVERSARIAL PERTURBATIONS - Tencent Blade Team - DEF CON 26 CAA...
TRANSFERABLE ADVERSARIAL PERTURBATIONS - Tencent Blade Team -  DEF CON 26 CAA...TRANSFERABLE ADVERSARIAL PERTURBATIONS - Tencent Blade Team -  DEF CON 26 CAA...
TRANSFERABLE ADVERSARIAL PERTURBATIONS - Tencent Blade Team - DEF CON 26 CAA...
GeekPwn Keen
 
ADVERSARIAL^2 TRAINING - Yao Zhao, Yuzhe Zhao - DEF CON 26 CAAD VILLAGE
ADVERSARIAL^2 TRAINING - Yao Zhao, Yuzhe Zhao - DEF CON 26 CAAD VILLAGEADVERSARIAL^2 TRAINING - Yao Zhao, Yuzhe Zhao - DEF CON 26 CAAD VILLAGE
ADVERSARIAL^2 TRAINING - Yao Zhao, Yuzhe Zhao - DEF CON 26 CAAD VILLAGE
GeekPwn Keen
 
HOW TO LEVERAGE THE OPEN-SOURCE INFORMATION TO MAKE AN EFFECTIVE ADVERSARIAL ...
HOW TO LEVERAGE THE OPEN-SOURCE INFORMATION TO MAKE AN EFFECTIVE ADVERSARIAL ...HOW TO LEVERAGE THE OPEN-SOURCE INFORMATION TO MAKE AN EFFECTIVE ADVERSARIAL ...
HOW TO LEVERAGE THE OPEN-SOURCE INFORMATION TO MAKE AN EFFECTIVE ADVERSARIAL ...
GeekPwn Keen
 
BOOSTING ADVERSARIAL ATTACKS WITH MOMENTUM - Tianyu Pang and Chao Du, THU - D...
BOOSTING ADVERSARIAL ATTACKS WITH MOMENTUM - Tianyu Pang and Chao Du, THU - D...BOOSTING ADVERSARIAL ATTACKS WITH MOMENTUM - Tianyu Pang and Chao Du, THU - D...
BOOSTING ADVERSARIAL ATTACKS WITH MOMENTUM - Tianyu Pang and Chao Du, THU - D...
GeekPwn Keen
 
RECENT PROGRESS IN ADVERSARIAL DEEP LEARNING ATTACK AND DEFENSE - Wenbo Guo a...
RECENT PROGRESS IN ADVERSARIAL DEEP LEARNING ATTACK AND DEFENSE - Wenbo Guo a...RECENT PROGRESS IN ADVERSARIAL DEEP LEARNING ATTACK AND DEFENSE - Wenbo Guo a...
RECENT PROGRESS IN ADVERSARIAL DEEP LEARNING ATTACK AND DEFENSE - Wenbo Guo a...
GeekPwn Keen
 
WEAPONS FOR DOG FIGHT:ADAPTING MALWARE TO ANTI-DETECTION BASED ON GAN - Zhuan...
WEAPONS FOR DOG FIGHT:ADAPTING MALWARE TO ANTI-DETECTION BASED ON GAN - Zhuan...WEAPONS FOR DOG FIGHT:ADAPTING MALWARE TO ANTI-DETECTION BASED ON GAN - Zhuan...
WEAPONS FOR DOG FIGHT:ADAPTING MALWARE TO ANTI-DETECTION BASED ON GAN - Zhuan...
GeekPwn Keen
 
Hardware Trojan Attacks on Neural Networks - Joseph Clements - DEF CON 26 CAA...
Hardware Trojan Attacks on Neural Networks - Joseph Clements - DEF CON 26 CAA...Hardware Trojan Attacks on Neural Networks - Joseph Clements - DEF CON 26 CAA...
Hardware Trojan Attacks on Neural Networks - Joseph Clements - DEF CON 26 CAA...
GeekPwn Keen
 
Huiming Liu-'resident evil' of smart phones--wombie attack
Huiming Liu-'resident evil' of smart phones--wombie attackHuiming Liu-'resident evil' of smart phones--wombie attack
Huiming Liu-'resident evil' of smart phones--wombie attack
GeekPwn Keen
 
Zhiyun Qian-what leaves attacker hijacking USA Today site
Zhiyun Qian-what leaves attacker hijacking USA Today siteZhiyun Qian-what leaves attacker hijacking USA Today site
Zhiyun Qian-what leaves attacker hijacking USA Today site
GeekPwn Keen
 
Bo Li-they’ve created images that reliably fool neural network
Bo Li-they’ve created images that reliably fool neural networkBo Li-they’ve created images that reliably fool neural network
Bo Li-they’ve created images that reliably fool neural network
GeekPwn Keen
 
Alexey kurakin-what's new in adversarial machine learning
Alexey kurakin-what's new in adversarial machine learningAlexey kurakin-what's new in adversarial machine learning
Alexey kurakin-what's new in adversarial machine learning
GeekPwn Keen
 
Nick Stephens-how does someone unlock your phone with nose
Nick Stephens-how does someone unlock your phone with noseNick Stephens-how does someone unlock your phone with nose
Nick Stephens-how does someone unlock your phone with nose
GeekPwn Keen
 

More from GeekPwn Keen (14)

PRACTICAL ADVERSARIAL ATTACKS AGAINST CHALLENGING MODELS ENVIRONMENTS - Moust...
PRACTICAL ADVERSARIAL ATTACKS AGAINST CHALLENGING MODELS ENVIRONMENTS - Moust...PRACTICAL ADVERSARIAL ATTACKS AGAINST CHALLENGING MODELS ENVIRONMENTS - Moust...
PRACTICAL ADVERSARIAL ATTACKS AGAINST CHALLENGING MODELS ENVIRONMENTS - Moust...
 
TARGETED ADVERSARIAL EXAMPLES FOR BLACK BOX AUDIO SYSTEMS - Rohan Taori, Amog...
TARGETED ADVERSARIAL EXAMPLES FOR BLACK BOX AUDIO SYSTEMS - Rohan Taori, Amog...TARGETED ADVERSARIAL EXAMPLES FOR BLACK BOX AUDIO SYSTEMS - Rohan Taori, Amog...
TARGETED ADVERSARIAL EXAMPLES FOR BLACK BOX AUDIO SYSTEMS - Rohan Taori, Amog...
 
TRANSFERABLE ADVERSARIAL PERTURBATIONS - Tencent Blade Team - DEF CON 26 CAA...
TRANSFERABLE ADVERSARIAL PERTURBATIONS - Tencent Blade Team -  DEF CON 26 CAA...TRANSFERABLE ADVERSARIAL PERTURBATIONS - Tencent Blade Team -  DEF CON 26 CAA...
TRANSFERABLE ADVERSARIAL PERTURBATIONS - Tencent Blade Team - DEF CON 26 CAA...
 
ADVERSARIAL^2 TRAINING - Yao Zhao, Yuzhe Zhao - DEF CON 26 CAAD VILLAGE
ADVERSARIAL^2 TRAINING - Yao Zhao, Yuzhe Zhao - DEF CON 26 CAAD VILLAGEADVERSARIAL^2 TRAINING - Yao Zhao, Yuzhe Zhao - DEF CON 26 CAAD VILLAGE
ADVERSARIAL^2 TRAINING - Yao Zhao, Yuzhe Zhao - DEF CON 26 CAAD VILLAGE
 
HOW TO LEVERAGE THE OPEN-SOURCE INFORMATION TO MAKE AN EFFECTIVE ADVERSARIAL ...
HOW TO LEVERAGE THE OPEN-SOURCE INFORMATION TO MAKE AN EFFECTIVE ADVERSARIAL ...HOW TO LEVERAGE THE OPEN-SOURCE INFORMATION TO MAKE AN EFFECTIVE ADVERSARIAL ...
HOW TO LEVERAGE THE OPEN-SOURCE INFORMATION TO MAKE AN EFFECTIVE ADVERSARIAL ...
 
BOOSTING ADVERSARIAL ATTACKS WITH MOMENTUM - Tianyu Pang and Chao Du, THU - D...
BOOSTING ADVERSARIAL ATTACKS WITH MOMENTUM - Tianyu Pang and Chao Du, THU - D...BOOSTING ADVERSARIAL ATTACKS WITH MOMENTUM - Tianyu Pang and Chao Du, THU - D...
BOOSTING ADVERSARIAL ATTACKS WITH MOMENTUM - Tianyu Pang and Chao Du, THU - D...
 
RECENT PROGRESS IN ADVERSARIAL DEEP LEARNING ATTACK AND DEFENSE - Wenbo Guo a...
RECENT PROGRESS IN ADVERSARIAL DEEP LEARNING ATTACK AND DEFENSE - Wenbo Guo a...RECENT PROGRESS IN ADVERSARIAL DEEP LEARNING ATTACK AND DEFENSE - Wenbo Guo a...
RECENT PROGRESS IN ADVERSARIAL DEEP LEARNING ATTACK AND DEFENSE - Wenbo Guo a...
 
WEAPONS FOR DOG FIGHT:ADAPTING MALWARE TO ANTI-DETECTION BASED ON GAN - Zhuan...
WEAPONS FOR DOG FIGHT:ADAPTING MALWARE TO ANTI-DETECTION BASED ON GAN - Zhuan...WEAPONS FOR DOG FIGHT:ADAPTING MALWARE TO ANTI-DETECTION BASED ON GAN - Zhuan...
WEAPONS FOR DOG FIGHT:ADAPTING MALWARE TO ANTI-DETECTION BASED ON GAN - Zhuan...
 
Hardware Trojan Attacks on Neural Networks - Joseph Clements - DEF CON 26 CAA...
Hardware Trojan Attacks on Neural Networks - Joseph Clements - DEF CON 26 CAA...Hardware Trojan Attacks on Neural Networks - Joseph Clements - DEF CON 26 CAA...
Hardware Trojan Attacks on Neural Networks - Joseph Clements - DEF CON 26 CAA...
 
Huiming Liu-'resident evil' of smart phones--wombie attack
Huiming Liu-'resident evil' of smart phones--wombie attackHuiming Liu-'resident evil' of smart phones--wombie attack
Huiming Liu-'resident evil' of smart phones--wombie attack
 
Zhiyun Qian-what leaves attacker hijacking USA Today site
Zhiyun Qian-what leaves attacker hijacking USA Today siteZhiyun Qian-what leaves attacker hijacking USA Today site
Zhiyun Qian-what leaves attacker hijacking USA Today site
 
Bo Li-they’ve created images that reliably fool neural network
Bo Li-they’ve created images that reliably fool neural networkBo Li-they’ve created images that reliably fool neural network
Bo Li-they’ve created images that reliably fool neural network
 
Alexey kurakin-what's new in adversarial machine learning
Alexey kurakin-what's new in adversarial machine learningAlexey kurakin-what's new in adversarial machine learning
Alexey kurakin-what's new in adversarial machine learning
 
Nick Stephens-how does someone unlock your phone with nose
Nick Stephens-how does someone unlock your phone with noseNick Stephens-how does someone unlock your phone with nose
Nick Stephens-how does someone unlock your phone with nose
 

Recently uploaded

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
ScyllaDB
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
DanBrown980551
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Zilliz
 
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Jeffrey Haguewood
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
akankshawande
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Alex Pruden
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
innovationoecd
 
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
Intelisync
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframeDigital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Precisely
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Postman
 
Trusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process MiningTrusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process Mining
LucaBarbaro3
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Zilliz
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 
Azure API Management to expose backend services securely
Azure API Management to expose backend services securelyAzure API Management to expose backend services securely
Azure API Management to expose backend services securely
Dinusha Kumarasiri
 

Recently uploaded (20)

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
 
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
 
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframeDigital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
 
Trusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process MiningTrusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process Mining
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 
Azure API Management to expose backend services securely
Azure API Management to expose backend services securelyAzure API Management to expose backend services securely
Azure API Management to expose backend services securely
 

THE VANISHING TRICK FOR SELF-DRIVING CARS - Weilin Xu - DEF CON 26 CAAD VILLAGE

  • 1. Magic Tricks for Self-driving Cars Weilin Xu, Zhenyu Zhong, Yunhan Jia
  • 2. • This is a proof-of-concept. • We are NOT targeting at any autonomous vehicle vendors. • Don’t try to fool your neighbor’s car. AUTHORS’ WARNING
  • 3. Autonomous Vehicle Framework Radar LiDAR Camera Image: https://medium.com/toyota-ai-ventures/https-medium-com-toyota-ai-ventures-announcingblackmore-7947eacc9e9e Camera
  • 5.
  • 6. Our Target: YOLOv3 YOLO v3 Object Detection Model [147 Layers, 62M Parameters] Input [416x416x3] Output [3549 Bounding Boxes] Image: http://media.nj.com/traffic_impact/photo/all-way-stop-sign-that-flashes-in-montclairjpg-30576ab330660eff.jpg
  • 7. Trained with the COCO Dataset • Common Objects in Context • 80 Classes: person, [car, truck, bus], [bicycle, motorcycle], [stop sign, traffic light], etc. Source: http://cocodataset.org/
  • 8. car 0.01% YOLOv3 Inference 116x9 0 156x 198 373x326Anchor Boxes 𝑏 𝑥 = 𝜎 𝑡 𝑥 + 𝑐 𝑥 𝑏 𝑦 = 𝜎 𝑡 𝑦 + 𝑐 𝑦 𝑏 𝑤 = 𝑝 𝑤 𝑒 𝑡 𝑤 Center Point Object Size 𝑏ℎ = 𝑝ℎ 𝑒 𝑡ℎ 𝑝ℎ 𝑝 𝑤 13 x 13 Grid (𝑐 𝑥, 𝑐 𝑦) = (11,2) Prediction Vector Bounding Box 80 Class ConfidenceObjectness 𝑡 𝑥 𝑡 𝑦 𝑡 𝑤 𝑡ℎ 𝑝 𝑜𝑏𝑗 𝑐1 𝑐2 … … 𝑐79 𝑐80 ■ stop sign 99% × car 0.01% car 0.01%
  • 9. Threat Model: Image Patch Attack Company Logo
  • 10. Threat Model: Image Patch Attack Company Logo
  • 11. Threat Model: Image Patch Attack Company Logo
  • 12. Threat Model: Image Patch Attack Company Logo
  • 13. Attack Algorithms • Input Construction • Objectives • Optimization
  • 15. Objectives • Object Production • Object Vanish • Object Transformation
  • 16. Object Production - Coarse We want more certain objects on the whole image. • Easy to implement. • May be difficult to optimize. Company Logo
  • 17. Object Production - Precise We want a certain object of a specific size in a specific location. Company Logo
  • 18. Object Vanish - Coarse We want a certain object class to vanish on the whole image.
  • 19. Object Transformation - Coarse We want certain object class to transform to other class.
  • 20. Optimization • Change of variable Convert to tanh() space to encode the [0, 1] interval constraint. Friendly to many off-the-shelf optimizers, e.g. Adam. • To optimize logits Skip sigmoid() to avoid vanishing gradient. Carlini, Nicholas, and David Wagner. “Towards Evaluating the Robustness of Neural Networks.” IEEE S&P (Oakland) 2016.
  • 21. But, Image Sensing is not an Identity Function Model Input [416x416x3] ▪ Limited Resolution ▪ Distortions ▪ Random Noise ▪ … Digital Image Scene
  • 22. Towards Robust Physical Adversarial Examples • [Limited Resolution] Smoother patch with the total variation regularization. • [Distortions] Color management with the non-printability loss. • [Inaccurate Patch] Random transformation during optimization iterations. • …… Sharif, Mahmood, et al. "Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition." ACM CCS 2016.
  • 23. Conclusion • Magicians can fool object detection models, so can attackers. • We should be cautious with self-driving cars that rely on computer vision.