In my presentation I will show you a couple of applications that use artificial intelligence in order to improve our security and how easily it is to use other AI to break it. You may like it or not, but natural language processing, deep learning, computer vision are being developed very rapidly and already have significant impact on your life, working behind the scenes of multiple services you use every day. However, as a great man once said "with great power comes great responsibility", same with the AI - the risk of abuse appears. I will show you how to beat AI using rogue AI, how a crowd-sourced human intelligence can beat AI, or finally how a small, unnoticed by human change in the input data (constructed by AI of course) can severly impact the output of AI processing. I will focus on applications that improve our security not only in the cyber world (like CAPTCHA), but also in real life world (e.g. car safety systems). Last, but not least, I will tell you how to prevent such abuses and why it is so important to understand how above-mentioned tools work.

1. www.securing.pl@drdr_zz
Artificial Intelligence –
a buzzword,
new era of IT or
new threats?
Damian Rusinek
InfoShare 2019

2. www.securing.pl@drdr_zz www.securing.pl@drdr_zz
A story from the 19th century

3. www.securing.pl@drdr_zz www.securing.pl@drdr_zz
• Intentions were good (bring peace to the world)
„On the day when two army corps may mutually
annihilate each other in a second, probably all civilized
nations will recoil with horror and disband their troops.”
• The reality was different
„I intend to leave after my death a large fund for the
promotion of the peace idea, but I am skeptical as to its
results.”
• The result – Nobel Prize
A story from the 19th century

4. www.securing.pl@drdr_zz
Back to the 21st century
• Artificial Intelligence
• Solves (very efficiently) problems that were unsolvable.
• Will AI revolutionize IT?

5. www.securing.pl@drdr_zz www.securing.pl@drdr_zz
Damian Rusinek
• Will AI introduce new threats?
• Will AI be the next dynamite?
• Will AI become a powerful weapon od 21st
century?
The security perspective
Security Researcher & Pentester
Assistant Professor

6. www.securing.pl@drdr_zz www.securing.pl
Rogue AI
MY AI WILL BREAK
YOUR AI

7. www.securing.pl@drdr_zz

8. www.securing.pl@drdr_zz
• Verifier must know which images present cars
• Simple solution:
• A big database of manually categorized images
• AI solution:
• Use AI solution to recognize objects on images
and categorize them
• AI ready to use solutions:
• Inception (GoogleNet)
• AlexNet
• ResNet
• VGG
AI behind reCAPTCHA

9. www.securing.pl@drdr_zz
• Use existing image recognition solutions to solve CAPTCHA puzzles
• Google Reverse Image Search, Clarifai, Alchemy, TDL, NeuralTalk, Caffe
• Target
• Google reCAPTCHA
• Facebook CAPTCHA
Rogue AI for reCAPTCHA

10. www.securing.pl@drdr_zz
• Number of collected CAPTCHA image puzzles
• 63 000 for Google reCAPTCHA
• 200 for Facebook CAPTCHA
• Results
• Google reCAPTCHA – 70% (19 seconds)
• Facebook CAPTCHA – 83%
• With 40.000+ CAPTCHAs per day per host
Rogue AI for reCAPTCHA

11. www.securing.pl@drdr_zz www.securing.pl
Crowd-sourced human intelligence
- AI will take care of your support
AI CHAT BOT

12. www.securing.pl@drdr_zz
• Goal
• Automatic support agent
• Uses AI to learn FAQ for new processes
• Natural Language Processing
• Experiment
• Tay (abbr. Thinking about you)
• A twitter account by Microsoft (@TayandYou)
• Designed to mimic the language patterns of
a 19-year-old American girl
AI Chat Bot

13. www.securing.pl@drdr_zz
• Learns from interacting with
human users of Twitter
• Threat
• Knowledge from untrusted source
• Anyone could teach Tay
What can go wrong?
Users posted incorrect and offensive tweets to Tay and made it…
AI Chat Bot

14. www.securing.pl@drdr_zz
• Tay became:
• Racist
• Nazi
AI Chat Bot

15. www.securing.pl@drdr_zz
• Taken down after 16 hours and 96 000 tweets
• Lesson learned
• Define the boundaries
• Do not allow untrusted source to teach your AI
• The next Tay – Zo
• Twitter, Facebook and Skype
• Does not talk about sensitive topics
Are you ready for a Nazi in your support team?
AI Chat Bot

16. www.securing.pl@drdr_zz www.securing.pl
Small change (unnoticed by human)
- Will you entrust your life to AI?
AUTOMOTIVE AI

17. www.securing.pl@drdr_zz
• Artificial Intelligence in Automotive
• Rain sensor
• AI recognizes rain drops on the windshield
• Lane recognition
• Autopilot keeps the car on the lane
• Attack:
• Funny – turn on the wipers
• Scary – make the car to change lane to the opposite
AI in automotive

18. www.securing.pl@drdr_zz
• Tencent Keen Security Lab
• Took out and analyze the autopilot component
Lane detection attack

19. www.securing.pl@drdr_zz
• Tencent Keen Security Lab
• Took out and analyze the autopilot component
Lane detection attack

20. www.securing.pl@drdr_zz
• Attack scenario
• Change the „input image” to fool AI.
• Challenges
• Find out how to change the image.
• Change the physical world.
„Most of the adversarial examples generated in digital domain are pixel level’s
change, so it’s hard to deploy them in physical world.”
AI in automotive

21. www.securing.pl@drdr_zz
• Easy to get if you have access to the AI internals.
Activation map
Learning Deep Features for Discriminative Localization, Zhou et al., MIT

22. www.securing.pl@drdr_zz
• Simple change in physical world
• Can you see it?
Successful lane detection attack

23. www.securing.pl@drdr_zz
• Simple change in physical world
• Can you see it?
Successful lane detection attack

24. www.securing.pl@drdr_zz
• What can you see?
How hard is it to generate malicious input?
Egyptian cat
78% (by alexnet)
Assault rifle
93% (by alexnet)

25. www.securing.pl@drdr_zz
How hard is it to generate malicious input?
Pixels modified Pixels modified
By more than 1%
Pixels modified
By more than 2%

26. www.securing.pl@drdr_zz
DEMO
How hard is it to generate malicious input?
Access to the AI internals?
Easy
No access?
Harder but possible

27. www.securing.pl@drdr_zz
Porn alert!
No need to change

28. www.securing.pl@drdr_zz
Buzzword
Back to the question

29. www.securing.pl@drdr_zz
Buzzword
New era of IT?
Back to the question

30. www.securing.pl@drdr_zz
Buzzword
New era of IT?
New security threats?
Back to the question

31. www.securing.pl@drdr_zz
Design
• Threat modelling
• Consider rogue AI as
threat
• Define boundaries
AI security
Development
• No untrusted source
teaching your AI
• Generate malicious
inputs and teach your AI
Use
• No critical decisions
based on AI only
• Monitor outputs from
AI (be up to date)
• Control boundaries
Architecture
Assessment
System
Testing
System
Monitoring

32. www.securing.pl@drdr_zz
Thank you!
Contact me:
damian.rusinek@securing.pl
Ready for ?
We are!
@drdr_zz