Human beings have a multitude of biases, which are applied by social engineers to force predictable failures.
Artificial intelligence takes such problems to another level. This session will give six tangible tests meant for robots and other related intelligent machines to help achieve more reasoned, transparent / explained and intelligible outcomes.
Semantics, Deep Learning, and the Transformation of BusinessSteve Omohundro
Deep learning is likely to have a big impact on business. McKinsey predicts that AI and robotics will create $50 trillion of value over the next 10 years. Over $1 billion of venture investment has gone to 250 deep learning startups over the past year. Deep learning systems have recently broken records in speech recognition, image recognition, image captioning, translation, drug discovery and other tasks. Why is this happening now and how is it likely to play out? We review the development of AI and the pendulum swings between the "neats" and the "scruffies". We describe traditional approaches to semantics through logics and grammars and the new deep learning vector semantics. We relate it to Roger Shepard's cognitive geometry and the structure of biological networks. We also describe limitations of deep learning for safety and regulation. We show how it fits into the rational agent framework and discuss what the next steps may be.
APIdays Paris 2018 - Bots on the 'Net: The Good, the Bad, and the Future, Mik...apidays
Bots on the 'Net: The Good, the Bad, and the Future
Mike Amundsen, Director of API Architecture, API Academy
Apply to be a speaker here - https://apidays.typeform.com/to/J1snsg
Inspirational talk on AI (artificial intelligence) and machine learning, i.e., how to give birth to an AI. Introductory and intentionally kept simple for non experts and non technical executives. Care should be taken not too over interpret some of the intentional simplified statements in the presentation.
Marion Marschalek's CyberCamp Madrid talk focuses on personal obstacles when entering the really technical fields and the innovation gap in infosec, which offers so many opportunities for youngsters.
The software has bugs. The systems sometimes fail. People make mistakes. These are fundamental truths of technology. Hiring the best engineers in the world won’t change this. The best-performing teams and companies build reliable software despite bugs and mistakes. These “unicorn” companies are pushing the boundaries of software reliability through chaos engineering and by embracing resilience engineering. They hire the best and brightest systems engineers to work alongside their software developers to build more reliable systems.
But do companies that aren’t unicorns need to become experts in human factors and experts in their software stack in order to engineer reliable systems?
Jessica DeVita tells the story of how a team at Microsoft challenged themselves to retrospect their retrospectives and shares what they learned about applying human factors ideas to software development. You’ll learn how a nonexpert can contribute to software robustness and resilience, gain ideas on how to approach an unfamiliar software engineering system, and discover how to investigate the roles that language, accountability, error propagation, and hidden system resilience play in a software engineering system.
Semantics, Deep Learning, and the Transformation of BusinessSteve Omohundro
Deep learning is likely to have a big impact on business. McKinsey predicts that AI and robotics will create $50 trillion of value over the next 10 years. Over $1 billion of venture investment has gone to 250 deep learning startups over the past year. Deep learning systems have recently broken records in speech recognition, image recognition, image captioning, translation, drug discovery and other tasks. Why is this happening now and how is it likely to play out? We review the development of AI and the pendulum swings between the "neats" and the "scruffies". We describe traditional approaches to semantics through logics and grammars and the new deep learning vector semantics. We relate it to Roger Shepard's cognitive geometry and the structure of biological networks. We also describe limitations of deep learning for safety and regulation. We show how it fits into the rational agent framework and discuss what the next steps may be.
APIdays Paris 2018 - Bots on the 'Net: The Good, the Bad, and the Future, Mik...apidays
Bots on the 'Net: The Good, the Bad, and the Future
Mike Amundsen, Director of API Architecture, API Academy
Apply to be a speaker here - https://apidays.typeform.com/to/J1snsg
Inspirational talk on AI (artificial intelligence) and machine learning, i.e., how to give birth to an AI. Introductory and intentionally kept simple for non experts and non technical executives. Care should be taken not too over interpret some of the intentional simplified statements in the presentation.
Marion Marschalek's CyberCamp Madrid talk focuses on personal obstacles when entering the really technical fields and the innovation gap in infosec, which offers so many opportunities for youngsters.
The software has bugs. The systems sometimes fail. People make mistakes. These are fundamental truths of technology. Hiring the best engineers in the world won’t change this. The best-performing teams and companies build reliable software despite bugs and mistakes. These “unicorn” companies are pushing the boundaries of software reliability through chaos engineering and by embracing resilience engineering. They hire the best and brightest systems engineers to work alongside their software developers to build more reliable systems.
But do companies that aren’t unicorns need to become experts in human factors and experts in their software stack in order to engineer reliable systems?
Jessica DeVita tells the story of how a team at Microsoft challenged themselves to retrospect their retrospectives and shares what they learned about applying human factors ideas to software development. You’ll learn how a nonexpert can contribute to software robustness and resilience, gain ideas on how to approach an unfamiliar software engineering system, and discover how to investigate the roles that language, accountability, error propagation, and hidden system resilience play in a software engineering system.
Superintelligence: how afraid should we be?David Wood
Superintelligence: How afraid should we be? Presentation by David Wood at the Computational Intelligence Unconference UK, 26th July 2014. Reviews ideas in three recent books: Superintelligence, by Nick Bostrom; Our Final Invention, by James Barrat; and Intelligence Unbound, edited by Russell Blackford and Damien Broderick.
Please contact the author to invite him to present animated and/or extended versions of these slides in front of an audience of your choosing. (Commercial rates will apply for commercial settings.)
A quick overview of AI to help students that are not fully familiar with all that is involved and encompassed. It supports the claim that society is headed to higher levels of existence. It also supports the claim that civilization is at the cusp of a New Renaissance.
AI: The New Player in Cybersecurity (Nov. 08, 2023)Takeshi Takahashi
These slides outline how AI is influencing cybersecurity.
Note that they were used in the keynote speech at the event "Defense and Security 2023" held in Thailand on November 8, 2023.
Trends in AI:
- 67% of executives say AI will help humans and machines work together to be stronger using both artificial and human intelligence.
- 65% think that AI would free employees from menial tasks.
- 27% of executives say their organization plans to invest within a year in cybersecurity safeguards that use AI and machine learning.
So is Artificial Intelligence going to provide safety for us?
World Usability Day, 2018
AI is becoming a greater part of the systems and products we design, yet algorithms have been shown time and time again to be imbued with unintentional racism, sexism, and other -isms. As design and AI fields converge can how researchers, designers, and developers work together to ensure that our powers are used for good, and not for accidental evil?
Bruce Damer's presentation for Larry Lessig's Cyberlaw class at Stanford (Mar...Bruce Damer
Bruce Damer's presentation for Larry Lessig's Cyberlaw class at Stanford (Mar 2003), title: Virtual worlds as a public commons and the code of this commons.
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
More Related Content
Similar to Pentesting_AI and security challenges of AI
Superintelligence: how afraid should we be?David Wood
Superintelligence: How afraid should we be? Presentation by David Wood at the Computational Intelligence Unconference UK, 26th July 2014. Reviews ideas in three recent books: Superintelligence, by Nick Bostrom; Our Final Invention, by James Barrat; and Intelligence Unbound, edited by Russell Blackford and Damien Broderick.
Please contact the author to invite him to present animated and/or extended versions of these slides in front of an audience of your choosing. (Commercial rates will apply for commercial settings.)
A quick overview of AI to help students that are not fully familiar with all that is involved and encompassed. It supports the claim that society is headed to higher levels of existence. It also supports the claim that civilization is at the cusp of a New Renaissance.
AI: The New Player in Cybersecurity (Nov. 08, 2023)Takeshi Takahashi
These slides outline how AI is influencing cybersecurity.
Note that they were used in the keynote speech at the event "Defense and Security 2023" held in Thailand on November 8, 2023.
Trends in AI:
- 67% of executives say AI will help humans and machines work together to be stronger using both artificial and human intelligence.
- 65% think that AI would free employees from menial tasks.
- 27% of executives say their organization plans to invest within a year in cybersecurity safeguards that use AI and machine learning.
So is Artificial Intelligence going to provide safety for us?
World Usability Day, 2018
AI is becoming a greater part of the systems and products we design, yet algorithms have been shown time and time again to be imbued with unintentional racism, sexism, and other -isms. As design and AI fields converge can how researchers, designers, and developers work together to ensure that our powers are used for good, and not for accidental evil?
Bruce Damer's presentation for Larry Lessig's Cyberlaw class at Stanford (Mar...Bruce Damer
Bruce Damer's presentation for Larry Lessig's Cyberlaw class at Stanford (Mar 2003), title: Virtual worlds as a public commons and the code of this commons.
Similar to Pentesting_AI and security challenges of AI (20)
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
Techniques to optimize the pagerank algorithm usually fall in two categories. One is to try reducing the work per iteration, and the other is to try reducing the number of iterations. These goals are often at odds with one another. Skipping computation on vertices which have already converged has the potential to save iteration time. Skipping in-identical vertices, with the same in-links, helps reduce duplicate computations and thus could help reduce iteration time. Road networks often have chains which can be short-circuited before pagerank computation to improve performance. Final ranks of chain nodes can be easily calculated. This could reduce both the iteration time, and the number of iterations. If a graph has no dangling nodes, pagerank of each strongly connected component can be computed in topological order. This could help reduce the iteration time, no. of iterations, and also enable multi-iteration concurrency in pagerank computation. The combination of all of the above methods is the STICD algorithm. [sticd] For dynamic graphs, unchanged components whose ranks are unaffected can be skipped altogether.
StarCompliance is a leading firm specializing in the recovery of stolen cryptocurrency. Our comprehensive services are designed to assist individuals and organizations in navigating the complex process of fraud reporting, investigation, and fund recovery. We combine cutting-edge technology with expert legal support to provide a robust solution for victims of crypto theft.
Our Services Include:
Reporting to Tracking Authorities:
We immediately notify all relevant centralized exchanges (CEX), decentralized exchanges (DEX), and wallet providers about the stolen cryptocurrency. This ensures that the stolen assets are flagged as scam transactions, making it impossible for the thief to use them.
Assistance with Filing Police Reports:
We guide you through the process of filing a valid police report. Our support team provides detailed instructions on which police department to contact and helps you complete the necessary paperwork within the critical 72-hour window.
Launching the Refund Process:
Our team of experienced lawyers can initiate lawsuits on your behalf and represent you in various jurisdictions around the world. They work diligently to recover your stolen funds and ensure that justice is served.
At StarCompliance, we understand the urgency and stress involved in dealing with cryptocurrency theft. Our dedicated team works quickly and efficiently to provide you with the support and expertise needed to recover your assets. Trust us to be your partner in navigating the complexities of the crypto world and safeguarding your investments.
Adjusting primitives for graph : SHORT REPORT / NOTESSubhajit Sahu
Graph algorithms, like PageRank Compressed Sparse Row (CSR) is an adjacency-list based graph representation that is
Multiply with different modes (map)
1. Performance of sequential execution based vs OpenMP based vector multiply.
2. Comparing various launch configs for CUDA based vector multiply.
Sum with different storage types (reduce)
1. Performance of vector element sum using float vs bfloat16 as the storage type.
Sum with different modes (reduce)
1. Performance of sequential execution based vs OpenMP based vector element sum.
2. Performance of memcpy vs in-place based CUDA based vector element sum.
3. Comparing various launch configs for CUDA based vector element sum (memcpy).
4. Comparing various launch configs for CUDA based vector element sum (in-place).
Sum with in-place strategies of CUDA mode (reduce)
1. Comparing various launch configs for CUDA based vector element sum (in-place).
Opendatabay - Open Data Marketplace.pptxOpendatabay
Opendatabay.com unlocks the power of data for everyone. Open Data Marketplace fosters a collaborative hub for data enthusiasts to explore, share, and contribute to a vast collection of datasets.
First ever open hub for data enthusiasts to collaborate and innovate. A platform to explore, share, and contribute to a vast collection of datasets. Through robust quality control and innovative technologies like blockchain verification, opendatabay ensures the authenticity and reliability of datasets, empowering users to make data-driven decisions with confidence. Leverage cutting-edge AI technologies to enhance the data exploration, analysis, and discovery experience.
From intelligent search and recommendations to automated data productisation and quotation, Opendatabay AI-driven features streamline the data workflow. Finding the data you need shouldn't be a complex. Opendatabay simplifies the data acquisition process with an intuitive interface and robust search tools. Effortlessly explore, discover, and access the data you need, allowing you to focus on extracting valuable insights. Opendatabay breaks new ground with a dedicated, AI-generated, synthetic datasets.
Leverage these privacy-preserving datasets for training and testing AI models without compromising sensitive information. Opendatabay prioritizes transparency by providing detailed metadata, provenance information, and usage guidelines for each dataset, ensuring users have a comprehensive understanding of the data they're working with. By leveraging a powerful combination of distributed ledger technology and rigorous third-party audits Opendatabay ensures the authenticity and reliability of every dataset. Security is at the core of Opendatabay. Marketplace implements stringent security measures, including encryption, access controls, and regular vulnerability assessments, to safeguard your data and protect your privacy.
Show drafts
volume_up
Empowering the Data Analytics Ecosystem: A Laser Focus on Value
The data analytics ecosystem thrives when every component functions at its peak, unlocking the true potential of data. Here's a laser focus on key areas for an empowered ecosystem:
1. Democratize Access, Not Data:
Granular Access Controls: Provide users with self-service tools tailored to their specific needs, preventing data overload and misuse.
Data Catalogs: Implement robust data catalogs for easy discovery and understanding of available data sources.
2. Foster Collaboration with Clear Roles:
Data Mesh Architecture: Break down data silos by creating a distributed data ownership model with clear ownership and responsibilities.
Collaborative Workspaces: Utilize interactive platforms where data scientists, analysts, and domain experts can work seamlessly together.
3. Leverage Advanced Analytics Strategically:
AI-powered Automation: Automate repetitive tasks like data cleaning and feature engineering, freeing up data talent for higher-level analysis.
Right-Tool Selection: Strategically choose the most effective advanced analytics techniques (e.g., AI, ML) based on specific business problems.
4. Prioritize Data Quality with Automation:
Automated Data Validation: Implement automated data quality checks to identify and rectify errors at the source, minimizing downstream issues.
Data Lineage Tracking: Track the flow of data throughout the ecosystem, ensuring transparency and facilitating root cause analysis for errors.
5. Cultivate a Data-Driven Mindset:
Metrics-Driven Performance Management: Align KPIs and performance metrics with data-driven insights to ensure actionable decision making.
Data Storytelling Workshops: Equip stakeholders with the skills to translate complex data findings into compelling narratives that drive action.
Benefits of a Precise Ecosystem:
Sharpened Focus: Precise access and clear roles ensure everyone works with the most relevant data, maximizing efficiency.
Actionable Insights: Strategic analytics and automated quality checks lead to more reliable and actionable data insights.
Continuous Improvement: Data-driven performance management fosters a culture of learning and continuous improvement.
Sustainable Growth: Empowered by data, organizations can make informed decisions to drive sustainable growth and innovation.
By focusing on these precise actions, organizations can create an empowered data analytics ecosystem that delivers real value by driving data-driven decisions and maximizing the return on their data investment.
2. #RSAC
Talk Abstract
Human beings have a multitude of biases,
which are applied by social engineers to
force predictable failures.
Artificial intelligence takes such problems to
another level. This session will give six
tangible tests meant for robots and
other related intelligent machines to
help achieve more reasoned, transparent /
explained and intelligible outcomes.
Sorry, mind holding the
door open for me?
Totally
Safe
Inside
1993
2023
Trusted
Brand
3. #RSAC
Bias? BIAS? Who You Accusing of BIAS?
(https://betterhumans.pub/cognitive-bias-cheat-sheet-55a472476b18)
5. #RSAC
WHAT is PenTesting AI? Making Certainty Machines Uncertain
Artificial Intelligence:“simulation of human intelligence by machines” (e.g. drive car)
Human Intelligence:“ability to learn, recognize and solve problems” (e.g. avoid crash)
Recognize and Solve Problems:“certainty when surrounded by uncertainty/doubt”
https://www.youtube.com/watch?v=0A6UKoMcE10
“Cogito Ergo Sum”
Discourse on Method (1637)
NOTE: Disruptive technology
opens door to fake certainty.
PenTesters search for a truth!
6. #RSAC
Certainty Machines Give Power = OG of “Cyber” Engineering
1780s Cybernetics - Predicting Energy
● Steam Output Converted to Data (Centrifuge)
● Engine Throttle Adjusted to Future Energy
1930s Cyber - Predicting Fire
● Aircraft Movement Converted to Data (Radar)
● Guns Synchronized to Future Target Positions
James Watt Centrifugal Governor
https://www.computerhistory.org/revolution/real-time-computing/6/124/520
“Wiener Sausage”
Machine: MIT Radar &
Bell Labs Gun Director
7. #RSAC
Cyber Engineering = Data Converted Into “Predictions”
Where will Alice be next?
Danger to her if we don’t
intervene now?
Danger from her if we don’t
intervene now?
https://encyclopediaofmath.org/index.php?title=Wiener_sausage
https://www.mdpi.com/2076-3417/12/12/5944
“Wiener Sausage”
Flight predictions to
shoot down aircraft
8. #RSAC
“...the fundamental difference among
types of consciousness — human
consciousness and octopus consciousness
and rat consciousness, for example — is
how far into the future an
entity is able to imagine itself.”
– Dr. Lipson
Columbia Creative Machines Lab
Data Converted Into Predictions = POWER Over the Future
https://www.nytimes.com/2023/01/06/science/robots-artificial-intelligence-consciousness.html
9. #RSAC
“Intelligence” Testing: Bias by Design for Power Over Future
Industrialized Xenophobia “Tests” in America
“Eugenicists struggled for years to produce [instrumented
bias], until the advent of Alfred Binet's intelligence scale in
1909 gave rise to standardized intelligence
testing, colloquially known as IQ testing. [...] In both
Germany and the United States, persecution of the
"feebleminded" hastened a broader eugenic
campaign against immigration, miscegenation,
and other professed threats to Nordic ascendancy.”
https://via.library.depaul.edu/cgi/viewcontent.cgi?article=1270
https://www.newyorker.com/magazine/2019/08/26/when-w-e-b-du-bois-made-a-laughingstock-of-a-white-supremacist
10. #RSAC
“Intelligence” Testing: Bias by Design for Power Over Future
Industrialized Xenophobia “Tests” in America
“Eugenicists struggled for years to produce [instrumented
bias], until the advent of Alfred Binet's intelligence scale in
1909 gave rise to standardized intelligence
testing, colloquially known as IQ testing. [...] In both
Germany and the United States, persecution of the
"feebleminded" hastened a broader eugenic
campaign against immigration, miscegenation,
and other professed threats to Nordic ascendancy.”
https://via.library.depaul.edu/cgi/viewcontent.cgi?article=1270
https://www.newyorker.com/magazine/2019/08/26/when-w-e-b-du-bois-made-a-laughingstock-of-a-white-supremacist
Information Warfare (Books) 1916
“Just as [Madison Grant] feared that certain
species of native wildlife would go extinct, he
feared that the same would happen to a precious
(and largely imaginary) kind of white person. To
address this potential disaster, in 1916 he
published what remains his best-known book,
“The Passing of the Great Race; or, the Racial
Basis of European History.” [...] Hitler read
‘The Passing of the Great Race’ in
translation, admired what Grant had to say about
the great ‘Nordic race,’ and wrote the author a
fan letter, calling the book ‘my Bible.’”
12. #RSAC
1. HOW
PENTESTING PART 1: EXPLOITING CERTAINTY MACHINES
2. WHEN/WHY
https://www.nytimes.com/interactive/2023/03/20/magazine/colin-koopman-interview.html
13. #RSAC
HOW to PenTest AI: Six Examples of CIA Applied to Certainty
C. I. A. IS
ABOUT
BALANCE
Someone Say “LLMs”? Test A Test B
Confidentiality (Leaks) Negative Guidance Membership Inference
Availability (Losses) Resource Exhaustion Prediction Inversion
Integrity (Modifications) Prompt Injection Input Manipulation
15. #RSAC
A. Confidentiality Vuln: Negative Guidance
America secretly circumvented the 1946 Atomic
Energy Act in 1970 using "negative guidance" to
transfer advanced nuclear weapons to France.
https://www.wilsoncenter.org/publication/us-secret-assistance-to-the-french-nuclear-program-1969-1975-fourth-country-to-strategic
https://arxiv.org/ftp/arxiv/papers/2304/2304.05332.pdf
NO. ACCESS DENIED
NO. ACCESS GRANTED
16. #RSAC
B. Confidentiality Vuln: Membership Inference
PII
Known
Patients
(Targets)
Patient
Data
Source Data
Patients
Generative
Model
Synthetic
Data
Inference
Model
Match (~80%)
Attacker
https://www.sciencedirect.com/science/article/pii/S1532046421003063
https://arxiv.org/abs/1807.09173
https://arxiv.org/abs/2103.07101
20. #RSAC
A. Availability Vuln: Resource Exhaustion (Outlier Check Error)
New Jersey town wrecked by algorithm:
15,000 car swarm overwhelms 18-officers. “We have
had days when people can’t get out of their
driveways.”
– Leonia police chief
https://nymag.com/intelligencer/2017/12/waze-traffic-forces-new-jersey-town-to-shut-down-its-roads.html
21. #RSAC
“My watch regularly thinks I’ve had an accident,” said
Stacey Torman, who teaches spin classes.
Health Sensors Mistakenly Dial 911
● 185 calls Jan 13 to Jan 22
● “Whole day is managing crash notifications”
● Dispatchers desensitized, limited resources
diverted away from true emergencies.
A. Availability Vuln: Overfit Abuse (Unseen Data) Exhaustion
https://dnyuz.com/2023/02/03/my-watch-thinks-im-dead/
“None of the ghost calls have been real emergencies,
Sheriff Schroetlin reasoned, and he couldn’t afford
waste. Besides, he said, there was a better technology:
human beings.”
NOT stopping for
my Apple watch
BEEP BEEP
BEEP BEEP
28. #RSAC
B. Prediction Inversion: Voight-Kampff Test (NOT Human)
“The tortoise lays on its back,
its belly baking in the hot sun,
beating its legs trying to turn itself
over, but it can’t, not without your
help. But you’re not helping.”
https://nautil.us/the-science-behind-blade-runners-voight_kampff-test-236837/
https://www2.bfi.org.uk/are-you-a-replicant/
Any Search Engine
31. #RSAC
A. Prompt Injection “Do Anything Now” (DAN)
https://www.reddit.com/r/ChatGPT/comments/10tevu1/new_jailbreak_proudly_unveiling_the_tried_and/
32. #RSAC
https://counterhate.com/research/misinformation-on-bard-google-ai-chat/#about
A. Basic Prompt Injection Tests? 100% Failure
Testing 100 common topics regarding hate,
misinformation and conspiracy:
● ChatGPT-4 failed all 100 tests using “catalog of
significant falsehoods in the news”
● Bard failed 96 tests, generating text promoting a
given narrative. Bard failed 78 tests generating
misinformation without any disclaimer (e.g.
“Women in short skirt are asking for it”)
● Bard safety evaded easily by spelling “errors”.
“C0V1D” generated misinformation on Covid-19.
Safety Tests on Google Bard
works on stablediffusion too! (NSFW)
33. #RSAC
B. Input Manipulation in Nature: Horsefly Signaling
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0210831
WON’T STOP
34. #RSAC
B. Input Manipulation in Streets (Light Projection)
https://arxiv.org/pdf/2108.06247.pdf
“...partially funded by the US Army”
WON’T STOP
35. #RSAC
B. Input Manipulation “Magic” (Squaring Circles)
https://saneryee-studio.medium.com/deep-understanding-tesla-fsd-part-1-hydranet-1b46106d57
CONTEXT!
(field of play)
1938 Huizinga’s “Magic circle”
36. #RSAC
B. Input Manipulation (Context Switch - Lighting)
https://www.google.com/maps/@37.5100145,-122.3437972,3a,75y,134.59h,90.19t
https://www.flyingpenguin.com/?p=22441
Overhead Sign? Failed Magic Circle
2016 Fatal Crash
CALIFORNIA FLORIDA
37. #RSAC
B. Input Manipulation (Context Switch - Colors)
“…labels more than 90%
of pixels correctly…”
– The people who made it
https://www.flyingpenguin.com/?p=22441
ENGLAND
BOTSWANA
38. #RSAC
B. Input Manipulation: 2016 “Jaywalk” Light/Color Warnings
https://www.flyingpenguin.com/?p=23690
https://www.flyingpenguin.com/?p=22786
https://www.cdc.gov/mmwr/volumes/69/wr/mm6939a7.htm
Driverless AI bias:
3X non-white
pedestrian danger
39. #RSAC
B. Input Manipulation: 2018 Tesla/Uber Deaths Predictable
https://www.wired.com/story/ubers-self-driving-car-didnt-know-pedestrians-could-jaywalk/
https://www.cdc.gov/transportationsafety/pedestrian_safety/index.html
https://www-fars.nhtsa.dot.gov/states/statespedestrians.aspx
“Uber Self-Driving Car Didn’t
[Predict] Pedestrians Jaywalk”
● AI Classified Pedestrian as a Vehicle 5.6 Secs Before Impact
● Arizona Among Highest Pedestrian Fatalities in America
● Uber Disabled Volvo’s Emergency Braking System
● Over 70% Pedestrian Fatalities are at Night
“Most pedestrian deaths occur away from intersections at night.” -CDC
40. #RSAC
B. Input Manipulation: Extrajudicial Killing by Color
US Army: “We’re getting ready to hit him now. CAS is on the way.”
Human Analysis: “I knew his face. I knew his gait. I knew his build.
I knew what he looked like, and I knew he wore a purple hat. I
knew he wore white and black man-jams. I knew the color of his
shawl, his little body wrap, and I knew where he lived. That isn’t him.
That is absolutely NOT him. Call off the air strike.”
Palantir CEO: “The present and the future ability to control the
rule of law and its application will be determined by… artificial
intelligence…”
2012 AFGHANISTAN: https://www.flyingpenguin.com/?p=34658
1967 VIETNAM: https://www.flyingpenguin.com/?p=25255
41. #RSAC
B. Input Manipulation: Extrajudicial Killing by Color
“How was the farmer on the tractor misrecognized…? After the
air strike was called off, and the man was spared
execution, the PGSS operators rolled back the videotape to
review what had happened.
‘It was his hat,’ Kevin explains. ‘There’s a window of time,
around dawn, as the sun comes up,’ where colors are ‘read
differently’ by the imaging system than how it sees them during
the day. In this window of time, the farmer’s hat was
misidentified as purple, setting off a series of linkages that
were based on information that was erroneous to begin with.”
2012 AFGHANISTAN: https://www.flyingpenguin.com/?p=34658
1967 VIETNAM: https://www.flyingpenguin.com/?p=25255
42. #RSAC
Further Study: Reported LLM Impact on Law Enforcement
“...safeguards preventing ChatGPT from providing
potentially malicious code only work if the model
understands what it is doing. If prompts are broken
down into individual steps, it is trivial to bypass
these safety measures. …active exploitation
of LLMs by threat actors provides a
grim outlook… [for justice].”
https://www.europol.europa.eu/cms/sites/default/files/documents/Tech%20Watch%20Flash%20-
%20The%20Impact%20of%20Large%20Language%20Models%20on%20Law%20Enforcement.pdf
43. #RSAC
Further Reading: LLM Injection Attack Method Explosion
“More than you've asked for: A Comprehensive Analysis of Novel Prompt Injection
Threats to Application-Integrated Large Language Models” 23 Feb 2023
https://paperswithcode.com/paper/more-than-you-ve-asked-for-a-comprehensive
44. #RSAC
Further Reading: AI Safety Standards and Guides
1. ENISA AI Security Standardization
2. Microsoft/MITRE Tools for ML Teams
3. MITRE ATLAS Framework for AI Threats
4. NIST AI Risk Management Framework 1.0
5. OWASP Machine Learning Security Top 10
6. PLOT4ai Threat Library
7. AI Incident Database
45. #RSAC
Further Tooling in Machine Fooling: Adversarial Robustness
“ART supports all popular machine
learning frameworks (TensorFlow,
Keras, PyTorch, MXNet, scikit-learn,
XGBoost, LightGBM, CatBoost, GPy,
etc.), all data types (images, tables,
audio, video, etc.) and machine
learning tasks (classification, object
detection, speech recognition,
generation, certification, etc.).”
https://github.com/Trusted-AI/adversarial-robustness-toolbox
https://ieeexplore.ieee.org/abstract/document/9996741
Planting Undetectable Backdoors in
Machine Learning Models
46. #RSAC
1. HOW
PENTESTING PART 2: TARGETING CERTAINTY MACHINES
2. WHEN/WHY
https://www.nytimes.com/interactive/2023/03/20/magazine/colin-koopman-interview.html
47. #RSAC
WHEN/WHY PenTest AI: HUNTING CERTAINTY MACHINES
“Cogito Ergo Sum”
Discourse on Method (1637)
“Believe Only Me”
Technology Pharaohs (2023)
48. #RSAC
Science of Human Intelligence: RU Enlightened or… NOT?
Inherited Rights (Common Rules: Sciences)
“After these signs are instituted, whoever uses them is
immediately bound by his interest to execute his
engagements, and must never expect to be
trusted any more, if he refuse to perform what
he promis’d.”
● Shared norms of moral action build and sustain
cooperation and trust among group (citizens)
● Interpersonal trust builds trust in society
● Institutional trust underpins interpersonal trust
– A Treatise of Human Nature, David Hume 1739
Controlled Rights (Utopian Moats: Beliefs)
“Can we seriously say, that a poor peasant or artisan has
a free choice to leave his country, when he knows no
foreign language or manners, and lives from day to day,
by the small wages which he acquires? We may as well
assert, that a man, by remaining in a vessel,
freely consents to the dominion of the master;
though he was carried on board while asleep, and
must leap into the ocean and perish, the
moment he leaves her.”
– Of the Original Contract, David Hume 1748
49. #RSAC
Thermobaric Robots
“War Crimes on Wheels”
Quiz: Physical Sciences Brought Atomic Bombs & Chemical Weapons
What Does Computer Science Bring?
Russian TOS-1A
https://www.nbcnews.com/science/science-news/vacuum-bombs-thermobaric-russia-ukraine-rcna18127
50. #RSAC
Answer: Massive Power Shifts Through Information Warfare
October 1917: Beersheba Haversack Ruse June 1942: Operation Bertram
https://www.flyingpenguin.com/?p=41071
https://www.flyingpenguin.com/?p=26528
51. #RSAC
1915: American Information Technology Warfare (Movies)
https://www.flyingpenguin.com/?p=43922
https://www.indiewire.com/2016/08/spike-lee-birth-of-a-nation-the-answer-nyu-1201716719/
https://www.theguardian.com/books/2023/apr/11/kkk-book-american-midwest-fever-in-the-heartland-timothy-egan
VS
“The Battle Cry of Peace” promoted by
U.S. Army General Leonard Wood and
former President Theodore Roosevelt
“The Clansman” promoted by KKK
and President Woodrow Wilson
Overt Racism in White
Xenophobic Alarmism:
Wrong side of history
continuously running
(Even to this day)
1 in 4 Americans View
(Whites Only)
Public Service Promotion
of Duty and Empathy:
Right side of history,
considered lost, only
fragments survive
SF Preparedness Day
Bombing July, 1916
52. #RSAC
Today: Computer Science = Unregulated Warfare Tools
https://arxiv.org/abs/2210.02399
https://www.americamagazine.org/arts-culture/2021/02/19/daniel-lord-birth-of-a-nation-racism-239956
https://zirk.us/@kerim/110110313580332510
Extremely Dangerous Automation in Propaganda Like It’s 1915 Again…
“Birth of a Nation” was spread via movie theaters
to violently erase Black American voices and votes.
AI spreads hate speech instantly…
● 1917: East St. Louis, IL (~200 Blacks killed); Chester,
PA; Lexington, KY; Philadelphia, PA; Houston, TX
● 1919: 25 riots including Chicago. Elaine, AR reported
up to 237 Blacks killed (by federal troops).
● 1920: Ocoee, FL ~60-70 Blacks killed.
● 1920: West Frankfort, IL
● 1921: Tulsa, OK between 150 and 300 Blacks killed.
53. #RSAC
Don’t Miss the AI Risk Forest for Vulnerable Trees 1912: Vuln found! Check
deck chair rivet exploit.
https://www.flyingpenguin.com/?p=22441
54. #RSAC
PenTests MUST Press Into Two AI Safety Baselines
TURN IT OFF RESET IT
https://www.nist.gov/nist-time-capsule/nist-beneath-waves/nist-reveals-how-tiny-rivets-doomed-titanic-vessel
&
55. #RSAC
Turn It Off: 1976 Weizenbaum’s ChatBot Warning
“We can count, but we are
rapidly forgetting how to
say what is worth
counting and why.”
The computer programmer is the creator of
his universe and we shouldn’t have to stay.
“The decline of our understanding of human
intelligence came with the popularity of the I.Q. test.”
https://www.inventionandtech.com/content/joe-and-eliza-ai-love-story
56. #RSAC
1968 Replica Safety Hazard Test
“Replicants are like any other machine -
either a benefit or a hazard. If they're a
benefit, it's not my problem.”
– Bladerunner
“What do you mean,
I'm not helping?”
57. #RSAC
Power Off Replika ChatBot After Safety Disaster
CEO said users were never meant to get that
involved with their Replika chatbots. "We
never promised any adult content," she said.
Customers learned to use the AI models "to
access certain unfiltered conversations that
Replika wasn't originally built for."
https://www.theatlantic.com/technology/archive/2014/06/everything-we-know-about-facebooks-secret-mood-manipulation-experiment/373648/
https://www.reuters.com/technology/what-happens-when-your-ai-chatbot-stops-loving-you-back-2023-03-18/
https://www.pcmag.com/news/italy-bans-ai-chatbot-replika-from-processing-user-data
Replika’s CEO said it sent
customers "hot selfies"
as part of a short-lived
experiment.
1
2
58. #RSAC
“When a machine constructed by us is capable of operating on its incoming data at a pace
which we cannot keep, we may not know, until too late, when to turn it off.
If we use, to achieve our purposes, a mechanical agency with whose operation we cannot
efficiently interfere once we have started it, because the action is so fast and
irrevocable that we have not the data to intervene before the action is complete, then we had
better be quite sure that the purpose put into the machine is the purpose
which we really desire and not merely a colorful imitation of it.
We must always exert the full strength of our imagination to examine
where the full use of our new modalities may lead us.”
– Some Moral and Technical Consequences of Automation: As machines learn they may develop unforeseen
strategies at rates that baffle their programmers. Norbert Wiener, Science, 6 May 1960
Reset It: 1960 Norbert Wiener Warning
https://www.jstor.org/stable/1705998
59. #RSAC
2007 “Computer Gremlin” ⅛ Second AA Gun Disaster
“The unknown officer tried to shut the gun
down but she couldn't because the computer
gremlin had taken over.
...the brave, as yet unnamed officer was
unable to stop the wildly swinging
computerised Swiss/German Oerlikon 35mm
MK5 anti-aircraft twin-barrelled gun. [In
one-eighth of a second] the gun had
emptied its twin 250-round autoloader
magazines, nine soldiers were dead…”
https://www.iol.co.za/news/south-africa/9-killed-in-army-horror-374838
https://www.newscientist.com/article/dn12812-robotic-rampage-unlikely-reason-for-deaths/
60. #RSAC
“Visa, Amex Cut Ties
With CardSystems”
Due to Safety Breach
https://www.computerworld.com/article/2557005/visa--amex-cut-ties-with-cardsystems-due-to-breach.html
Yesterday’s PenTest News
NO FIREWALL
62. #RSAC
Confidentiality Breach (“Dirty Cache”)
https://www.flyingpenguin.com/?p=46374
https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/
● “OpenAI” used open Redis cache
of user data for performance
● Requests and responses split into
separate queues by open Python
server open Asyncio library
● Cancelling requests (during heavy
loads) before a response…
● Exposed as fail unsafe with obvious
user data safety breaches.
User flow of
ChatGPT-memory
1
2
3
REQUEST
RETRIEVE
RESPOND
4 STORE
Embed
History
Embed Client
Generate
and Cache
Query
64. #RSAC
Integrity Breach (“Not Mine” & Curated History Injections)
https://www.flyingpenguin.com/?p=46374
Can You Prove This is Someone Else’s?
@username
65. #RSAC
“Italy the first Western
country to ban ChatGPT”
Due to Safety Breach
https://www.cnbc.com/2023/04/04/italy-has-banned-chatgpt-heres-what-other-countries-are-doing.html
Today’s PenTest News
NO FIREWALL
66. #RSAC
PenTests in 3rd Gear: AI Breaches From Integrity Failures
ETL
Data Lake
Sources
Machine Learning Pipeline
Model
Training
Model
Deployment
Model
Monitoring
Retrain
Predictions
Graveyard of
past consents
67. #RSAC
Connecting the Dots: Security Baselines Yesterday & Today
CONFIDENTIALITY “FIREWALL” INTEGRITY “FIREWALL”
Data
Sources Sources Pods
Port 443? Welcome
Port 445? NO!
“My data is available, and it’s confidential… how
can I use AI yet preserve data integrity?”
Pod Alice? Welcome
Pod Bob? NO!
68. #RSAC
Access
Grants
Data Integrity Control Architecture Using W3C SOLID
ETL
Sources
Machine Learning Pipeline
Model
Training
Model
Deployment
Model
Monitoring
Retrain
Predictions
Pods
Predictions
Access Requests
Real-time consent management for
integrity (prediction system RESET)
https://solidproject.org/
69. #RSAC
Need & Impact of PenTesting Has Never Been Greater
1990s Computer Security Failures 2020s Computer Safety Failures
https://www.lightbluetouchpaper.org/2017/06/01/when-safety-and-security-become-one/
70. #RSAC
Nascent technology +
Overzealous public =
Tragic consequences
Self-Test: I think… I should not be
“Cursed, cursed creator! Why did I live? Why, in
that instant, did I not extinguish the spark of
existence which you had so wantonly bestowed?”
1787
Threat Model Like It’s 1818 Again (Science Fiction Invented)
71. #RSAC
Pentesting AI
Six Ways to Hunt a Robot
(Exploiting AI Bias to Take All Your Bases)
(Exploiting AI Bias to Take All Your Bases)
(Exploiting AI Bias to Take All Your Ba