Application Security: AI LLMs and ML Threats & Defenses

rGrupe
:|:
application
security
AI
Large Language Models (LLM) &
Machine Learning (ML)
Threats & Defenses
2023-09-25
robertGrupe, CSSLP PMP CISSP
© 2023 Robert Grupe. All Rights Reserved.

rGrupe
:|:
application
security
AI Very Brief Selected Summary
2
1990s
•Medical diagnosis expert systems (Hypercard)
•IBM Deep Blue's victory over Garry Kasparov in chess 1997
•Anti-Malware heuristic analysis & detection
2000s
•Real-time audio to text
•Anti-Spam probabilistic analysis
•Web Search Engine poisoning – malware Ads, personal information resell
2010s
•Anti-Phishing
•IBM Watson: can 'see', 'hear', 'read', 'talk', 'taste', 'interpret', 'learn' and 'recommend'.
•advanced natural language processing,
•information retrieval,
•knowledge representation,
•automated reasoning, and
•machine learning technologies
•2011 IBM Watson victory against Jeopardy! Champions
•2011 API for 3rd party apps development
•2013 Healthcare utilization management decisions in lung cancer treatment
•at Memorial Sloan Kettering Cancer Center, New York City, in conjunction with WellPoint
•2016 Google Translate neural machine translation
2020s
•OpenAI/MS ChatGPT3
•2022-11-30 free public beta marketing promotion campaigns to crowdsource training
•2023-06 Lawyers sanctioned for ChatGPT research hallucinated content
rGrupe
:|:
application
security

rGrupe
:|:
application
security
Agenda
3
What Are We Talking
About?
Terminology In A Nutshell
AI, LLM, ML
What’s
That??
AI Application Security
Threats
Defenses
rGrupe
:|:
application
security

rGrupe
:|:
application
security
Terminology
In a Nutshell
AI, LLM, ML

rGrupe
:|:
application
security
Artificial Intelligence (AI) & Machine Learning (ML)
5

rGrupe
:|:
application
security
Artificial Intelligence: Parts
6

rGrupe
:|:
application
security
AI Large Language Model (LLM)
• Designed to understand natural languages.
• Humans
• Code
• Capable of processing and generating text
• Can be used for a wide range of applications
• language translation,
• summarization, and
• question-answering.
• Composed of a large number of interconnected nodes (neural network of a brain)
• to prediction words likely to come next in a sentence, based on usage context.
• Parameters represent the “knowledge” that the model has acquired during its training.
• The more parameters a model has, the more accurate its predictions are likely to be, since it has access to a
wider range of contextual information.
• GPT-3 has 175 BILLION parameters.
• GPT-4 has 100 TRILLION parameters
• Self-attention provides input context
by weighting certain words based on their relevance to the sequence.
• Each word is transformed into a vector representation called an embedding.
• Vectors represent the meaning of the word in the context of the sequence.
• The model calculates three types of vectors for each word:
• the query vector,
• the key vector, and
• the value vector.
• These vectors are used to calculate the attention score for each word in the sequence.
• The attention score reflects how important each word is to the context of the input sequence.
• The attention scores are then used to weigh the value vectors for each word.
• The weighted value vectors are then summed to produce a context vector, which represents the context of the input
sequence.
• The context vector is then used to predict the probability of the next word in the sequence.
• The output of the model is a probability distribution over the vocabulary, which can be sampled to generate new text.
7

rGrupe
:|:
application
security
AI Transformers AKA Foundation Models
8
• A Transformer model is a
• Neural network that learns
context and thus meaning by
tracking relationships in
sequential data like the words
in sentences.
• Transformer models apply an evolving set
of mathematical techniques,
called attention or self-attention,
• to detect subtle ways even
distant data elements in a
series influence and depend on
each other

rGrupe
:|:
application
security
Information Security
Special
Considerations
for AI

rGrupe
:|:
application
security
AI Security & Compliance:
AppSec + IT Security + Legal
10
Information Security: protecting information in all its forms, digital or analog, from
unauthorized access, modification, or removal.
Encompasses physical and intellectual information protection.
* Cyber Security: protecting data, devices, and networks.
AI Risk: Sensitive Data Loss
•The Civil Rights Acts of 1964 and 1991
•The Americans with Disabilities Act
•The Genetic Information Nondiscrimination Act
•The Health Insurance Portability and Accountability Act
•Computer Fraud and Abuse Act (CFAA)
•Electronic Communications Privacy Act
•The Copyright Act
•Child Pornography and Prevention Act of 1996
•The Family Educational Rights and Privacy Act (FERPA)
•The Fair Housing Act
•Federal Reserve SR 11-7
•The EU Greater Data Privacy Regulation (GDPR)
•New York Cybersecurity Regulations
•Workplace AI recruitment selection regulations in New York, Illinois and Maryland
•Individual US State consumer privacy and breach notification laws
•Interpretive Guidance and Regulations of the Security and Exchange Commission
(SEC) and National Credit Union Administration (NCUA) and Regulations of The
Federal Financial Institution Examination Council (FFIEC)
AI Legal & Regulatory Compliance Risk

rGrupe
:|:
application
security
AI Challenges
• LLM
• Contextual understanding
• Common Sense
• Only as good as its training data
• Bias
• ML
• Not Deterministic
• Hallucinations
• Can’t Unlearn/Forget
• Corrupted Data
• Post Deployment
• Continuous Validation & Monitoring
• Can’t easily restore from backups
11

rGrupe
:|:
application
security
LLM Challenges
12
Contextual understanding
They don't always get it right
and are often unable to
understand the context,
leading to inappropriate or
just plain wrong answers.
Common Sense
Common sense is difficult to
quantify, but humans learn
this from an early age.
LLMs only understand what
has been supplied through
their training data,
and this does not give them a
true comprehension of the
world they exist in.
Only as good as its training
data
Accuracy can never be guaranteed:
"Garbage In, Garbage Out“
Bias
Any biases present in the
training data can often be
present in responses.
This includes biases towards
gender, race, geography, and
culture.

rGrupe
:|:
application
security
AI Challenges: Bias
Not Deterministic Expert System
Bias: Even with the best of intentions …
• Anchoring bias
• Apophenia
• Availability bias
• Cognitive dissonance
• Confirmation bias
• Egocentric bias
• Extension neglect
• False priors
• The framing effect
• Logical fallacy
• Prospect theory
• Self-assessment
• Truth judgment
13

rGrupe
:|:
application
security
Solution Designs & Cognitive Bias
““Even with the best of intentions, what could possibly go wrong?”
Anchoring bias: The tendency to rely too heavily—to "anchor"—on one trait or piece
of information when making decisions (usually the first piece of information acquired
on that subject)
Apophenia: The tendency to perceive meaningful connections between unrelated
things.
Availability bias: The tendency to overestimate the likelihood of events with greater
"availability" in memory, which can be influenced by how recent the memories are or
how unusual or emotionally charged they may be.
Cognitive dissonance : Perception of contradictory information and the mental toll of
it.
Confirmation bias: The tendency to search for, interpret, focus on and remember
information in a way that confirms one's preconceptions.
Egocentric bias: The tendency to rely too heavily on one's own perspective and/or
have a different perception of oneself relative to others.
Extension neglect: When the sample size is ignored when its determination is
relevant.
False priors: Initial beliefs and knowledge which interfere with the unbiased
evaluation of factual evidence and lead to incorrect conclusions.
The framing effect: The tendency to draw different conclusions from the same
information, depending on how that information is presented.
Logical fallacy: The use of invalid or otherwise faulty reasoning in the construction of
an argument that may appear to be well-reasoned if unnoticed.
Prospect theory: How individuals assess their loss and gain perspectives in an
asymmetric manner.
Self-assessment: the tendency for unskilled individuals to overestimate their own
ability and the tendency for experts to underestimate their own ability.
Truth judgment: Belief bias, an effect where someone's evaluation of the logical
strength of an argument is biased by the believability of the conclusion.
◦ Illusory truth effect, the tendency to believe that a statement is true if it is easier to
process, or if it has been stated multiple times, regardless of its actual veracity. These
are specific cases of truthiness.
14

rGrupe
:|:
application
security
AI ML vs Deterministic Expert System Apps
• Deterministic Apps
• Hard coded algorithms
• Designed from data analysis and defined
outcomes
• No changes without purposeful
development
• Pass/Fail Unit Testing
• Regression Testable
• Deep Learning Machine Learning
Apps
• Self-Adjusting Algorithms
• Adjust based on new data and feedback
• Initial starting point results change over
time
• Acceptable range variations
• Regression results vary with experience
15

rGrupe
:|:
application
security
ML Challenges: Hallucination
AI Hallucination
• Hallucination is a statistically inevitable byproduct of any imperfect generative model
• that is trained to maximize training likelihood
• such as GPT-3
• Causes
• Errors in encoding and decoding between text and representations
• Training to produce diverse responses
• Training on a dataset with labeled summaries
• That despite being factually accurate,
• Are not directly grounded in the labeled data purportedly being "summarized“
• In systems such as GPT-3, an AI generates each next word
• based on a sequence of previous words (including the words it has itself previously generated during the same conversation), causing a cascade of possible hallucination as the
response grows longer.
• Larger datasets can create a problem of parametric knowledge
• (knowledge that is hard-wired in learned system parameters),
• creating hallucinations if the system is overconfident in its hardwired knowledge.
• Avoidance
• Requires active learning (such as Reinforcement learning from human feedback)
16

rGrupe
:|:
application
security
ML Challenges
Can’t “Unlearn”/Forget
• Training data that has ever included
• Confidential Private/Sensitive data
• Incorrect data
• Malicious data
• An A.I. model isn’t just sequential coding:
• It’s a learned set of statistical relations
• between points in a particular dataset,
• encompassing subtle relationships
• that are often too complex for humans trace.
• Once the model establishes relationships
• No simple way to get the model to ignore some portion of what it has learned.
17

rGrupe
:|:
application
security
THREATS
18

rGrupe
:|:
application
security
MITRE ATLAS
Machine Learning Attack Chain Model
(Adversarial Threat Landscape for Artificial-Intelligence Systems)
With Case Studies for each Tactic
Example for Resource Development
19

rGrupe
:|:
application
security
AI Threat Attack Surface
20

rGrupe
:|:
application
security
Threat
Modeling
Process
Assess: Security and
Privacy Risk
Examining software design based
organization standards & regulatory
requirements helps a team identify
which portions of a project will
require threat modeling and security
design reviews before release and
determine the Privacy Impact Rating
of a feature
Reduction Analysis:
Attack Surface
Reducing the opportunities for
attackers to exploit a potential weak
spot or vulnerability requires
thoroughly analyzing overall attack
surface and includes disabling or
restricting access to system services,
applying the principle of least
privilege, and employing layered
defenses wherever possible.
Threat Modeling
Applying a structured approach to
threat scenarios during design helps
a team more effectively and less
expensively identify security
vulnerabilities, determine risks from
those threats, and establish
appropriate mitigations.
Mitigation Controls:
Design Requirements
Considering security and privacy
concerns early helps minimize the
risk of schedule disruptions and
reduce a project's expense.
© 2023 Robert Grupe. All Rights Reserved. 21
Red
7
:|:
application
security

rGrupe
:|:
application
security
Think Like A
Baddie
(Malicious
Actor)
• Marketing Advertisers
• Social/Political Influence
• Adversarial Nation States
Types Motivation Objectives/Threats (STRIDE)
Data Thief Financial gain from reselling user account
information
User accounts data, payment card details,
personal identify information (SSN)
Hacktevist Disruption, PR stunt Denial of Service, service disruption, exfiltrate
confidential information, deface application,
spoof mirroring
Disgruntled
Employee
Spiteful revenge Organizational confidential information and
intellectual property for unintended disclosure
Business services disruption
Data destruction
Trolls/Vandals Fame, bragging Defacing screens, disrupting services
Script Kiddie,
Hacker,
Unsolicited
bounty hunter
Knowledge, power
Notoriety
Privileged access into application and network
Prosecuting
Investigators
Information for civil/legal prosecution Sensitive information access, privileged network
access
Journalist Obtaining confidential information about
individuals or business
User personal information
Competitive spy Insights into product services and plans
Intellectual property
Organizational processes, data access, personal
information
Blackmailer Financial gain Control of application and data
Bots/Screen
Scrapers
Unapproved mapping/reuse of application
© 2023 Robert Grupe. All Rights Reserved. 22
Red
7
:|:
application
security

rGrupe
:|:
application
security
STRIDE Threat List
(Attacker Perspective)
Type Threat Control
Spoofing Pretending to be someone else.
Illegally access and use another user's credentials, such as username and password.
Authentication
Tampering Modifying data that shouldn’t be changed.
Maliciously change/modify persistent data, such as persistent data in a database, and
the alteration of data in transit between two computers over an open network, such as
the Internet.
Integrity
Repudiation Claiming someone didn’t do something.
Perform illegal operations in a system that lacks the ability to trace the prohibited
operations.
Non-Repudiation
Information Disclosure Exposing information.
Read a file that one was not granted access to, or to read data in transit.
Confidentiality
Denial of Service Preventing system from providing service.
Deny access to valid users, such as by making a web server temporarily unavailable or
unusable.
Availability
Elevation of Privilege Doing things that one is not supposed to be able to do.
Gain privileged access to resources for gaining unauthorized access to information or to
compromise a system.
Authorization
23

rGrupe
:|:
application
security
ATTACKS
24

rGrupe
:|:
application
security
OWASP.org:
Open Web Application Security Project
25

rGrupe
:|:
application
security
Application Security Risks
Risk: the possibility of losing something of value.
26

rGrupe
:|:
application
security
• For Machine Learning • For Large Language Model
LLM01: Prompt
Injection
LLM02: Insecure
Output Handling
LLM03: Training
Data Poisoning
LLM04: Model
Denial of Service
LLM05: Supply
Chain
Vulnerabilities
LLM06: Sensitive
Information
Disclosure
LLM07: Insecure
Plugin Design
LLM08: Excessive
Agency
LLM09:
Overreliance
LLM10: Model
Theft
OWASP Top 10 AI 2023 Attack Risks
ML01:
Adversarial
Attack
ML02: Data
Poisoning Attack
ML03: Model
Inversion Attack
ML04:
Membership
Inference Attack
ML05: Model
Stealing
ML06: Corrupted
Packages
ML07: Transfer
Learning Attack
ML08: Model
Skewing
ML09: Output
Integrity Attack
ML10: Neural
Net
Reprogramming
27

rGrupe
:|:
application
security
Attack Risk Similarities: ML & LLM
OWASP 2023: Top 10 Machine Learning OWASP 2023: Top 10 Large Language Models
ML01: Adversarial Attack LLM01: Prompt Injection
ML02: Data Poisoning Attack LLM03: Training Data Poisoning
ML09: Output Integrity Attack LLM02: Insecure Output Handling
ML04: Membership Inference Attack LLM06: Sensitive Information Disclosure
ML05: Model Stealing LLM10: Model Theft
ML06: Corrupted Packages LLM05: Supply Chain Vulnerabilities
ML03: Model Inversion Attack
ML07: Transfer Learning Attack
ML08: Model Skewing
ML10: Neural Net Reprogramming
LLM04: Model Denial of Service
LLM07: Insecure Plugin Design
LLM08: Excessive Agency
LLM09: Overreliance
28

rGrupe
:|:
application
security
ML & LLM Attacks
1. UI
2. Data
3. Design
4. Abuse
29

rGrupe
:|:
application
security
ML & LLM Attacks
UI
30

rGrupe
:|:
application
security
OWASP 2023: LLM01 Prompt Injection
Attackers can manipulate LLM’s through crafted inputs,
Causing it to execute the attacker's intentions.
Directly through the system prompt or indirectly through manipulated external
inputs,
potentially leading to data exfiltration, social engineering, and other issues.
ATTACK SCENARIOS
+ Attacker provides a direct prompt injection to an LLM support chatbot
+ Attacker embeds an indirect prompt injection in a webpage
+ User uses LLM to summarize a webpage containing an indirect prompt injection
EXAMPLES
+ Direct prompt injections overwrite system prompts
+ Indirect prompt injections hijack the conversation context
+ A user employs an LLM to summarize a webpage containing an indirect prompt
injection
PREVENTION
+ Enforce privilege control on LLM access to backend systems
+ Implement human in the loop for extensible functionality
+ Segregate external content from user prompts
+ Establish trust boundaries between the LLM, external
sources, and extensible functionality
DEFENSES AppSec Coding Standards
+ D3 Access Control
+ D6 Input / Output
31

rGrupe
:|:
application
security
OWASP 2023 v0.2: ML01 Adversarial Attack
An attacker deliberately alters input data to mislead the model.
RISK FACTORS
• Model’s complexity, sensitivity, and transparency
ATTACK SCENARIOS
+ Scenario #1: Image classification
+ Model is trained to classify images, e.g. dogs and cats.
+ Create adversarial image, but perturbations to misclassify cat as dog
+ Use image to bypass security measures or harm system.
+ Scenario #2: Network intrusion detection
+ Model is trained to detect intrusions in a network.
+ Create adversarial network traffic by carefully crafting packets
for intrusion detection system evasion
PREVENTION
+ Adversarial training
+ Include adversarial examples to reduce being misled.
+ Robust models
+ incorporate defense mechanisms.
+ Input validation
+ Checking the input data for anomalies, such as
unexpected values or patterns, and rejecting inputs
that are likely to be malicious.
+ D6 Input / Output
+ SSDLC Security Functional/UAT Testing
32

rGrupe
:|:
application
security
OWASP 2023 v0.2: ML04 Membership Inference
Attack
Attacker manipulates the model’s training data in order to cause it to behave in
a way that exposes sensitive information.
RISK FACTORS
• The model’s overfitting, generalization, and confidence
ATTACK SCENARIO
+ Scenario: Inferencing financial data from a machine learning model
+ Attacker wants to gain access to sensitive financial information of individuals.
+ Uses a dataset of financial records from a financial organization
+ Query whether or not a particular individual’s record is in the training data.
+ Use results to infer the financial history and sensitive information of individuals.
• SDLC AI Design
• SDLC Sensitive Data Analysis
• SSDLC Design Threat Assessment
• D7 Monitoring & Alerting
• SDLC Maintenance Functional Testing
PREVENTION
+ Model training on randomized or shuffled data
+ More difficult for an attacker to analyze
+ Model Obfuscation
+ Add random noise or use differential privacy techniques
+ Regularisation
+ L1 or L2 regularization reduces model’s ability to determine
whether a particular example was in the training dataset.
+ Reducing the training data
+ Remove redundant or highly correlated features
+ Testing and monitoring
+ Model’s behavior for anomalies
33

rGrupe
:|:
application
security
OWASP 2023: LLM04 Model Denial of Service
Attacker interacts with a LLM
in a way that consumes an exceptionally high amount of resources.
This can result in a decline in the quality of service for them and other users,
as well as potentially incurring high resource costs.
ATTACK SCENARIOS
+ Attackers send multiple requests that are difficult and costly to process
+ A piece of text on a webpage is encountered while an LLM-driven tool is
collecting information to respond to a benign query
+ Attackers overwhelm the LLM with input that exceeds its context window
EXAMPLES
+ Posing queries that lead to recurring resource usage
through high volume generation of tasks in a queue
+ Sending queries that are unusually resource-consuming
+ Continuous input overflow: An attacker sends a stream of input to the LLM
that exceeds its context window
PREVENTION
+ Input validation and sanitization
to ensure input within defined limits, and cap resource use
per request or step
+ Enforce API rate limits to restrict the number of requests an
individual user or IP address can make
+ Limit the number of queued actions and the number of total
actions in a system reacting to LLM responses
+ D6 Input / Output
34

rGrupe
:|:
application
security
ML & LLM Attacks
Data
35

rGrupe
:|:
application
security
OWASP 2023 v0.2: ML02 Data Poisoning Attack
Attacker manipulates the training data to cause the model to behave in an undesirable way;
mis-classifications, performance degradations.
RISK FACTORS
• The data source, data size, data diversity, and data
ATTACK SCENARIOS
+ Scenario #1: Training a spam classifier
+ Attacker poisons training data for classifying spam email
+ Inject maliciously labeled spam emails into training data
+ By compromising the data storage system (hacking/vulnerabilities)
+ Manipulate data labeling process
+ Falsifying the labeling of the emails
+ Bribing the data labelers to provide incorrect labels.
+ Scenario #2: Training a network traffic classification system
+ Attacker poisons the training data, e.g. email, web browsing, video streaming.
+ Introduce a large number of poisoned data examples
+ D3 Access Management
+ D5 Data
+ D7 Monitoring & Alerting
+ SSDLC Design Reviews
PREVENTION
+ Data validation and verification
+ Employ multiple data labelers to validate the accuracy of the data labeling.
+ Secure data storage:
+ Using encryption, secure data transfer protocols, and firewalls.
+ Data separation
+ Separate the training data from the production data
+ Access control
+ Limit who can access the training data and when they can access it.
+ Monitoring and auditing:
+ Monitor training data for any anomalies and conduct data tampering audits
+ Model validation
+ Use separate validation set that has not been used during training.
+ Model ensembles:
+ Train multiple models using different subsets of the training data, and use an
ensemble of these models to make predictions..
+ Anomaly detection:
+ Abnormal behavior in the training data, such as sudden changes in the data
distribution or data labeling.
36

rGrupe
:|:
application
security
OWASP 2023: LLM03 Training Data Poisoning
ATTACK SCENARIOS
+ Output misleading users to biased opinions
+ Malicious user try to influence and inject toxic data into the model
+ Malicious actor or competitor creates falsified information targeting training data
+ Prompt Injection could be an attack vector if insufficient sanitization and filtering
EXAMPLES
+ Malicious actor creates malicious documents targeting training data
+ Model trains using falsified information which is reflected in output
PREVENTION
+ Verify legitimacy of targeted data sources during initial
and fine-tuning training
+ Craft different models via separate training data and
different use-cases
+ Use strict vetting or input filters for specific training
data or categories of data sources
+ D5 Data
+ SSDLC Design Reviews
Manipulating the data or fine-tuning process
To introduce vulnerabilities, backdoors or biases
That could compromise the model’s security, effectiveness or ethical behavior.
This risks performance degradation, downstream software exploitation and
reputation damage.
37

rGrupe
:|:
application
security
OWASP 2023 v0.2: ML08 Model Skewing
Attacker manipulates the distribution of the training data to cause the model to
behave in an undesirable way.
RISK FACTORS
• The model’s adaptability, feedback loop, and influence, as well as the
attacker’s access, knowledge, and resources
ATTACK SCENARIOS
+ Scenario: Financial gain through model skewing
+ Financial institution predicting creditworthiness of loan applicants
+ Attacker manipulates the feedback loop in the MLOps system.
+ Provide fake feedback data to the system:
high-risk applicants have been approved for loans in the past,
feedback is used to update the model’s training data.
+ Result: model’s predictions skewed towards low-risk applicants
+ D5 Data
+ D6 Input / Output
PREVENTION
+ Implement robust access controls:
+ Only authorized personnel to access MLOps system and feedback loops
+ Verify the authenticity of feedback data:
+ Digital signatures and checksums to verify genuine feedback data
+ Use data validation and cleaning techniques
+ Clean and validate the feedback data before training use
+ Implement anomaly detection
+ Statistical and machine learning-based methods to detect and alert
+ Regularly/Continuously monitor
+ All user/consumer activities logged and audited
+ Model performance: compare predictions with actual outcomes
to detect deviations or skewing.
+ Continuously train the model:
+ Regularly retrain the model using updated and verified training data,
to ensure that it continues to reflect the latest information and trends.
38

rGrupe
:|:
application
security
ML & LLM Attacks
Design
39

rGrupe
:|:
application
security
OWASP 2023: LLM06 Sensitive Information
Disclosure
LLM applications can inadvertently disclose sensitive information, proprietary
algorithms, or confidential data,
Leading to unauthorized access, intellectual property theft, and privacy
breaches.
LLM applications should employ data sanitization, implement appropriate usage
policies, and restrict the types of data returned.
ATTACK SCENARIOS
+ Legitimate user exposed to other user data via LLM
+ Crafted prompts used to bypass input filters and reveal sensitive data
+ Personal data leaked into the model via training data increases risk
EXAMPLES
+ Incomplete filtering of sensitive data in responses
+ Overfitting or memorizing sensitive data during training
+ Unintended disclosure of confidential information due to errors
PREVENTION
+ Data sanitization and scrubbing techniques
+ Input validation and sanitization
+ Limit access to external data sources
+ Rule of least privilege when training models
+ Maintain a secure supply chain and strict access control
+ D6 Input / Output
+ SSDLC Security Design Review
+ SSDLC Security Functional Testing
40

rGrupe
:|:
application
security
OWASP 2023: LLM05 Supply Chain Vulnerabilities
Vulnerabilities can compromise training data, ML models, and deployment
platforms,
causing biased results, security breaches, or total system failures.
Such vulnerabilities can stem from outdated software, susceptible pre-trained
models, poisoned training data, and insecure plugin designs.
ATTACK SCENARIOS
+ Attackers exploit a vulnerable Python library
+ Attacker tricks developers via a compromised PyPi package
+ Publicly available models are poisoned to spread misinformation
+ A compromised supplier employee steals IP
+ An LLM operator changes T&Cs to misuse application data
EXAMPLES
+ Using outdated third-party packages
+ Fine-tuning with a vulnerable pre-trained model
+ Training using poisoned crowd-sourced data
+ Utilizing deprecated, unmaintained models
+ Lack of visibility into the supply chain
PREVENTION
+ Vet data sources and use indep. audited security systems
+ Use trusted plugins tested for your requirements
+ Apply MLOps best practices for own models
+ Use model and code signing for external models
+ Implement monitoring for vulnerabilities and maintain a
patching policy
+ Regularly review supplier security and access.
+ D2 Frameworks & Components
+ SSDLC AppSec Testing (continuous SCA)
41

rGrupe
:|:
application
security
OWASP 2023 v0.2: ML06 Corrupted Packages
Attacker modifies or replaces a machine learning library or model that is used
by a system.
RISK FACTORS
• The package’s popularity, dependency, and vulnerability, as well as the
attacker’s access, knowledge, and resources.
ATTACK SCENARIOS
+ Scenario: Attack on a machine learning project in an organization
+ Attacker knows the solution relies on several open-source packages and libraries
+ Modify code of dependency package/s, e.g. as NumPy or Scikit-learn.
+ Uploads modified packages to public repository, e.g. PyPI
+ Victim organization downloads and installs packages
+ Malicious code is also installed and can be used to compromise the solution and data.
+ DevSecOps SBOM
+ SSDLC SCA vulnerability testing (continuous)
+ SSDLC Security Code Reviews
+ SSDF AppSec Defenders Training
PREVENTION
+ Verify Package Signatures:
+ Verify digital signatures to ensure that they have not been tampered with in-
transit.
+ Use Secure Package Repositories
+ That enforce strict security measures and have a vetting process, e.g.
Anaconda
+ Keep Packages Up-to-date:
+ Regularly update all packages to ensure that any vulnerabilities are
patched.
+ Use Virtual Environments:
+ Virtual environments to isolate packages and libraries from the rest of the
system.
+ Perform Code Reviews:
+ Regularly perform code reviews on all packages and libraries used in a
project to detect any malicious code.
+ Use Package Verification Tools:
+ Such as PEP 476 and Secure Package Install to verify the authenticity and
integrity
+ Educate Developers:
+ About risks associated with Corrupted Packages Attacks
and the importance of verifying packages before installation.
42

rGrupe
:|:
application
security
OWASP 2023: LLM07 Insecure Plugin Design
Plugins can be prone to malicious requests
Leading to harmful consequences like data exfiltration, remote code execution,
and privilege escalation
Due to insufficient access controls and improper input validation.
Developers must follow robust security measures to prevent exploitation, like
strict parameterized inputs and secure access control guidelines.
ATTACK SCENARIOS
+ Attackers craft requests to inject their own content with controlled domains
+ Attacker exploits a plugin accepting free-form input to perform data exfiltration or
privilege escalation
+ Attacker stages a SQL attack via a plugin accepting SQL WHERE clauses as
advanced filters
EXAMPLES
+ Plugins accepting all parameters in a single text field
or raw SQL or programming statements
+ Authentication without explicit authorization to a particular plugin
+ Plugins treating all LLA content as user-created and performing actions without
additional authorization
PREVENTION
+ Parameterized input and
+ Type and range checks
+ Inspections and tests including SAST, DAST/IAST
+ Appropriate authentication identities and API Keys for
authorization and access control
+ Manual user authorization for sensitive plugin actions
+ D6 Input / Output
+ SSDLC Design Threat Assessment
+ SSDLC AppSec Testing
43

rGrupe
:|:
application
security
OWASP 2023: LLM08 Excessive Agency
Over-functionality, excessive permissions, too much autonomy.
Developers need to limit plugin functionality, permissions, and autonomy
to what's absolutely necessary, track user authorization, require human
approval for all actions, and implement authorization in downstream systems.
ATTACK SCENARIOS
+ An LLM-based personal assistant app with excessive permissions and autonomy
is tricked by a malicious email into sending spam.
EXAMPLES
+ An LLM agent accesses unnecessary functions from a plugin
+ An LLM plugin fails to filter unnecessary input instructions
+ A plugin possesses unneeded permissions on other systems
+ An LLM plugin accesses downstream systems with high-privileged identity
PREVENTION
+ Limiting functionality, permissions
+ Requiring user approval
+ Implement rate limiting
+ Minimize plugins/tools and LLM agents access
+ Avoid open-ended functions,
+ Use plugins with granular functionality
+ Require human approvals for all actions
+ Track user authorization
+ Log and monitor the activity of LLM plugins/tools and downstream
systems
+ D6 Input / Output
44

rGrupe
:|:
application
security
OWASP 2023: LLM02 Insecure Output Handling
Downstream component blindly accepts large language model (LLM) output without
proper scrutiny, such as passing LLM output directly to backend, privileged, or client-
side functions.
Since LLM generated content can be controlled by prompt input, this behavior is
similar to providing users indirect access to additional functionality
Successful exploitation can result in XSS and CSR; in web browsers as well as SSRF,
privilege escalation, or remote code execution on backend systems.
The following conditions can increase the impact of this vulnerability:
The application grants the LLM privileges beyond what is intended for end users,
enabling escalation of privileges or remote code executionZ _ The application is
vulnerable to external prompt injection attacks, which could allow an attacker to
gain privileged access to a target user's environment.
ATTACK SCENARIOS
+ Application passes LLM response into an internal function responsible for
executing system commands without proper validation
+ User utilizes a website summarizer tool powered by a LLM
to generate a concise summary of an article, which includes a prompt injection
+ LLM allows users to craft SQL queries for a backend database
through a chat-like feature
EXAMPLES
+ LLM output is entered directly into a system shell or similar function, resulting in
remote code execution
+ JavaScript or Markdown is generated by the LLM and returned to a user,
resulting in XSS
PREVENTION
+ Input validation on responses from model to backend
functions
+ Encode output from model back to users
to mitigate undesired code interpretations
+ D6 Input / Output
45

rGrupe
:|:
application
security
OWASP 2023: LLM09 Overreliance
Overreliance on LLMs can lead to serious consequences such as
misinformation, legal issues, and security vulnerabilities.
It occurs when an LLM is trusted to make critical decisions or generate
content without adequate oversight or validation.
ATTACK SCENARIOS
+ AI fed misleading info leading to disinformation
+ AI's code suggestions introduce security vulnerabilities
+ Developer unknowingly integrates malicious package suggested by AI
EXAMPLES
+ LLM provides incorrect information
+ LLM generates nonsensical text
+ LLM suggests insecure code
+ Inadequate risk communication from LLM providers
PREVENTION
+ Regular monitoring and review of LLM outputs
+ Cross-check LLM output with trusted sources
+ Enhance model with fine-tuning or embeddings
+ Implement automatic validation mechanisms
+ Break tasks into manageable subtasks
+ Clearly communicate LLM risks and limitations
+ Secure coding practices in development environments
+ SSDF AppSec Coding Standards
+ SDLC continuous functional verification testing
+ Developer and User AI Risks Training
46

rGrupe
:|:
application
security
ML & LLM Attacks
Abuse
47

rGrupe
:|:
application
security
OWASP 2023 v0.2: ML09 Output Integrity Attack
Attacker aims to modify or manipulate the output of a machine learning model
in order to change its behavior or cause harm to the system it is used in.
RISK FACTORS
• The output’s sensitivity, integrity, and verifiability, as well as the attacker’s
access, knowledge, and resources.
ATTACK SCENARIOS
+ Scenario: Modification of patient health records
+ Attacker has access to the output of disease diagnosis model
+ Modifies the output to provide incorrect diagnoses
+ Result patients are given incorrect treatments, leading harm or death
+ D4 Communications
+ D5 Data
+ D6 Input / Output
PREVENTION
+ Cryptographic methods:
+ Digital signatures and hashes to verify results authenticity
+ Secure communication channels
+ Secure protocols such as SSL/TLS.
+ Input Validation
+ Results checking for unexpected or manipulated values.
+ Tamper-evident logs
+ All input and output interactions for integrity attack detect and
response
+ Regular software updates
+ Fixed vulnerabilities
+ Monitoring and auditing:
+ suspicious interactions between model and interfaces
48

rGrupe
:|:
application
security
OWASP 2023 v0.2: ML03 Model Inversion Attack
Attacker reverse-engineers the model to extract information from it.
RISK FACTORS
• The model’s output, input, and parameters
ATTACK SCENARIOS
+ Scenario #1: Stealing personal info from facial recognition model
+ Attacker trains model to perform facial recognition.
+ Use adversarial model to perform attack on someone else’s facial recognition model
+ Input images individuals into the model and recovers the personal information
+ Scenario #2: Bypassing a bot detection model in online advertising
+ Online advertising platforms use bot detection model
+ Advertiser trains own model for bot detection
+ Use adversarial model to invert the predictions of the bot detection model
+ Advertiser inputs their bot into the model and is able to make the bot appear human
+ D6 Input / Output
PREVENTION
+ Access control
+ Requiring authentication, encryption, or other forms of security
for accessing the model or its predictions.
+ Input validation
+ Checking the format, range, and consistency of inputs before
process
+ Model transparency
+ Logging all inputs and outputs
+ Providing explanations for the model’s predictions
+ Allowing users to inspect the model’s internal representations.
+ Regular monitoring
+ Tracking the distribution of inputs and outputs, comparing the
model’s predictions to ground truth data,
+ Monitoring the model’s performance over time.
+ Model retraining:
+ Incorporating new data and correcting any inaccuracies in the
model’s predictions.
49

rGrupe
:|:
application
security
OWASP 2023 v0.2: ML07 Transfer Learning Attack
Attacker trains a model on one task and then fine-tunes it on another task to
cause it to behave in an undesirable way.
RISK FACTORS
• The dataset’s quality, provenance, and diversity, as well as the attacker’s
access, knowledge, and resources.
ATTACK SCENARIOS
+ Scenario: Training a model on a malicious dataset
+ Target face recognition system used for identity verification.
+ Malicious dataset that contains manipulated images of faces.
+ Attacker transfers the model’s knowledge to the target system.
+ Target system starts using the manipulated model for verification.
+ Face recognition system starts making incorrect predictions, allows
attacker to bypass the security and gain access.
PREVENTION
+ Regularly monitor and update the training datasets
+ Prevent the transfer of malicious knowledge from the attacker's model
+ Use secure and trusted training datasets
+ Prevent the transfer of malicious knowledge from the attacker’s
model to the target model.
+ Implement model isolation
+ Prevent transfer of malicious knowledge from one model to another.
+ Separating the training and deployment environments
+ Use differential privacy
+ Protect the privacy of individual records in the training dataset
+ Perform regular security audits:
+ Identify and address system vulnerabilities
50

rGrupe
:|:
application
security
OWASP 2023 v0.2: ML10 Neural Net
Reprogramming
Attacker manipulates the model's parameters to cause it to behave in an
undesirable way.
RISK FACTORS
• The neural net’s functionality, complexity, and vulnerability, as well as the
attacker’s access, knowledge, and resources.
ATTACK SCENARIOS
+ Scenario: Financial gain through neural net reprogramming
+ Bank identifying handwritten characters on cheques to automate clearing
+ The model has been trained on a large dataset of handwritten characters,
and it has been designed to accurately identify the characters based on
specific parameters such as size, shape, slant, and spacing.
+ Attacker alters the images in the training dataset or modifies the model
parameters
+ Digits incorrect resulting in incorrect amounts
+ Forged cheques
+ SSDLC Data Sensitivity Analysis, AI Design, Security Threat Assessment
+ D3 Access Control
+ D5 Data (encryption)
PREVENTION
+ Regularisation
+ L1 or L2 regularization addition to the loss function prevents
overfitting
+ Robust Model Design:
+ Architectures and activation functions.
+ Cryptographic Techniques
+ Securing the model parameters and weights from manipulation
51

rGrupe
:|:
application
security
OWASP 2023: LLM05 Supply Chain Vulnerabilities
Vulnerabilities can compromise training data, ML models, and
deployment platforms,
causing biased results, security breaches, or total system failures.
Such vulnerabilities can stem from outdated software, susceptible pre-
trained models, poisoned training data, and insecure plugin designs.
ATTACK SCENARIOS
+ Attackers exploit a vulnerable Python library
+ Attacker tricks developers via a compromised PyPi package
+ Publicly available models are poisoned to spread misinformation
+ A compromised supplier employee steals IP
+ An LLM operator changes T&Cs to misuse application data
EXAMPLES
+ Using outdated third-party packages
+ Fine-tuning with a vulnerable pre-trained model
+ Training using poisoned crowd-sourced data
+ Utilizing deprecated, unmaintained models
+ Lack of visibility into the supply chain
PREVENTION
+ Vet data sources and use indep. audited security systems
+ Use trusted plugins tested for your requirements
+ Apply MLOps best practices for own models
+ Use model and code signing for external models
+ Implement monitoring for vulnerabilities and maintain a
patching policy
+ Regularly review supplier security and access.
+ SSDLC AppSec Testing (continuous SCA)
52

rGrupe
:|:
application
security
OWASP 2023: LLM10 Model Theft
Unauthorized access to and exfiltration of LLM models,
risking economic loss, reputation damage, and
unauthorized access to sensitive data.
ATTACK SCENARIOS
+ Unauthorized access to LLM repository for data theft
+ Leaked model artifacts by disgruntled employee
+ Creation of a shadow model through API queries
+ Data leaks due to supply-chain control failure
+ Side-channel attack to retrieve model information
EXAMPLES
+ Attacker gains unauthorized access to LLM model
+ Disgruntled employee leaks model artifacts
+ Attacker crafts inputs to collect model outputs
+ Side-channel attack to extract model info
+ Use of stolen model for adversarial attacks
PREVENTION
+ Strong access controls, authentication
+ Monitor/audit access logs regularly
+ Implement rate limiting of API calls
+ Watermarking framework in LLM's lifecycle
+ Automate MLOps deployment with governance
TBDs
+ D5 Data
+ SSDLC Pen Testing/Attack Sim
+ MLOps
53

rGrupe
:|:
application
security
Defenses
Baseline Cybersecurity: NIST 800-53 (includes NIST CSF and ISO 27002)
Secure Software Development Framework (SSDF) NIST 800-812
With Host Platform Security Hardening
With Secure Software Development Life Cycle (SSDLC) practice
With AI Attack Hardening
54

rGrupe
:|:
application
security
AI Production Deployment is Not the Final Step
Operational Level Monitoring
• System performance metrics
• System reliability
• Data pipelines
• Model pipeline
• Security Incident Response Drills (SOC & DR)
ML Functional Level Monitoring
• Data (input)
• Data quality (integrity)
• Preprocessing production data
• Changes to the source data schema
• Data loss/corruption at the source
• Data/feature drift
• Outliers
• Model
• Model drift
• Model configuration and artifacts
• Model versions
• Concerted adversaries (attackers)
• Predictions (Output)
• Model evaluation metrics: Ground truth vs actual
55

rGrupe
:|:
application
security
Secure Software Development Framework (SSDF)
Prepare the Organization (PO) Protect the Software (PS) Produce Well-Secured Software (PW) Respond to Vulnerabilities (RV)
1 People, Processes, and Technology
1.1: Security requirements for SDLC infrastructure and
process
1.2: In-house developed AppSec requirements
1.3: 3rd Party Component Security Requirements
2 Roles & Responsibilities
2.1 Document roles and responsibilities
2.2 Role based training
2.3 Upper management commitment
3 Supporting Toolchains
3.1 Specify toolchains and tools
3.2 Securely deploy and maintain
3.3 Ensure logging and reporting
4 Software Security Checks
4.1 Define and document SDLC security checks
4.2 Implement processes
5 Secure Environments
5.1 Separate and protect environments
5.2 Secure and harden developer end-points
1 Unauthorized Access and Tampering
1.1 Secure source code storage
2 Release Integrity
2.1 Provide verification info available
3 Release protected archive
3.1 Archive securely
3.2 Software Bill of Materials (SBOM)
1 Meet requirements
1.1 Threat and risk assessments
1.2 Document design and requirements
1.3 Implement standard functionality
2 Design compliance review
3 <moved>
4 Reuse well secured products
4.1 Use COTS solutions
4.2 Custom solutions per SSDLC
4.4 <moved>
4.5 Maintained and supported
5 Secure Coding Practices
5.1 Follow secure coding practices
6 Compile, Interpret, Build
6.1 Use tools securely
6.2 Define tools usage
7 Code Reviews
7.1 Policy for manual vs automated
7.2 Conduct reviews and remediate
8 Executable Code Testing
8.1 Police for executable testing
8.2 Conduct test and remediate
9 Default Security Settings
9.1 Define baselines
9.2 Implement defaults
1 Regularly confirm vulnerabilities
1.1 Information from public sources
1.2 Review/test for new vulnerabilities
1.3 Vulnerability Management policy
2 Remediate Vulnerabilities
2.1 Issue track all vulnerabilities
2.2 Plan remediations
3 Vulnerability Root Cause Analysis
3.1 Analyze for root causes
3.2 Analyze over time for patterns
3.3 Proactively fix similar software
3.4 Analyze for SSDF improvements
NIST 800-218 SSDF, CyberSecurity Framework
AppSec Engineering SSDF Practices
1. Secure Coding Standards
2. SSDLC delivery processes
3. Testing
4. Training: Skills & Processes
56

rGrupe
:|:
application
security
Agile SSDLC (Secure Software Development Life Cycle)
Automation
• SCA 3rd Party Vulnerabilities
• SAST in IDE
• SAST in CI/CD pipeline
• DAST
• AppSec Compliance Testing
Automation
• Fuzzing
MONITOR
• Pen/Vuln Testing
• Bug Bounty
CREATE
• Approved Tools
• Deprecate Functions
• Static Analysis
• AppSec Rqmnts Unit Tests
VERIFY
• Dynamic Analysis
• Attack Surface/
Secure Code Review
• Fuzz/Penetration Testing
PLAN
• Solution Design: Attack Surface/Threat Analysis
• Security Requirements, Frameworks, Patterns
• GRC & Data Governance Assessment
• Hosting Platform WAR
Automation (Continuous Security)
• Vulnerability scans (SCA, SAST, DAST)
• SIEM alerting
• RASP
CONFIGURE
• Platform settings
• Network defenses
Continuous Improvement
• Training
Product Life
• Support
Security Incident Response Plan
• Retire Deprecated Functionality
Decommissioning Plan
57

rGrupe
:|:
application
security
Agile SSDLC Deliverables for MVP Production Releases
0. Agile Design Diagrams
◦ Solution Summary: Context Diagram |OR| API Design & Integration Information
◦ Authorization: Roles and Permissions Matrix
◦ Functionality: Use Cases Flow Diagrams |OR| API Functional Sequence Diagrams
◦ Communications: Data Flow Diagrams |OR| API Documentation (Swagger)
◦ Sensitivity: Data Map
1. Security & Compliance Assessments
◦ Data Privacy and Governance Data Sensitivity Review
◦ App Design Threat Analysis – including AI Attacks
◦ Hosting Platform WAR (Well Architected Review) – including AI Model
◦ GRC Risk Assessment
◦ AppSec Logging Design Review
◦ Marketing & Legal UI Text Review
2. Security & Compliance Testing
◦ SAST Vulnerability Scan
◦ DAST Vulnerability Scan
◦ Security Code Review
◦ DevSecOps Compliance Tests
◦ Pen Tests – including AI Attacks
◦ OpSec Verification
3. Production Support
◦ Security Incident Response Playbook
◦ Continuous Security Testing (SCA, SAST, DAST, IAC, Pen)
◦ Continuous AI Monitoring
MVP
DevSecOps/Agile SSDLC
58

rGrupe
:|:
application
security
Info: https://rgrupe.com Email: appsec@rgrupe.com
Weekly AppSec news
roundup:
subscribe@red7newsbits.com
Thank You!
59

Application Security: AI LLMs and ML Threats & Defenses

Application Security: AI LLMs and ML Threats & Defenses

Recommended

More Related Content

What's hot

What's hot(20)

Similar to Application Security: AI LLMs and ML Threats & Defenses

Similar to Application Security: AI LLMs and ML Threats & Defenses(20)

More from Robert Grupe, CSSLP CISSP PE PMP

More from Robert Grupe, CSSLP CISSP PE PMP(19)

Recently uploaded

Recently uploaded(20)

Application Security: AI LLMs and ML Threats & Defenses

Editor's Notes