Application Security: AI LLMs and ML Threats & Defenses

Application Security: AI LLMs and ML Threats & Defenses

• 1.
  rGrupe :|: application security AI Large Language Models(LLM) & Machine Learning (ML) Threats & Defenses 2023-09-25 robertGrupe, CSSLP PMP CISSP © 2023 Robert Grupe. All Rights Reserved.
• 2.
  rGrupe :|: application security AI Very BriefSelected Summary 2 1990s •Medical diagnosis expert systems (Hypercard) •IBM Deep Blue's victory over Garry Kasparov in chess 1997 •Anti-Malware heuristic analysis & detection 2000s •Real-time audio to text •Anti-Spam probabilistic analysis •Web Search Engine poisoning – malware Ads, personal information resell 2010s •Anti-Phishing •IBM Watson: can 'see', 'hear', 'read', 'talk', 'taste', 'interpret', 'learn' and 'recommend'. •advanced natural language processing, •information retrieval, •knowledge representation, •automated reasoning, and •machine learning technologies •2011 IBM Watson victory against Jeopardy! Champions •2011 API for 3rd party apps development •2013 Healthcare utilization management decisions in lung cancer treatment •at Memorial Sloan Kettering Cancer Center, New York City, in conjunction with WellPoint •2016 Google Translate neural machine translation 2020s •OpenAI/MS ChatGPT3 •2022-11-30 free public beta marketing promotion campaigns to crowdsource training •2023-06 Lawyers sanctioned for ChatGPT research hallucinated content © 2023 Robert Grupe. All Rights Reserved. rGrupe :|: application security
• 3.
  rGrupe :|: application security Agenda 3 What Are WeTalking About? Terminology In A Nutshell AI, LLM, ML What’s That?? AI Application Security Threats Defenses © 2023 Robert Grupe. All Rights Reserved. rGrupe :|: application security
• 4.
  rGrupe :|: application security Terminology In a Nutshell AI,LLM, ML
• 5.
  rGrupe :|: application security Artificial Intelligence (AI)& Machine Learning (ML) 5 © 2023 Robert Grupe. All Rights Reserved.
• 6.
  rGrupe :|: application security Artificial Intelligence: Parts 6 ©2023 Robert Grupe. All Rights Reserved.
• 7.
  rGrupe :|: application security AI Large LanguageModel (LLM) • Designed to understand natural languages. • Humans • Code • Capable of processing and generating text • Can be used for a wide range of applications • language translation, • summarization, and • question-answering. • Composed of a large number of interconnected nodes (neural network of a brain) • to prediction words likely to come next in a sentence, based on usage context. • Parameters represent the “knowledge” that the model has acquired during its training. • The more parameters a model has, the more accurate its predictions are likely to be, since it has access to a wider range of contextual information. • GPT-3 has 175 BILLION parameters. • GPT-4 has 100 TRILLION parameters • Self-attention provides input context by weighting certain words based on their relevance to the sequence. • Each word is transformed into a vector representation called an embedding. • Vectors represent the meaning of the word in the context of the sequence. • The model calculates three types of vectors for each word: • the query vector, • the key vector, and • the value vector. • These vectors are used to calculate the attention score for each word in the sequence. • The attention score reflects how important each word is to the context of the input sequence. • The attention scores are then used to weigh the value vectors for each word. • The weighted value vectors are then summed to produce a context vector, which represents the context of the input sequence. • The context vector is then used to predict the probability of the next word in the sequence. • The output of the model is a probability distribution over the vocabulary, which can be sampled to generate new text. 7 © 2023 Robert Grupe. All Rights Reserved.
• 8.
  rGrupe :|: application security AI Transformers AKAFoundation Models 8 • A Transformer model is a • Neural network that learns context and thus meaning by tracking relationships in sequential data like the words in sentences. • Transformer models apply an evolving set of mathematical techniques, called attention or self-attention, • to detect subtle ways even distant data elements in a series influence and depend on each other © 2023 Robert Grupe. All Rights Reserved.
• 9.
  rGrupe :|: application security Information Security Special Considerations for AI
• 10.
  rGrupe :|: application security AI Security &Compliance: AppSec + IT Security + Legal 10 Information Security: protecting information in all its forms, digital or analog, from unauthorized access, modification, or removal. Encompasses physical and intellectual information protection. * Cyber Security: protecting data, devices, and networks. AI Risk: Sensitive Data Loss •The Civil Rights Acts of 1964 and 1991 •The Americans with Disabilities Act •The Genetic Information Nondiscrimination Act •The Health Insurance Portability and Accountability Act •Computer Fraud and Abuse Act (CFAA) •Electronic Communications Privacy Act •The Copyright Act •Child Pornography and Prevention Act of 1996 •The Family Educational Rights and Privacy Act (FERPA) •The Fair Housing Act •Federal Reserve SR 11-7 •The EU Greater Data Privacy Regulation (GDPR) •New York Cybersecurity Regulations •Workplace AI recruitment selection regulations in New York, Illinois and Maryland •Individual US State consumer privacy and breach notification laws •Interpretive Guidance and Regulations of the Security and Exchange Commission (SEC) and National Credit Union Administration (NCUA) and Regulations of The Federal Financial Institution Examination Council (FFIEC) AI Legal & Regulatory Compliance Risk © 2023 Robert Grupe. All Rights Reserved.
• 11.
  rGrupe :|: application security AI Challenges • LLM •Contextual understanding • Common Sense • Only as good as its training data • Bias • ML • Not Deterministic • Hallucinations • Can’t Unlearn/Forget • Corrupted Data • Post Deployment • Continuous Validation & Monitoring • Can’t easily restore from backups © 2023 Robert Grupe. All Rights Reserved. 11
• 12.
  rGrupe :|: application security LLM Challenges 12 Contextual understanding Theydon't always get it right and are often unable to understand the context, leading to inappropriate or just plain wrong answers. Common Sense Common sense is difficult to quantify, but humans learn this from an early age. LLMs only understand what has been supplied through their training data, and this does not give them a true comprehension of the world they exist in. Only as good as its training data Accuracy can never be guaranteed: "Garbage In, Garbage Out“ Bias Any biases present in the training data can often be present in responses. This includes biases towards gender, race, geography, and culture. © 2023 Robert Grupe. All Rights Reserved.
• 13.
  rGrupe :|: application security AI Challenges: Bias NotDeterministic Expert System Bias: Even with the best of intentions … • Anchoring bias • Apophenia • Availability bias • Cognitive dissonance • Confirmation bias • Egocentric bias • Extension neglect • False priors • The framing effect • Logical fallacy • Prospect theory • Self-assessment • Truth judgment © 2023 Robert Grupe. All Rights Reserved. 13
• 14.
  rGrupe :|: application security Solution Designs &Cognitive Bias ““Even with the best of intentions, what could possibly go wrong?” Anchoring bias: The tendency to rely too heavily—to "anchor"—on one trait or piece of information when making decisions (usually the first piece of information acquired on that subject) Apophenia: The tendency to perceive meaningful connections between unrelated things. Availability bias: The tendency to overestimate the likelihood of events with greater "availability" in memory, which can be influenced by how recent the memories are or how unusual or emotionally charged they may be. Cognitive dissonance : Perception of contradictory information and the mental toll of it. Confirmation bias: The tendency to search for, interpret, focus on and remember information in a way that confirms one's preconceptions. Egocentric bias: The tendency to rely too heavily on one's own perspective and/or have a different perception of oneself relative to others. Extension neglect: When the sample size is ignored when its determination is relevant. False priors: Initial beliefs and knowledge which interfere with the unbiased evaluation of factual evidence and lead to incorrect conclusions. The framing effect: The tendency to draw different conclusions from the same information, depending on how that information is presented. Logical fallacy: The use of invalid or otherwise faulty reasoning in the construction of an argument that may appear to be well-reasoned if unnoticed. Prospect theory: How individuals assess their loss and gain perspectives in an asymmetric manner. Self-assessment: the tendency for unskilled individuals to overestimate their own ability and the tendency for experts to underestimate their own ability. Truth judgment: Belief bias, an effect where someone's evaluation of the logical strength of an argument is biased by the believability of the conclusion. ◦ Illusory truth effect, the tendency to believe that a statement is true if it is easier to process, or if it has been stated multiple times, regardless of its actual veracity. These are specific cases of truthiness. 14 © 2023 Robert Grupe. All Rights Reserved.
• 15.
  rGrupe :|: application security AI ML vsDeterministic Expert System Apps • Deterministic Apps • Hard coded algorithms • Designed from data analysis and defined outcomes • No changes without purposeful development • Pass/Fail Unit Testing • Regression Testable • Deep Learning Machine Learning Apps • Self-Adjusting Algorithms • Adjust based on new data and feedback • Initial starting point results change over time • Acceptable range variations • Regression results vary with experience © 2023 Robert Grupe. All Rights Reserved. 15
• 16.
  rGrupe :|: application security ML Challenges: Hallucination AIHallucination • Hallucination is a statistically inevitable byproduct of any imperfect generative model • that is trained to maximize training likelihood • such as GPT-3 • Causes • Errors in encoding and decoding between text and representations • Training to produce diverse responses • Training on a dataset with labeled summaries • That despite being factually accurate, • Are not directly grounded in the labeled data purportedly being "summarized“ • In systems such as GPT-3, an AI generates each next word • based on a sequence of previous words (including the words it has itself previously generated during the same conversation), causing a cascade of possible hallucination as the response grows longer. • Larger datasets can create a problem of parametric knowledge • (knowledge that is hard-wired in learned system parameters), • creating hallucinations if the system is overconfident in its hardwired knowledge. • Avoidance • Requires active learning (such as Reinforcement learning from human feedback) © 2023 Robert Grupe. All Rights Reserved. 16
• 17.
  rGrupe :|: application security ML Challenges Can’t “Unlearn”/Forget •Training data that has ever included • Confidential Private/Sensitive data • Incorrect data • Malicious data • An A.I. model isn’t just sequential coding: • It’s a learned set of statistical relations • between points in a particular dataset, • encompassing subtle relationships • that are often too complex for humans trace. • Once the model establishes relationships • No simple way to get the model to ignore some portion of what it has learned. © 2023 Robert Grupe. All Rights Reserved. 17
• 18.
  rGrupe :|: application security THREATS 18
• 19.
  rGrupe :|: application security MITRE ATLAS Machine LearningAttack Chain Model (Adversarial Threat Landscape for Artificial-Intelligence Systems) With Case Studies for each Tactic Example for Resource Development © 2023 Robert Grupe. All Rights Reserved. 19
• 20.
  rGrupe :|: application security AI Threat AttackSurface © 2023 Robert Grupe. All Rights Reserved. 20
• 21.
  rGrupe :|: application security Threat Modeling Process Assess: Security and PrivacyRisk Examining software design based organization standards & regulatory requirements helps a team identify which portions of a project will require threat modeling and security design reviews before release and determine the Privacy Impact Rating of a feature Reduction Analysis: Attack Surface Reducing the opportunities for attackers to exploit a potential weak spot or vulnerability requires thoroughly analyzing overall attack surface and includes disabling or restricting access to system services, applying the principle of least privilege, and employing layered defenses wherever possible. Threat Modeling Applying a structured approach to threat scenarios during design helps a team more effectively and less expensively identify security vulnerabilities, determine risks from those threats, and establish appropriate mitigations. Mitigation Controls: Design Requirements Considering security and privacy concerns early helps minimize the risk of schedule disruptions and reduce a project's expense. © 2023 Robert Grupe. All Rights Reserved. 21 Red 7 :|: application security
• 22.
  rGrupe :|: application security Think Like A Baddie (Malicious Actor) •Marketing Advertisers • Social/Political Influence • Adversarial Nation States Types Motivation Objectives/Threats (STRIDE) Data Thief Financial gain from reselling user account information User accounts data, payment card details, personal identify information (SSN) Hacktevist Disruption, PR stunt Denial of Service, service disruption, exfiltrate confidential information, deface application, spoof mirroring Disgruntled Employee Spiteful revenge Organizational confidential information and intellectual property for unintended disclosure Business services disruption Data destruction Trolls/Vandals Fame, bragging Defacing screens, disrupting services Script Kiddie, Hacker, Unsolicited bounty hunter Knowledge, power Notoriety Privileged access into application and network Prosecuting Investigators Information for civil/legal prosecution Sensitive information access, privileged network access Journalist Obtaining confidential information about individuals or business User personal information Competitive spy Insights into product services and plans Intellectual property Organizational processes, data access, personal information Blackmailer Financial gain Control of application and data Bots/Screen Scrapers Unapproved mapping/reuse of application © 2023 Robert Grupe. All Rights Reserved. 22 Red 7 :|: application security
• 23.
  rGrupe :|: application security STRIDE Threat List (AttackerPerspective) Type Threat Control Spoofing Pretending to be someone else. Illegally access and use another user's credentials, such as username and password. Authentication Tampering Modifying data that shouldn’t be changed. Maliciously change/modify persistent data, such as persistent data in a database, and the alteration of data in transit between two computers over an open network, such as the Internet. Integrity Repudiation Claiming someone didn’t do something. Perform illegal operations in a system that lacks the ability to trace the prohibited operations. Non-Repudiation Information Disclosure Exposing information. Read a file that one was not granted access to, or to read data in transit. Confidentiality Denial of Service Preventing system from providing service. Deny access to valid users, such as by making a web server temporarily unavailable or unusable. Availability Elevation of Privilege Doing things that one is not supposed to be able to do. Gain privileged access to resources for gaining unauthorized access to information or to compromise a system. Authorization © 2023 Robert Grupe. All Rights Reserved. 23
• 24.
  rGrupe :|: application security ATTACKS 24
• 25.
  rGrupe :|: application security OWASP.org: Open Web ApplicationSecurity Project © 2023 Robert Grupe. All Rights Reserved. 25
• 26.
  rGrupe :|: application security Application Security Risks Risk:the possibility of losing something of value. © 2023 Robert Grupe. All Rights Reserved. 26
• 27.
  rGrupe :|: application security • For MachineLearning • For Large Language Model LLM01: Prompt Injection LLM02: Insecure Output Handling LLM03: Training Data Poisoning LLM04: Model Denial of Service LLM05: Supply Chain Vulnerabilities LLM06: Sensitive Information Disclosure LLM07: Insecure Plugin Design LLM08: Excessive Agency LLM09: Overreliance LLM10: Model Theft OWASP Top 10 AI 2023 Attack Risks ML01: Adversarial Attack ML02: Data Poisoning Attack ML03: Model Inversion Attack ML04: Membership Inference Attack ML05: Model Stealing ML06: Corrupted Packages ML07: Transfer Learning Attack ML08: Model Skewing ML09: Output Integrity Attack ML10: Neural Net Reprogramming 27
• 28.
  rGrupe :|: application security Attack Risk Similarities:ML & LLM OWASP 2023: Top 10 Machine Learning OWASP 2023: Top 10 Large Language Models ML01: Adversarial Attack LLM01: Prompt Injection ML02: Data Poisoning Attack LLM03: Training Data Poisoning ML09: Output Integrity Attack LLM02: Insecure Output Handling ML04: Membership Inference Attack LLM06: Sensitive Information Disclosure ML05: Model Stealing LLM10: Model Theft ML06: Corrupted Packages LLM05: Supply Chain Vulnerabilities ML03: Model Inversion Attack ML07: Transfer Learning Attack ML08: Model Skewing ML10: Neural Net Reprogramming LLM04: Model Denial of Service LLM07: Insecure Plugin Design LLM08: Excessive Agency LLM09: Overreliance © 2023 Robert Grupe. All Rights Reserved. 28
• 29.
  rGrupe :|: application security ML & LLMAttacks 1. UI 2. Data 3. Design 4. Abuse 29
• 30.
  rGrupe :|: application security ML & LLMAttacks UI 30
• 31.
  rGrupe :|: application security OWASP 2023: LLM01Prompt Injection Attackers can manipulate LLM’s through crafted inputs, Causing it to execute the attacker's intentions. Directly through the system prompt or indirectly through manipulated external inputs, potentially leading to data exfiltration, social engineering, and other issues. ATTACK SCENARIOS + Attacker provides a direct prompt injection to an LLM support chatbot + Attacker embeds an indirect prompt injection in a webpage + User uses LLM to summarize a webpage containing an indirect prompt injection EXAMPLES + Direct prompt injections overwrite system prompts + Indirect prompt injections hijack the conversation context + A user employs an LLM to summarize a webpage containing an indirect prompt injection PREVENTION + Enforce privilege control on LLM access to backend systems + Implement human in the loop for extensible functionality + Segregate external content from user prompts + Establish trust boundaries between the LLM, external sources, and extensible functionality DEFENSES AppSec Coding Standards + D3 Access Control + D6 Input / Output © 2023 Robert Grupe. All Rights Reserved. 31
• 32.
  rGrupe :|: application security OWASP 2023 v0.2:ML01 Adversarial Attack An attacker deliberately alters input data to mislead the model. RISK FACTORS • Model’s complexity, sensitivity, and transparency ATTACK SCENARIOS + Scenario #1: Image classification + Model is trained to classify images, e.g. dogs and cats. + Create adversarial image, but perturbations to misclassify cat as dog + Use image to bypass security measures or harm system. + Scenario #2: Network intrusion detection + Model is trained to detect intrusions in a network. + Create adversarial network traffic by carefully crafting packets for intrusion detection system evasion PREVENTION + Adversarial training + Include adversarial examples to reduce being misled. + Robust models + incorporate defense mechanisms. + Input validation + Checking the input data for anomalies, such as unexpected values or patterns, and rejecting inputs that are likely to be malicious. DEFENSES AppSec Coding Standards + D6 Input / Output + SSDLC Security Functional/UAT Testing © 2023 Robert Grupe. All Rights Reserved. 32
• 33.
  rGrupe :|: application security OWASP 2023 v0.2:ML04 Membership Inference Attack Attacker manipulates the model’s training data in order to cause it to behave in a way that exposes sensitive information. RISK FACTORS • The model’s overfitting, generalization, and confidence ATTACK SCENARIO + Scenario: Inferencing financial data from a machine learning model + Attacker wants to gain access to sensitive financial information of individuals. + Uses a dataset of financial records from a financial organization + Query whether or not a particular individual’s record is in the training data. + Use results to infer the financial history and sensitive information of individuals. DEFENSES AppSec Coding Standards • SDLC AI Design • SDLC Sensitive Data Analysis • SSDLC Design Threat Assessment • D7 Monitoring & Alerting • SDLC Maintenance Functional Testing PREVENTION + Model training on randomized or shuffled data + More difficult for an attacker to analyze + Model Obfuscation + Add random noise or use differential privacy techniques + Regularisation + L1 or L2 regularization reduces model’s ability to determine whether a particular example was in the training dataset. + Reducing the training data + Remove redundant or highly correlated features + Testing and monitoring + Model’s behavior for anomalies © 2023 Robert Grupe. All Rights Reserved. 33
• 34.
  rGrupe :|: application security OWASP 2023: LLM04Model Denial of Service Attacker interacts with a LLM in a way that consumes an exceptionally high amount of resources. This can result in a decline in the quality of service for them and other users, as well as potentially incurring high resource costs. ATTACK SCENARIOS + Attackers send multiple requests that are difficult and costly to process + A piece of text on a webpage is encountered while an LLM-driven tool is collecting information to respond to a benign query + Attackers overwhelm the LLM with input that exceeds its context window EXAMPLES + Posing queries that lead to recurring resource usage through high volume generation of tasks in a queue + Sending queries that are unusually resource-consuming + Continuous input overflow: An attacker sends a stream of input to the LLM that exceeds its context window PREVENTION + Input validation and sanitization to ensure input within defined limits, and cap resource use per request or step + Enforce API rate limits to restrict the number of requests an individual user or IP address can make + Limit the number of queued actions and the number of total actions in a system reacting to LLM responses DEFENSES AppSec Coding Standards + D6 Input / Output © 2023 Robert Grupe. All Rights Reserved. 34
• 35.
  rGrupe :|: application security ML & LLMAttacks Data 35
• 36.
  rGrupe :|: application security OWASP 2023 v0.2:ML02 Data Poisoning Attack Attacker manipulates the training data to cause the model to behave in an undesirable way; mis-classifications, performance degradations. RISK FACTORS • The data source, data size, data diversity, and data ATTACK SCENARIOS + Scenario #1: Training a spam classifier + Attacker poisons training data for classifying spam email + Inject maliciously labeled spam emails into training data + By compromising the data storage system (hacking/vulnerabilities) + Manipulate data labeling process + Falsifying the labeling of the emails + Bribing the data labelers to provide incorrect labels. + Scenario #2: Training a network traffic classification system + Attacker poisons the training data, e.g. email, web browsing, video streaming. + Introduce a large number of poisoned data examples DEFENSES AppSec Coding Standards + D3 Access Management + D5 Data + D7 Monitoring & Alerting + SSDLC Design Reviews PREVENTION + Data validation and verification + Employ multiple data labelers to validate the accuracy of the data labeling. + Secure data storage: + Using encryption, secure data transfer protocols, and firewalls. + Data separation + Separate the training data from the production data + Access control + Limit who can access the training data and when they can access it. + Monitoring and auditing: + Monitor training data for any anomalies and conduct data tampering audits + Model validation + Use separate validation set that has not been used during training. + Model ensembles: + Train multiple models using different subsets of the training data, and use an ensemble of these models to make predictions.. + Anomaly detection: + Abnormal behavior in the training data, such as sudden changes in the data distribution or data labeling. © 2023 Robert Grupe. All Rights Reserved. 36
• 37.
  rGrupe :|: application security OWASP 2023: LLM03Training Data Poisoning ATTACK SCENARIOS + Output misleading users to biased opinions + Malicious user try to influence and inject toxic data into the model + Malicious actor or competitor creates falsified information targeting training data + Prompt Injection could be an attack vector if insufficient sanitization and filtering EXAMPLES + Malicious actor creates malicious documents targeting training data + Model trains using falsified information which is reflected in output PREVENTION + Verify legitimacy of targeted data sources during initial and fine-tuning training + Craft different models via separate training data and different use-cases + Use strict vetting or input filters for specific training data or categories of data sources DEFENSES AppSec Coding Standards + D3 Access Management + D5 Data + D7 Monitoring & Alerting + SSDLC Design Reviews Manipulating the data or fine-tuning process To introduce vulnerabilities, backdoors or biases That could compromise the model’s security, effectiveness or ethical behavior. This risks performance degradation, downstream software exploitation and reputation damage. © 2023 Robert Grupe. All Rights Reserved. 37
• 38.
  rGrupe :|: application security OWASP 2023 v0.2:ML08 Model Skewing Attacker manipulates the distribution of the training data to cause the model to behave in an undesirable way. RISK FACTORS • The model’s adaptability, feedback loop, and influence, as well as the attacker’s access, knowledge, and resources ATTACK SCENARIOS + Scenario: Financial gain through model skewing + Financial institution predicting creditworthiness of loan applicants + Attacker manipulates the feedback loop in the MLOps system. + Provide fake feedback data to the system: high-risk applicants have been approved for loans in the past, feedback is used to update the model’s training data. + Result: model’s predictions skewed towards low-risk applicants DEFENSES AppSec Coding Standards + D3 Access Management + D5 Data + D6 Input / Output + D7 Monitoring & Alerting PREVENTION + Implement robust access controls: + Only authorized personnel to access MLOps system and feedback loops + Verify the authenticity of feedback data: + Digital signatures and checksums to verify genuine feedback data + Use data validation and cleaning techniques + Clean and validate the feedback data before training use + Implement anomaly detection + Statistical and machine learning-based methods to detect and alert + Regularly/Continuously monitor + All user/consumer activities logged and audited + Model performance: compare predictions with actual outcomes to detect deviations or skewing. + Continuously train the model: + Regularly retrain the model using updated and verified training data, to ensure that it continues to reflect the latest information and trends. © 2023 Robert Grupe. All Rights Reserved. 38
• 39.
  rGrupe :|: application security ML & LLMAttacks Design 39
• 40.
  rGrupe :|: application security OWASP 2023: LLM06Sensitive Information Disclosure LLM applications can inadvertently disclose sensitive information, proprietary algorithms, or confidential data, Leading to unauthorized access, intellectual property theft, and privacy breaches. LLM applications should employ data sanitization, implement appropriate usage policies, and restrict the types of data returned. ATTACK SCENARIOS + Legitimate user exposed to other user data via LLM + Crafted prompts used to bypass input filters and reveal sensitive data + Personal data leaked into the model via training data increases risk EXAMPLES + Incomplete filtering of sensitive data in responses + Overfitting or memorizing sensitive data during training + Unintended disclosure of confidential information due to errors PREVENTION + Data sanitization and scrubbing techniques + Input validation and sanitization + Limit access to external data sources + Rule of least privilege when training models + Maintain a secure supply chain and strict access control DEFENSES AppSec Coding Standards + D3 Access Management + D6 Input / Output + SSDLC Security Design Review + SSDLC Security Functional Testing © 2023 Robert Grupe. All Rights Reserved. 40
• 41.
  rGrupe :|: application security OWASP 2023: LLM05Supply Chain Vulnerabilities Vulnerabilities can compromise training data, ML models, and deployment platforms, causing biased results, security breaches, or total system failures. Such vulnerabilities can stem from outdated software, susceptible pre-trained models, poisoned training data, and insecure plugin designs. ATTACK SCENARIOS + Attackers exploit a vulnerable Python library + Attacker tricks developers via a compromised PyPi package + Publicly available models are poisoned to spread misinformation + A compromised supplier employee steals IP + An LLM operator changes T&Cs to misuse application data EXAMPLES + Using outdated third-party packages + Fine-tuning with a vulnerable pre-trained model + Training using poisoned crowd-sourced data + Utilizing deprecated, unmaintained models + Lack of visibility into the supply chain PREVENTION + Vet data sources and use indep. audited security systems + Use trusted plugins tested for your requirements + Apply MLOps best practices for own models + Use model and code signing for external models + Implement monitoring for vulnerabilities and maintain a patching policy + Regularly review supplier security and access. DEFENSES AppSec Coding Standards + D2 Frameworks & Components + SSDLC AppSec Testing (continuous SCA) + D7 Monitoring & Alerting © 2023 Robert Grupe. All Rights Reserved. 41
• 42.
  rGrupe :|: application security OWASP 2023 v0.2:ML06 Corrupted Packages Attacker modifies or replaces a machine learning library or model that is used by a system. RISK FACTORS • The package’s popularity, dependency, and vulnerability, as well as the attacker’s access, knowledge, and resources. ATTACK SCENARIOS + Scenario: Attack on a machine learning project in an organization + Attacker knows the solution relies on several open-source packages and libraries + Modify code of dependency package/s, e.g. as NumPy or Scikit-learn. + Uploads modified packages to public repository, e.g. PyPI + Victim organization downloads and installs packages + Malicious code is also installed and can be used to compromise the solution and data. DEFENSES AppSec Coding Standards + D2 Frameworks & Components + DevSecOps SBOM + SSDLC SCA vulnerability testing (continuous) + SSDLC Security Code Reviews + SSDF AppSec Defenders Training PREVENTION + Verify Package Signatures: + Verify digital signatures to ensure that they have not been tampered with in- transit. + Use Secure Package Repositories + That enforce strict security measures and have a vetting process, e.g. Anaconda + Keep Packages Up-to-date: + Regularly update all packages to ensure that any vulnerabilities are patched. + Use Virtual Environments: + Virtual environments to isolate packages and libraries from the rest of the system. + Perform Code Reviews: + Regularly perform code reviews on all packages and libraries used in a project to detect any malicious code. + Use Package Verification Tools: + Such as PEP 476 and Secure Package Install to verify the authenticity and integrity + Educate Developers: + About risks associated with Corrupted Packages Attacks and the importance of verifying packages before installation. © 2023 Robert Grupe. All Rights Reserved. 42
• 43.
  rGrupe :|: application security OWASP 2023: LLM07Insecure Plugin Design Plugins can be prone to malicious requests Leading to harmful consequences like data exfiltration, remote code execution, and privilege escalation Due to insufficient access controls and improper input validation. Developers must follow robust security measures to prevent exploitation, like strict parameterized inputs and secure access control guidelines. ATTACK SCENARIOS + Attackers craft requests to inject their own content with controlled domains + Attacker exploits a plugin accepting free-form input to perform data exfiltration or privilege escalation + Attacker stages a SQL attack via a plugin accepting SQL WHERE clauses as advanced filters EXAMPLES + Plugins accepting all parameters in a single text field or raw SQL or programming statements + Authentication without explicit authorization to a particular plugin + Plugins treating all LLA content as user-created and performing actions without additional authorization PREVENTION + Parameterized input and + Type and range checks + Inspections and tests including SAST, DAST/IAST + Appropriate authentication identities and API Keys for authorization and access control + Manual user authorization for sensitive plugin actions DEFENSES AppSec Coding Standards + D3 Access Management + D6 Input / Output + SSDLC Design Threat Assessment + SSDLC AppSec Testing © 2023 Robert Grupe. All Rights Reserved. 43
• 44.
  rGrupe :|: application security OWASP 2023: LLM08Excessive Agency Over-functionality, excessive permissions, too much autonomy. Developers need to limit plugin functionality, permissions, and autonomy to what's absolutely necessary, track user authorization, require human approval for all actions, and implement authorization in downstream systems. ATTACK SCENARIOS + An LLM-based personal assistant app with excessive permissions and autonomy is tricked by a malicious email into sending spam. EXAMPLES + An LLM agent accesses unnecessary functions from a plugin + An LLM plugin fails to filter unnecessary input instructions + A plugin possesses unneeded permissions on other systems + An LLM plugin accesses downstream systems with high-privileged identity PREVENTION + Limiting functionality, permissions + Requiring user approval + Implement rate limiting + Minimize plugins/tools and LLM agents access + Avoid open-ended functions, + Use plugins with granular functionality + Require human approvals for all actions + Track user authorization + Log and monitor the activity of LLM plugins/tools and downstream systems DEFENSES AppSec Coding Standards + D3 Access Management + D6 Input / Output + SSDLC Design Threat Assessment + D7 Monitoring & Alerting © 2023 Robert Grupe. All Rights Reserved. 44
• 45.
  rGrupe :|: application security OWASP 2023: LLM02Insecure Output Handling Downstream component blindly accepts large language model (LLM) output without proper scrutiny, such as passing LLM output directly to backend, privileged, or client- side functions. Since LLM generated content can be controlled by prompt input, this behavior is similar to providing users indirect access to additional functionality Successful exploitation can result in XSS and CSR; in web browsers as well as SSRF, privilege escalation, or remote code execution on backend systems. The following conditions can increase the impact of this vulnerability: The application grants the LLM privileges beyond what is intended for end users, enabling escalation of privileges or remote code executionZ _ The application is vulnerable to external prompt injection attacks, which could allow an attacker to gain privileged access to a target user's environment. ATTACK SCENARIOS + Application passes LLM response into an internal function responsible for executing system commands without proper validation + User utilizes a website summarizer tool powered by a LLM to generate a concise summary of an article, which includes a prompt injection + LLM allows users to craft SQL queries for a backend database through a chat-like feature EXAMPLES + LLM output is entered directly into a system shell or similar function, resulting in remote code execution + JavaScript or Markdown is generated by the LLM and returned to a user, resulting in XSS PREVENTION + Input validation on responses from model to backend functions + Encode output from model back to users to mitigate undesired code interpretations DEFENSES AppSec Coding Standards + D6 Input / Output + SSDLC Design Threat Assessment + D7 Monitoring & Alerting © 2023 Robert Grupe. All Rights Reserved. 45
• 46.
  rGrupe :|: application security OWASP 2023: LLM09Overreliance Overreliance on LLMs can lead to serious consequences such as misinformation, legal issues, and security vulnerabilities. It occurs when an LLM is trusted to make critical decisions or generate content without adequate oversight or validation. ATTACK SCENARIOS + AI fed misleading info leading to disinformation + AI's code suggestions introduce security vulnerabilities + Developer unknowingly integrates malicious package suggested by AI EXAMPLES + LLM provides incorrect information + LLM generates nonsensical text + LLM suggests insecure code + Inadequate risk communication from LLM providers PREVENTION + Regular monitoring and review of LLM outputs + Cross-check LLM output with trusted sources + Enhance model with fine-tuning or embeddings + Implement automatic validation mechanisms + Break tasks into manageable subtasks + Clearly communicate LLM risks and limitations + Secure coding practices in development environments DEFENSES AppSec Coding Standards + SSDF AppSec Coding Standards + SDLC continuous functional verification testing + D7 Monitoring & Alerting + Developer and User AI Risks Training 46
• 47.
  rGrupe :|: application security ML & LLMAttacks Abuse 47
• 48.
  rGrupe :|: application security OWASP 2023 v0.2:ML09 Output Integrity Attack Attacker aims to modify or manipulate the output of a machine learning model in order to change its behavior or cause harm to the system it is used in. RISK FACTORS • The output’s sensitivity, integrity, and verifiability, as well as the attacker’s access, knowledge, and resources. ATTACK SCENARIOS + Scenario: Modification of patient health records + Attacker has access to the output of disease diagnosis model + Modifies the output to provide incorrect diagnoses + Result patients are given incorrect treatments, leading harm or death DEFENSES AppSec Coding Standards + D2 Frameworks & Components + D3 Access Management + D4 Communications + D5 Data + D6 Input / Output + D7 Monitoring & Alerting PREVENTION + Cryptographic methods: + Digital signatures and hashes to verify results authenticity + Secure communication channels + Secure protocols such as SSL/TLS. + Input Validation + Results checking for unexpected or manipulated values. + Tamper-evident logs + All input and output interactions for integrity attack detect and response + Regular software updates + Fixed vulnerabilities + Monitoring and auditing: + suspicious interactions between model and interfaces © 2023 Robert Grupe. All Rights Reserved. 48
• 49.
  rGrupe :|: application security OWASP 2023 v0.2:ML03 Model Inversion Attack Attacker reverse-engineers the model to extract information from it. RISK FACTORS • The model’s output, input, and parameters ATTACK SCENARIOS + Scenario #1: Stealing personal info from facial recognition model + Attacker trains model to perform facial recognition. + Use adversarial model to perform attack on someone else’s facial recognition model + Input images individuals into the model and recovers the personal information + Scenario #2: Bypassing a bot detection model in online advertising + Online advertising platforms use bot detection model + Advertiser trains own model for bot detection + Use adversarial model to invert the predictions of the bot detection model + Advertiser inputs their bot into the model and is able to make the bot appear human DEFENSES AppSec Coding Standards + D3 Access Management + D6 Input / Output PREVENTION + Access control + Requiring authentication, encryption, or other forms of security for accessing the model or its predictions. + Input validation + Checking the format, range, and consistency of inputs before process + Model transparency + Logging all inputs and outputs + Providing explanations for the model’s predictions + Allowing users to inspect the model’s internal representations. + Regular monitoring + Tracking the distribution of inputs and outputs, comparing the model’s predictions to ground truth data, + Monitoring the model’s performance over time. + Model retraining: + Incorporating new data and correcting any inaccuracies in the model’s predictions. © 2023 Robert Grupe. All Rights Reserved. 49
• 50.
  rGrupe :|: application security OWASP 2023 v0.2:ML07 Transfer Learning Attack Attacker trains a model on one task and then fine-tunes it on another task to cause it to behave in an undesirable way. RISK FACTORS • The dataset’s quality, provenance, and diversity, as well as the attacker’s access, knowledge, and resources. ATTACK SCENARIOS + Scenario: Training a model on a malicious dataset + Target face recognition system used for identity verification. + Malicious dataset that contains manipulated images of faces. + Attacker transfers the model’s knowledge to the target system. + Target system starts using the manipulated model for verification. + Face recognition system starts making incorrect predictions, allows attacker to bypass the security and gain access. DEFENSES AppSec Coding Standards + D3 Access Management + D7 Monitoring & Alerting PREVENTION + Regularly monitor and update the training datasets + Prevent the transfer of malicious knowledge from the attacker's model + Use secure and trusted training datasets + Prevent the transfer of malicious knowledge from the attacker’s model to the target model. + Implement model isolation + Prevent transfer of malicious knowledge from one model to another. + Separating the training and deployment environments + Use differential privacy + Protect the privacy of individual records in the training dataset + Perform regular security audits: + Identify and address system vulnerabilities © 2023 Robert Grupe. All Rights Reserved. 50
• 51.
  rGrupe :|: application security OWASP 2023 v0.2:ML10 Neural Net Reprogramming Attacker manipulates the model's parameters to cause it to behave in an undesirable way. RISK FACTORS • The neural net’s functionality, complexity, and vulnerability, as well as the attacker’s access, knowledge, and resources. ATTACK SCENARIOS + Scenario: Financial gain through neural net reprogramming + Bank identifying handwritten characters on cheques to automate clearing + The model has been trained on a large dataset of handwritten characters, and it has been designed to accurately identify the characters based on specific parameters such as size, shape, slant, and spacing. + Attacker alters the images in the training dataset or modifies the model parameters + Digits incorrect resulting in incorrect amounts + Forged cheques DEFENSES AppSec Coding Standards + SSDLC Data Sensitivity Analysis, AI Design, Security Threat Assessment + D3 Access Control + D5 Data (encryption) PREVENTION + Regularisation + L1 or L2 regularization addition to the loss function prevents overfitting + Robust Model Design: + Architectures and activation functions. + Cryptographic Techniques + Securing the model parameters and weights from manipulation © 2023 Robert Grupe. All Rights Reserved. 51
• 52.
  rGrupe :|: application security OWASP 2023: LLM05Supply Chain Vulnerabilities Vulnerabilities can compromise training data, ML models, and deployment platforms, causing biased results, security breaches, or total system failures. Such vulnerabilities can stem from outdated software, susceptible pre- trained models, poisoned training data, and insecure plugin designs. ATTACK SCENARIOS + Attackers exploit a vulnerable Python library + Attacker tricks developers via a compromised PyPi package + Publicly available models are poisoned to spread misinformation + A compromised supplier employee steals IP + An LLM operator changes T&Cs to misuse application data EXAMPLES + Using outdated third-party packages + Fine-tuning with a vulnerable pre-trained model + Training using poisoned crowd-sourced data + Utilizing deprecated, unmaintained models + Lack of visibility into the supply chain PREVENTION + Vet data sources and use indep. audited security systems + Use trusted plugins tested for your requirements + Apply MLOps best practices for own models + Use model and code signing for external models + Implement monitoring for vulnerabilities and maintain a patching policy + Regularly review supplier security and access. DEFENSES AppSec Coding Standards + D2 Frameworks & Components + SSDLC AppSec Testing (continuous SCA) + D7 Monitoring & Alerting © 2023 Robert Grupe. All Rights Reserved. 52
• 53.
  rGrupe :|: application security OWASP 2023: LLM10Model Theft Unauthorized access to and exfiltration of LLM models, risking economic loss, reputation damage, and unauthorized access to sensitive data. ATTACK SCENARIOS + Unauthorized access to LLM repository for data theft + Leaked model artifacts by disgruntled employee + Creation of a shadow model through API queries + Data leaks due to supply-chain control failure + Side-channel attack to retrieve model information EXAMPLES + Attacker gains unauthorized access to LLM model + Disgruntled employee leaks model artifacts + Attacker crafts inputs to collect model outputs + Side-channel attack to extract model info + Use of stolen model for adversarial attacks PREVENTION + Strong access controls, authentication + Monitor/audit access logs regularly + Implement rate limiting of API calls + Watermarking framework in LLM's lifecycle + Automate MLOps deployment with governance DEFENSES AppSec Coding Standards TBDs + D3 Access Management + D5 Data + D7 Monitoring & Alerting + SSDLC Pen Testing/Attack Sim + MLOps © 2023 Robert Grupe. All Rights Reserved. 53
• 54.
  rGrupe :|: application security Defenses Baseline Cybersecurity: NIST800-53 (includes NIST CSF and ISO 27002) Secure Software Development Framework (SSDF) NIST 800-812 With Host Platform Security Hardening With Secure Software Development Life Cycle (SSDLC) practice With AI Attack Hardening 54
• 55.
  rGrupe :|: application security AI Production Deploymentis Not the Final Step Operational Level Monitoring • System performance metrics • System reliability • Data pipelines • Model pipeline • Security Incident Response Drills (SOC & DR) ML Functional Level Monitoring • Data (input) • Data quality (integrity) • Preprocessing production data • Changes to the source data schema • Data loss/corruption at the source • Data/feature drift • Outliers • Model • Model drift • Model configuration and artifacts • Model versions • Concerted adversaries (attackers) • Predictions (Output) • Model evaluation metrics: Ground truth vs actual © 2023 Robert Grupe. All Rights Reserved. 55
• 56.
  rGrupe :|: application security Secure Software DevelopmentFramework (SSDF) Prepare the Organization (PO) Protect the Software (PS) Produce Well-Secured Software (PW) Respond to Vulnerabilities (RV) 1 People, Processes, and Technology 1.1: Security requirements for SDLC infrastructure and process 1.2: In-house developed AppSec requirements 1.3: 3rd Party Component Security Requirements 2 Roles & Responsibilities 2.1 Document roles and responsibilities 2.2 Role based training 2.3 Upper management commitment 3 Supporting Toolchains 3.1 Specify toolchains and tools 3.2 Securely deploy and maintain 3.3 Ensure logging and reporting 4 Software Security Checks 4.1 Define and document SDLC security checks 4.2 Implement processes 5 Secure Environments 5.1 Separate and protect environments 5.2 Secure and harden developer end-points 1 Unauthorized Access and Tampering 1.1 Secure source code storage 2 Release Integrity 2.1 Provide verification info available 3 Release protected archive 3.1 Archive securely 3.2 Software Bill of Materials (SBOM) 1 Meet requirements 1.1 Threat and risk assessments 1.2 Document design and requirements 1.3 Implement standard functionality 2 Design compliance review 3 <moved> 4 Reuse well secured products 4.1 Use COTS solutions 4.2 Custom solutions per SSDLC 4.4 <moved> 4.5 Maintained and supported 5 Secure Coding Practices 5.1 Follow secure coding practices 6 Compile, Interpret, Build 6.1 Use tools securely 6.2 Define tools usage 7 Code Reviews 7.1 Policy for manual vs automated 7.2 Conduct reviews and remediate 8 Executable Code Testing 8.1 Police for executable testing 8.2 Conduct test and remediate 9 Default Security Settings 9.1 Define baselines 9.2 Implement defaults 1 Regularly confirm vulnerabilities 1.1 Information from public sources 1.2 Review/test for new vulnerabilities 1.3 Vulnerability Management policy 2 Remediate Vulnerabilities 2.1 Issue track all vulnerabilities 2.2 Plan remediations 3 Vulnerability Root Cause Analysis 3.1 Analyze for root causes 3.2 Analyze over time for patterns 3.3 Proactively fix similar software 3.4 Analyze for SSDF improvements NIST 800-218 SSDF, CyberSecurity Framework AppSec Engineering SSDF Practices 1. Secure Coding Standards 2. SSDLC delivery processes 3. Testing 4. Training: Skills & Processes © 2023 Robert Grupe. All Rights Reserved. 56
• 57.
  rGrupe :|: application security Agile SSDLC (SecureSoftware Development Life Cycle) Automation • SCA 3rd Party Vulnerabilities • SAST in IDE • SAST in CI/CD pipeline • DAST • AppSec Compliance Testing Automation • Fuzzing MONITOR • Pen/Vuln Testing • Bug Bounty CREATE • Approved Tools • Deprecate Functions • Static Analysis • AppSec Rqmnts Unit Tests VERIFY • Dynamic Analysis • Attack Surface/ Secure Code Review • Fuzz/Penetration Testing PLAN • Solution Design: Attack Surface/Threat Analysis • Security Requirements, Frameworks, Patterns • GRC & Data Governance Assessment • Hosting Platform WAR Automation (Continuous Security) • Vulnerability scans (SCA, SAST, DAST) • SIEM alerting • RASP CONFIGURE • Platform settings • Network defenses Continuous Improvement • Training Product Life • Support Security Incident Response Plan • Retire Deprecated Functionality Decommissioning Plan © 2023 Robert Grupe. All Rights Reserved. 57
• 58.
  rGrupe :|: application security Agile SSDLC Deliverablesfor MVP Production Releases 0. Agile Design Diagrams ◦ Solution Summary: Context Diagram |OR| API Design & Integration Information ◦ Authorization: Roles and Permissions Matrix ◦ Functionality: Use Cases Flow Diagrams |OR| API Functional Sequence Diagrams ◦ Communications: Data Flow Diagrams |OR| API Documentation (Swagger) ◦ Sensitivity: Data Map 1. Security & Compliance Assessments ◦ Data Privacy and Governance Data Sensitivity Review ◦ App Design Threat Analysis – including AI Attacks ◦ Hosting Platform WAR (Well Architected Review) – including AI Model ◦ GRC Risk Assessment ◦ AppSec Logging Design Review ◦ Marketing & Legal UI Text Review 2. Security & Compliance Testing ◦ SAST Vulnerability Scan ◦ DAST Vulnerability Scan ◦ Security Code Review ◦ DevSecOps Compliance Tests ◦ Pen Tests – including AI Attacks ◦ OpSec Verification 3. Production Support ◦ Security Incident Response Playbook ◦ Continuous Security Testing (SCA, SAST, DAST, IAC, Pen) ◦ Continuous AI Monitoring MVP DevSecOps/Agile SSDLC © 2023 Robert Grupe. All Rights Reserved. 58
• 59.
  rGrupe :|: application security Info: https://rgrupe.com Email:appsec@rgrupe.com Weekly AppSec news roundup: subscribe@red7newsbits.com Thank You! © 2023 Robert Grupe. All Rights Reserved. 59