I present a Tale of Two AIs. First, we'll delve into the intricacies of Gen AI and then discuss the unique security risks posed by Gen AI, including adversarial attacks, unintended biases, and emergent behaviors. We'll then explore how Gen AI can be utilized to strengthen security defenses by automating vulnerability detection, assisting in threat analysis, and even generating secure code. This talk will equip you with the knowledge to navigate the complex landscape of Gen AI security by building an adoption friendly responsible AI program at your organization. Join us as we explore the glitches and the guardians, and discover how to leverage the power of Gen AI to secure your applications in the future.

1. The Duality of Gen AI: From
Glitches to Guardians
Trupti Shiralkar

2. About me
●Software & Product security professional
- MS In Security Engineering, Johns Hopkins University
- Founder, TrueNil.io
- Previously led ProdSec at Datadog, Illumio, Amazon, Q2, ATSEC & HP
●Yoga Alliance Certified Instructor(200 hours)
- Breathing exercises
- Meditation
●When I am not doing security
- Public speaking (30+ conferences)
- Mindfulness promoter
- Paint
- Community building
LinkedIn - /trupti-shiralkar-0a085a8/
Email - s.trupts@gmail.com

3. Gratitude
Special Thanks
• Prashant Venkatesh
• OWASP SV chapter
organizers
My Machine Learning
Support group
• Pallavi Tyagi
• Abraham Kang
• Satish Narale
NIST Trustworthy & Responsible AI team
• Apostol Vassilev
• Alina Oprea
• Alie Fordyce

4. Agenda
• Why learn about AI, Gen AI & securing them?
• The Basics
• Go Purple - Adversarial attacks & Defenses
• Leverage Gen AI to augment security solutions
and operations
• Get started with your Gen AI Security Journey

5. Astronomical Growth in Gen AI
• 80% enterprises are adopting LLMs
by 2026
• EU law enforcement predicts 90%
of online content will be AI-made.
• Open AI's platform fuels a booming
app market with 2 million
developers.
• Generative AI is set for a massive
$180 billion growth in eight years
• 5000+ startups in Gen AI space

6. Why it is important to learn Gen AI
• Complex LLM Life cycle
• Architectural choices
matter
• Intricate data pathways
• Attack surface is evolving
• Use cases vary based on
problem statement and
the industry vertical
Evaluation
LLM
Definition
Data
Collection
Fine
Tuning
Data
Cleaning
Model
Architecture
Pre
Training
Deployment
Feedback
& Iteration
Monitoring
Fig 1: Basic LLM building Life cycle

7. Fig 2: Fine tuning LLMs Ref: Debmalya Biswas

8. Back to basics …
Fig 3: Overview of AI

9. LLM ~Gen AI +NLP
Artificial Intelligence (AI): Computers or machines that can think and learn like
humans
Machine Learning (ML): Teaching computers to learn from data, kind of like how we
learn from experience
Deep Learning (DL): A part of machine learning, where computers use "neural
networks" to learn, inspired by our brain's structure
Natural Language Processing (NLP): Making computers understand and talk in
human language
Predictive AI: Leverages historic data for future trend forecasting
Generative AI (Gen AI): Application of AI that is cable of generating text, images,
videos based on prompt
Large Language Model(LLM): AI model that can understand and generate human
like text
NLP GEN AI
LLM
Fig 4: LLM~ intersection
of NLP & Gen AI

10. Basics of Gen AI
• Tokenization: Process of converting raw text into smaller units
such as words or subwords so that model can understand during
the training phase.
• Embeddings: Converting words or tokens into high-dimensional
vectors, so that LLMs can learn the relationships .
• Transformer Architecture: Utilizing transformer-based neural
network architectures for processing and generating text data
efficiently.

11. Supervised Machine Learning
Data Gathering
and
understanding
(X)
Prediction
(Y)
Feature
Engineering
Feature
Selection
Test Data
(input X)
Model
Training
(multiple
sets of (x,y)
Model
Testing
Data Cleaning & formatting
Modeling
Known
Questions
Known
Answers
Unknown
Questions
Answers
f(x)
Data engineering
Fig 5: Overview of supervised ML

12. Deep Learning
Deep Neural Net Prediction
Known
Questions
Known
Answers
Un-Known
Questions
Answers
Data Gathering
and
understanding
Prediction
Feature
Engineering
Feature
Selection
Test Data
Model
Training
Model
Testing
Data Cleaning
Modeling
Data Gathering
and
understanding
Fig 6: Overview of Deep Learning

13. Transformer
It should get us same input
as output
Input Input
Encoder
Neural Net
Decoder
Neural Net
Fig 6: Transformer

14. Transformer
Known Questions Known Answers
Unknown
Questions
Answers
Known-Sentence
part-1
Known-Sentence
part-2
Known-Sentence
part-1
Un Known- part-2
Encoder
Neural Net
Decoder
Neural Net
Fig 7: Transformer architecture

15. Basics of Gen AI
• Attention Mechanism: Allowing the model to focus on relevant
parts of the input text when generating responses or making
predictions during training and inference phase.
• Pre-training: Training the model on a large corpus of text data to
learn general language patterns and structures.
• Fine-tuning: Fine-tuning the pre-trained model on specific
downstream tasks to adapt it to particular applications, such as
text generation, translation, or sentiment analysis.
• Decoding Mechanism: Generating coherent and contextually
relevant text based on learned patterns and input prompts

16. Adversarial attacks & defenses

17. Adversarial Machine Learning
Attack techniques used to fool or misguide a model with malicious input
Data
Public
API
Infrastructure
Model
Data protection, privacy attacks
Classic API/web
attacks, prompt
injections
Firewalls, IDS,
encryption mechanisms
Access controls
Tampering of model
architectures,
parameters, I/O
Ethics
Weaponization of
LLMs, unethical use
Bias, misinformation
Fig 8: AML attack surface
Hallucinations

18. Resource
Control
Source Data
Control
Training
Data
Control
Query
Access
Availability
Indirect
Prompt Injection
Denial of Service
Data poisoning
Increased Computation
Prompt Injection
Resource
Control
Source Data
Control
Training
Data
Control
Query
Access
Abuse
Indirect
Prompt Injection
Prompt Injection
Resource
Control
Source Data
Control
Training
Data
Control
Query
Access
Integrity
Prompt Injection
Indirect
Prompt Injection
Misaligned
Input
Targeting Poisoning
Data Poisoning
Backdoor Poisoning
R
e
s
o
u
r
c
e
C
o
n
t
r
o
l
Q
u
e
r
y
A
c
c
e
s
s
Model
Privacy
Data
Q
u
e
r
y
A
c
c
e
s
s
R
e
s
o
u
r
c
e
C
o
n
t
r
o
l
Prompt Injection
Prompt Extraction
Backdoor
Poisoning
Indirect
Prompt Injection
Unauthorized
Disclosure
Training Data
Attacks Data Extraction
Membership Inference
Information
Gathering
Fig 9: NIST’s Taxonomy of
attacks on Gen AI system

19. Red teaming
perspective ..
Ref: MITRE ATLAS Metrix

20. Ref: MITRE ATLAS Metrix
Red teaming
perspective ..

21. Augment security solutions and
operations

22. AppSec Static Code Analysis
Automated Security Incident
Response
Vulnerability discovery,
correlation and
prioritization
03
04
02
01
Security content,
awareness
Training Creation

23. Social Engineering detection
Malware Analysis & Detection
Vulnerability discovery,
correlation and
prioritization
07
08
06
05
Security content,
awareness
Training Creation

24. Duality
• Sophisticated social engineering attacks
• Gen AI generated Code exploits &
Malware
• Productivity improvements vs ethical
responsibility
Question: Do we need ML models to combat
above issues?

25. Get started AI Security Journey

26. • General understanding: Talk to data science expert
• Outline the objective of your learning
• NIST’s Trustworthy & responsible AI
• EU AI ACT
• Mitre ATLAS
• OWASP AI & LLM top 10
• OWASP Comprehensive checklist
• Open-source tooling
• Build GPTs ~ Llama, open AI

27. Open-Source security Efforts
• WhyLabs LLM security
• CalypsoAI Moderator
• Adversa AI
• LLM attack Chains by Praetorian
• LLMGuard
• Lakera
• LLM Guardian
• Burp GPT
• Garak – Vuln scanners for LLMs

28. Additional resources
THEORY AND PRACTICE: MACHINE
LEARNING INTRODUCTION WITH
THREATS AND VULNERABILITIES ~
Abraham Kang, Blackhat (Aug 3-6)
Link: https://www.blackhat.com/us-24/training/schedule/#theory-and-practice-
machine-learning-introduction-with-threats-and-vulnerabilities-37480