The document outlines the importance of real-time data streaming for enhancing generative AI applications, emphasizing the integration of operational data across enterprises to provide reliable and scalable insights. It highlights how large language models (LLMs) and vector stores can be modularized for improved performance, requiring contextual and trustworthy data to drive value. Additionally, it details the process for AI application development, including data augmentation, inference, workflows, and post-processing, facilitated by Confluent's platform.

1. Il Data Streaming per un’AI
real-time di nuova generazione
Samuele Dell’Angelo
Senior Solutions Engineer, Confluent

2. 4:30 - 5:15
5:15 - 5:30
5:30
Il Data Streaming per un’AI real-time di nuova generazione.
Perchè Confluent.
- Integrazione di dati operativi provenienti da diverse parti dell'azienda, in
tempo reale, per un utilizzo affidabile
- Disaccoppiamento delle applicazioni customer-facing rispetto alle
chiamate LLM per offrire esperienze affidabili, reattive e scalabili
orizzontalmente
- Gestione di LLM, archivi vettoriali e modelli di embedding come
componenti modulari, sostituibili man mano che la tecnologia migliora
Q&A Session
Chiusura Lavori
Agenda 2

3. CONNECT
PROCESS
GOVERN
SHARE
Custom Apps &
Microservices
Data Systems
STREAM
AI/ML Modeling
Inventory Payments
Personalization
Fraud Supply Chain
Recommendations
From Data Mess To Data Products
To Instant Value
Everywhere
Brief Introduction to Confluent

4. “Our latest research estimates that generative AI could add
the equivalent of $2.6 trillion to $4.4 trillion annually across the
63 use cases we analyzed”
- McKinsey study

5. Generative AI is a revolutionary tool…
…and it’s only getting better.
/imagine prompt:Street style photo of a woman shot on Kodak
July 2022 July 2023
Source: https://twitter.com/nickfloats/status/1676279157620199424?s=46&t=plcKoQYXnokFvxs3ieVg3Q
June 2024
June 2024

6. Generative AI: the hottest topic in tech…
…but what makes it different?
- AI models that generate content (e.g., text, pictures) by
making predictions based on patterns in training data.
- Uses Foundation Models (e.g., LLMs) that are prohibitively
expensive ($100M+) to train.
- Models are trained on 1+ year-old public data.
- However, models are inherently reusable.

7. Implication: The democratization of AI…
…but app-specific data management remains.
In traditional ML, most of the data engineering work happens at model creation time…
…but with large language models, data engineering happens with every query.

8. LLMs can drive value for your business…
…but only if they have context from your data.

9. Without contextualized, trusted, current data…
…LLMs can’t drive meaningful value.
What is the status of my flight to New York?
It is currently delayed by 2 hours and expected to
depart at 5 pm GMT.
Is there another flight available to the same city that
will depart and arrive sooner? What are the seating
options and cost?
Can your GenAI assistant
remember data from an earlier
conversation?
What is the source of this
information? Is this trustworthy?
Is it fresh and accurate?
How do you augment customer
data with real-time data and
process them on the fly to
provide meaningful insights?
The next available flight to New York with United
departs later but will arrive faster than your current
flight.
The only available seats in this flight are first class
window seats and costs $1,500.

10. Generative AI unlocks competitive advantage…
…but needs your data for context.
Common Use Case Requires domain-
specific data
Better with domain-
specific data
Better with real-time data
Semantic Search
Customer Service
Content Discovery/
Recommendation
Code Development
Content Generation
Agents (task
automation)

11. Traditional enterprise data architecture
is a GenAI innovation bottleneck
Historic Public Data
Generative
AI Model
Intelligent
Business-Specific
Co-Pilot
User Interaction
??
Enterprise data architecture
In-context learning &
prompt-time assembly

12. Confluent enables real-time GenAI-powered
applications at scale
How?
• Integrates disparate operational data across the enterprise
in real time for reliable, trustworthy use
• Organizes unstructured enterprise data embeddings into
vector stores
• Decouples customer-facing applications from LLM call
management to provide reliable, reactive experiences that
scale horizontally
• Enables LLMs, vector stores, and embedding models to be
treated as modular components that can substituted as
technology improves

13. LLM-enabled Applications have four steps
1. Data Augmentation that prepares data for contextualization
in LLM queries with activities such as chunking, creating
embeddings, and storing in a vector store;
2. Inference that includes engineering prompts and handling
responses;
3. Workflows that are composed of agents and chains of
inference steps that form GenAI-enabled applications; and,
4. Post-Processing that validates outputs and enforces business
logic.
Let’s walk through how Confluent can help with each general step.

14. Data Augmentation: General Pattern

15. Data Augmentation: Example Implementation

16. Confluent Partners offering Vector Search
Available Now

17. Inference: General Pattern

18. Inference: Example Implementation

19. Workflows: General Pattern

20. Workflows: Example Implementation

21. Post-Processing: General Pattern

22. Post-Processing: Example Implementation

23. AI Model Inference in
Confluent Cloud
Simplify the development and
deployment of AI applications
by providing a unified
platform for both data
processing and AI/ML tasks
Simplify development by using
familiar SQL syntax to work directly
with AI models, reducing the need for
specialized tools and languages.
Enable seamless coordination between
data processing and ML workflows to
improve efficiency and reduce
operational complexity
Facilitate accurate, real-time AI-driven
decision-making by leveraging fresh,
contextual streaming data
EARLY ACCESS

24. INSERT INTO enriched_reviews
SELECT id
, review
,
invoke_openai(prompt,review) as
score
FROM product_reviews
;
K
N
B
Kate
4 hours ago
This was the worst decision ever.
Nikola
1 day ago
Not bad. Could have been cheaper.
Brian
3 days ago
Amazing! Game Changer!
K
N
B
Kate
★★★★★ 4 hours ago
This was the worst decision ever.
Nikola
★★★★★ 1 day ago
Not bad. Could have been cheaper.
Brian
★★★★★ 3 days ago
Amazing! Game Changer!
The Prompt
“Score the following text on a scale of 1
and 5 where 1 is negative and 5 is
positive returning only the number”
DATA STREAMING PLATFORM
Enrich real-time data streams with Generative AI directly
from Flink SQL
Next

25. Remote AI Model with Confluent Cloud for Apache Flink
Next
CREATE TABLE text_stream (
id BIGINT, text STRING
);
INSERT INTO text_stream SELECT 1 id, 'The mitochondria are the powerhouse of the cell'
text;
INSERT INTO text_stream SELECT 2 id, 'Happy Birthday! You are great!'text;
INSERT INTO text_stream SELECT 3 id, 'You are bad and you should feel bad.'
text;
SET 'sql.secrets.my_api_key'= '<YOUR_API_KEY>'
;
CREATE MODEL sentimentmodel
INPUT(text STRING)
OUTPUT(sentiment STRING)
COMMENT 'sentiment analysis model'
WITH (
'provider' = 'openai',
'task' = 'text_generation'
,
'openai.endpoint' = 'https://api.openai.com/v1/chat/completions'
,
'openai.api_key' = '{{sessionconfig/sql.secrets.my_api_key}}'
,
'openai.model_version'= 'gpt-3.5-turbo'
,
'openai.system_prompt'= 'Analyze the sentiment of the text and return only POSITIVE, NEGATIVE, or NEUTRAL.'
);
SELECT id, text, sentiment FROM text_stream
, LATERAL TABLE(ML_PREDICT
('sentimentmodel'
, text));
Create the table for input text
Assign the secrets for
accessing your AWS account
resources
Run the following code to
create the text embedding
model
Run the inference statement
on the table and model

26. Q&A Session 2
6

27. sdellangelo@confluent.io.
Grazie! 2
7