[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines https://dssconf.pl/en/#agenda-section Integrating LLM with Streaming Data Pipelines Timothy Spann, Principal Developer Advocate, Cloudera APACHE NIFI, APACHE FLINK, APACHE KAFKA, LLM, HUGGINGFACE, REST, STREAMING LESS In this talk and demo I will walk through how to add LLMs to your streaming pipelines by integration through Apache NiFi. https://github.com/tspannhw/FLaNK-watsonx.ai Cloudera streaming llm generative ai slack to slack

1. Integrating LLM with
Streaming Data Pipelines
Tim Spann
Principal Dev, Cloudera
www.dssconf.pl PGE Narodowy + Online
ORGANIZERS:
23-24.11.2023

2. 2
FLaNK Stack Weekly by Tim Spann
This week in Apache NiFi, Apache Flink,
Apache Kafka, ML, AI, Apache Spark, Apache
Iceberg, Python, Java and Open Source
friends.
https://bit.ly/32dAJft
https://www.meetup.com/futureofdata-
princeton/

3. 3 / 3
Build a Streaming Data Pipeline
Integrate LLM via HTTPS / REST
Process, Route, Enrich, Transform Results
Store
Distribute
Agenda (35 minutes)

4. Build a Streaming Data
Pipeline

5. 5
© 2023 Cloudera, Inc. All rights reserved.
Streaming
data
Data at rest
Change data
capture
Any
DATA
Real-time
Processing
• Analyze data in
motion
• Continuous
monitoring
• Trends and
anomalies
Data
Lakehouse
Data
products
Any
BUSINESS
EVENT
Continuous
Results
• No-Code UI
• Author once
publish anywhere
• Analytics lifecycle
management for
dev/ops
Any
DATA
ANALYST
AI models
Event-driven
apps
Analytics apps
Any
DATA
CONSUMER
Data Relevance

6. LLM USE CASE
Vector DB
AI Model
Unstructured file types
Data in Motion
on Cloudera Data
Platform (CDP)
Capture, process &
distribute any data,
anywhere
Other enterprise data Open Data Lakehouse
Materialized Views
Structured Sources
Applications/API’s
Streams

7. Integrating LLM via HTTPS

8. 8 / 3
google/flan-ul2
google/flan-t5-xxl
bigscience/bloom
meta-llama/llama-2-70b-chat
ibm/mpt-7b-instruct2
ibm/granite-13b-instruct-v1
ibm/granite-13b-chat-v1
Models Used for Inference API

9. 9 / 3

10. 10 / 3
For Hugging Face:
https://api-inference.huggingface.co/models/bigscience/bloom
Pass in a simple JSON Prompt
{"inputs":"This is Prompt Text"}
Requires an HTTP Header with Authorization for Bearer YOURTOKEN
YOURTOKEN from https://huggingface.co/settings/tokens
HTTPS Calls

11. 11 / 3
For IBM WatsonX.AI:
https://iam.cloud.ibm.com/identity/token <- First call to get token by
sending an API key and grant.
Then send a JSON prompt call model server via
https://us-south.ml.cloud.ibm.com/ml/v1-beta/generation/text?version=2023
-05-29
Requires an HTTP Header with Authorization for Bearer ${readtoken}
HTTPS Calls

12. 12 / 3
{ "model_id": "meta-llama/llama-2-70b-chat",
"input": "this is prompt text”,
"parameters": {
"decoding_method": "greedy",
"max_new_tokens": 200,
"min_new_tokens": 50,
"stop_sequences": [],
"repetition_penalty": 1
},
"project_id": "0ead8ec4-d137-4f9c-8956-50b0da4a7068" }
WatsonX.AI JSON Prompt

13. 13 / 3

14. Process, Route, Enrich,
Transform Results

15. 15 / 3

16. 16 / 3

17. Store

18. 18 / 3

19. 19 / 3

20. Distribute

21. 21 / 3

22. 22 / 3

23. Resources

24. 24 / 3

25. Thank you for
watching!
Remember to leave your questions
and rate the presentation
in the section below.
www.dssconf.pl 23-24.11.2023 PGE Narodowy + Online
ORGANIZERS: