[DoKDayNA2022] - Architecting Your First Event Driven Serverless Streaming Applications on K8

Timothy Spann
Timothy SpannDeveloper Advocate
Tim Spann
“Serverless
Streaming”
“Architecting Your First Event Driven Serverless
Streaming Application on K8”
Tim Spann
“Serverless
Streaming” Tim Spann DoK Day North America 2022 @ KubeCon
“Serverless
Streaming”
Timothy Spann
Developer Advocate
FLiP(N) Stack = Flink, Pulsar and NiFi Stack
Streaming Systems & Data Architecture Expert
Experience:
15+ years of experience with streaming technologies including Pulsar, Flink, Spark,
NiFi, Kafka, Big Data, Cloud, MXNet, IoT and more.
Today, he helps to grow the Pulsar community sharing rich technical knowledge and
experience at both global conferences and through individual conversations.
Tim Spann
“Serverless
Streaming” Tim Spann DoK Day North America 2022 @ KubeCon
“Serverless
Streaming”
FLiP Stack Weekly
This week in Apache Flink, Apache Pulsar, Apache
NiFi, Apache Spark, Elasticsearch and open source
friends.
https://bit.ly/32dAJft
Tim Spann
“Serverless
Streaming” Tim Spann
“Serverless
Streaming”
Apache Pulsar is a Cloud-Native Messaging
and Event-Streaming Platform.
DoK Day North America 2022 @ KubeCon
Tim Spann
“Serverless
Streaming”
Apache Pulsar adoption is
being driven by
organizations seeking
cloud-native architectures
and new uses cases.
Microservices
Apache Pulsar - Built for Containers / Modern Cloud
Cloud Native
Hybrid & Multi-Cloud Containers
DoK Day North America 2022 @ KubeCon
Tim Spann
“Serverless
Streaming”
Key Pulsar Concepts: Messaging vs
Streaming
Message Queueing - Queueing
systems are ideal for work queues
that do not require tasks to be
performed in a particular order.
Streaming - Streaming works
best in situations where the
order of messages is important.
DoK Day North America 2022 @ KubeCon
Tim Spann
“Serverless
Streaming”
Pulsar Cluster
CLUSTER
Global ZooKeeper
Configuration Store
ZK3
ZK2
ZK1
Local
ZooKeeper
Quorum
Bookie 0 Bookie 1 Bookie 2
BookKeeper
Bookie
Ensemble
Pulsar Broker 0 Pulsar Broker 1 Pulsar Broker 2
DoK Day North America 2022 @ KubeCon
Tim Spann
“Serverless
Streaming”
● Consume messages from one
or more Pulsar topics.
● Apply user-supplied
processing logic to each
message.
● Publish the results of the
computation to another topic.
● Support multiple
programming languages
(Java, Python, Go)
● Can leverage 3rd-party
libraries
Pulsar Functions
DoK Day North America 2022 @ KubeCon
Tim Spann
“Serverless
Streaming”
import java.util.function.Function;
public class MyFunction implements Function<String, String> {
public String apply(String input) {
return doBusinessLogic(input);
}
}
Entire Function
Why Pulsar
Functions?
Tim Spann
“Serverless
Streaming”
from pulsar import Function
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
import json
class Chat(Function):
def __init__(self):
pass
def process(self, input, context):
fields = json.loads(input)
sid = SentimentIntensityAnalyzer()
ss = sid.polarity_scores(fields["comment"])
row = { }
row['id'] = str(msg_id)
if ss['compound'] < 0.00:
row['sentiment'] = 'Negative'
else:
row['sentiment'] = 'Positive'
row['comment'] = str(fields["comment"])
json_string = json.dumps(row)
return json_string
Entire Function
Pulsar Python
NLP Function
https://github.com/tspannhw/pulsar-pychat-function
Tim Spann
“Serverless
Streaming”
Function Mesh
Pulsar Functions, along with Pulsar IO/Connectors, provide a powerful API for ingesting,
transforming, and outputting data.
Function Mesh, another StreamNative project, makes it easier for developers to create entire
applications built from sources, functions, and sinks all through a declarative API.
Tim Spann
“Serverless
Streaming”
K8 Deploy
Tim Spann
“Serverless
Streaming”
Function Execution
DoK Day North America 2022 @ KubeCon
Tim Spann
“Serverless
Streaming”
Apache Pulsar Kafka K8
https://docs.streamnative.io/platform/v1.3.0/quickstart
DoK Day North America 2022 @ KubeCon
Tim Spann
“Serverless
Streaming”
Webinar Series:
Building Microservices
with Pulsar
Watch now Read more
Learn how Pulsar Functions, can be
leveraged to build a message bus for
event-driven microservices
DoK Day North America 2022 @ KubeCon
Tim Spann
“Serverless
Streaming”
Apache Pulsar Resources
● https://pulsar.apache.org/docs/en/kubernetes-helm/
● https://pulsar.apache.org/charts
● https://streamnative.io/streamnativecloud/
● https://medium.com/@_oleksii_/how-to-deploy-apache-puls
ar-cluster-in-kubernetes-808ecdfd87
Academy.StreamNative.io
Tim Spann
“Serverless
Streaming”
Thank you!
DoK Day North America 2022 @ KubeCon
Tim Spann
“Serverless
Streaming”
1 of 17

Recommended

Building Real-Time Travel Alerts by
Building Real-Time Travel AlertsBuilding Real-Time Travel Alerts
Building Real-Time Travel AlertsTimothy Spann
165 views48 slides
JConWorld_ Continuous SQL with Kafka and Flink by
JConWorld_ Continuous SQL with Kafka and FlinkJConWorld_ Continuous SQL with Kafka and Flink
JConWorld_ Continuous SQL with Kafka and FlinkTimothy Spann
156 views36 slides
[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines by
[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines
[EN]DSS23_tspann_Integrating LLM with Streaming Data PipelinesTimothy Spann
150 views25 slides
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo by
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines DemoEvolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines DemoTimothy Spann
162 views8 slides
CoC23_ Looking at the New Features of Apache NiFi by
CoC23_ Looking at the New Features of Apache NiFiCoC23_ Looking at the New Features of Apache NiFi
CoC23_ Looking at the New Features of Apache NiFiTimothy Spann
36 views24 slides
CoC23_ Let’s Monitor The Conditions at the Conference by
CoC23_ Let’s Monitor The Conditions at the ConferenceCoC23_ Let’s Monitor The Conditions at the Conference
CoC23_ Let’s Monitor The Conditions at the ConferenceTimothy Spann
17 views17 slides

More Related Content

More from Timothy Spann

The Never Landing Stream with HTAP and Streaming by
The Never Landing Stream with HTAP and StreamingThe Never Landing Stream with HTAP and Streaming
The Never Landing Stream with HTAP and StreamingTimothy Spann
254 views39 slides
Meetup - Brasil - Data In Motion - 2023 September 19 by
Meetup - Brasil - Data In Motion - 2023 September 19Meetup - Brasil - Data In Motion - 2023 September 19
Meetup - Brasil - Data In Motion - 2023 September 19Timothy Spann
319 views33 slides
Implement a Universal Data Distribution Architecture to Manage All Streaming ... by
Implement a Universal Data Distribution Architecture to Manage All Streaming ...Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Implement a Universal Data Distribution Architecture to Manage All Streaming ...Timothy Spann
28 views56 slides
Building Real-time Pipelines with FLaNK_ A Case Study with Transit Data by
Building Real-time Pipelines with FLaNK_ A Case Study with Transit DataBuilding Real-time Pipelines with FLaNK_ A Case Study with Transit Data
Building Real-time Pipelines with FLaNK_ A Case Study with Transit DataTimothy Spann
193 views45 slides
big data fest building modern data streaming apps by
big data fest building modern data streaming appsbig data fest building modern data streaming apps
big data fest building modern data streaming appsTimothy Spann
317 views55 slides
Using Apache NiFi with Apache Pulsar for Fast Data On-Ramp by
Using Apache NiFi with Apache Pulsar for Fast Data On-RampUsing Apache NiFi with Apache Pulsar for Fast Data On-Ramp
Using Apache NiFi with Apache Pulsar for Fast Data On-RampTimothy Spann
163 views27 slides

More from Timothy Spann(20)

The Never Landing Stream with HTAP and Streaming by Timothy Spann
The Never Landing Stream with HTAP and StreamingThe Never Landing Stream with HTAP and Streaming
The Never Landing Stream with HTAP and Streaming
Timothy Spann254 views
Meetup - Brasil - Data In Motion - 2023 September 19 by Timothy Spann
Meetup - Brasil - Data In Motion - 2023 September 19Meetup - Brasil - Data In Motion - 2023 September 19
Meetup - Brasil - Data In Motion - 2023 September 19
Timothy Spann319 views
Implement a Universal Data Distribution Architecture to Manage All Streaming ... by Timothy Spann
Implement a Universal Data Distribution Architecture to Manage All Streaming ...Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Timothy Spann28 views
Building Real-time Pipelines with FLaNK_ A Case Study with Transit Data by Timothy Spann
Building Real-time Pipelines with FLaNK_ A Case Study with Transit DataBuilding Real-time Pipelines with FLaNK_ A Case Study with Transit Data
Building Real-time Pipelines with FLaNK_ A Case Study with Transit Data
Timothy Spann193 views
big data fest building modern data streaming apps by Timothy Spann
big data fest building modern data streaming appsbig data fest building modern data streaming apps
big data fest building modern data streaming apps
Timothy Spann317 views
Using Apache NiFi with Apache Pulsar for Fast Data On-Ramp by Timothy Spann
Using Apache NiFi with Apache Pulsar for Fast Data On-RampUsing Apache NiFi with Apache Pulsar for Fast Data On-Ramp
Using Apache NiFi with Apache Pulsar for Fast Data On-Ramp
Timothy Spann163 views
OSSNA Building Modern Data Streaming Apps by Timothy Spann
OSSNA Building Modern Data Streaming AppsOSSNA Building Modern Data Streaming Apps
OSSNA Building Modern Data Streaming Apps
Timothy Spann155 views
GSJUG: Mastering Data Streaming Pipelines 09May2023 by Timothy Spann
GSJUG: Mastering Data Streaming Pipelines 09May2023GSJUG: Mastering Data Streaming Pipelines 09May2023
GSJUG: Mastering Data Streaming Pipelines 09May2023
Timothy Spann255 views
BestInFlowCompetitionTutorials03May2023 by Timothy Spann
BestInFlowCompetitionTutorials03May2023BestInFlowCompetitionTutorials03May2023
BestInFlowCompetitionTutorials03May2023
Timothy Spann11 views
Cloudera Sandbox Event Guidelines For Workflow by Timothy Spann
Cloudera Sandbox Event Guidelines For WorkflowCloudera Sandbox Event Guidelines For Workflow
Cloudera Sandbox Event Guidelines For Workflow
Timothy Spann32 views
Meet the Committers Webinar_ Lab Preparation by Timothy Spann
Meet the Committers Webinar_ Lab PreparationMeet the Committers Webinar_ Lab Preparation
Meet the Committers Webinar_ Lab Preparation
Timothy Spann32 views
Best Practices For Workflow by Timothy Spann
Best Practices For WorkflowBest Practices For Workflow
Best Practices For Workflow
Timothy Spann89 views
Meetup: Streaming Data Pipeline Development by Timothy Spann
Meetup:  Streaming Data Pipeline DevelopmentMeetup:  Streaming Data Pipeline Development
Meetup: Streaming Data Pipeline Development
Timothy Spann337 views
DevNexus: Apache Pulsar Development 101 with Java by Timothy Spann
DevNexus:  Apache Pulsar Development 101 with JavaDevNexus:  Apache Pulsar Development 101 with Java
DevNexus: Apache Pulsar Development 101 with Java
Timothy Spann261 views
Conf42 Python_ ML Enhanced Event Streaming Apps with Python Microservices by Timothy Spann
Conf42 Python_ ML Enhanced Event Streaming Apps with Python MicroservicesConf42 Python_ ML Enhanced Event Streaming Apps with Python Microservices
Conf42 Python_ ML Enhanced Event Streaming Apps with Python Microservices
Timothy Spann443 views
ITPC Building Modern Data Streaming Apps by Timothy Spann
ITPC Building Modern Data Streaming AppsITPC Building Modern Data Streaming Apps
ITPC Building Modern Data Streaming Apps
Timothy Spann797 views
PythonWebConference_ Cloud Native Apache Pulsar Development 202 with Python by Timothy Spann
PythonWebConference_ Cloud Native Apache Pulsar Development 202 with PythonPythonWebConference_ Cloud Native Apache Pulsar Development 202 with Python
PythonWebConference_ Cloud Native Apache Pulsar Development 202 with Python
Timothy Spann430 views
PhillyJug Getting Started With Real-time Cloud Native Streaming With Java by Timothy Spann
PhillyJug  Getting Started With Real-time Cloud Native Streaming With JavaPhillyJug  Getting Started With Real-time Cloud Native Streaming With Java
PhillyJug Getting Started With Real-time Cloud Native Streaming With Java
Timothy Spann625 views
Why Spring Belongs In Your Data Stream (From Edge to Multi-Cloud) by Timothy Spann
Why Spring Belongs In Your Data Stream (From Edge to Multi-Cloud)Why Spring Belongs In Your Data Stream (From Edge to Multi-Cloud)
Why Spring Belongs In Your Data Stream (From Edge to Multi-Cloud)
Timothy Spann18 views

Recently uploaded

Basic Design Flow for Field Programmable Gate Arrays by
Basic Design Flow for Field Programmable Gate ArraysBasic Design Flow for Field Programmable Gate Arrays
Basic Design Flow for Field Programmable Gate ArraysUsha Mehta
10 views21 slides
Global airborne satcom market report by
Global airborne satcom market reportGlobal airborne satcom market report
Global airborne satcom market reportdefencereport78
8 views13 slides
REACTJS.pdf by
REACTJS.pdfREACTJS.pdf
REACTJS.pdfArthyR3
39 views16 slides
Robotics in construction enterprise by
Robotics in construction enterpriseRobotics in construction enterprise
Robotics in construction enterpriseKhalid Abdel Naser Abdel Rahim
5 views1 slide
dummy.pptx by
dummy.pptxdummy.pptx
dummy.pptxJamesLamp
7 views2 slides
Web Dev Session 1.pptx by
Web Dev Session 1.pptxWeb Dev Session 1.pptx
Web Dev Session 1.pptxVedVekhande
23 views22 slides

Recently uploaded(20)

Basic Design Flow for Field Programmable Gate Arrays by Usha Mehta
Basic Design Flow for Field Programmable Gate ArraysBasic Design Flow for Field Programmable Gate Arrays
Basic Design Flow for Field Programmable Gate Arrays
Usha Mehta10 views
REACTJS.pdf by ArthyR3
REACTJS.pdfREACTJS.pdf
REACTJS.pdf
ArthyR339 views
Web Dev Session 1.pptx by VedVekhande
Web Dev Session 1.pptxWeb Dev Session 1.pptx
Web Dev Session 1.pptx
VedVekhande23 views
BCIC - Manufacturing Conclave - Technology-Driven Manufacturing for Growth by Innomantra
BCIC - Manufacturing Conclave -  Technology-Driven Manufacturing for GrowthBCIC - Manufacturing Conclave -  Technology-Driven Manufacturing for Growth
BCIC - Manufacturing Conclave - Technology-Driven Manufacturing for Growth
Innomantra 22 views
Design of Structures and Foundations for Vibrating Machines, Arya-ONeill-Pinc... by csegroupvn
Design of Structures and Foundations for Vibrating Machines, Arya-ONeill-Pinc...Design of Structures and Foundations for Vibrating Machines, Arya-ONeill-Pinc...
Design of Structures and Foundations for Vibrating Machines, Arya-ONeill-Pinc...
csegroupvn16 views
GDSC Mikroskil Members Onboarding 2023.pdf by gdscmikroskil
GDSC Mikroskil Members Onboarding 2023.pdfGDSC Mikroskil Members Onboarding 2023.pdf
GDSC Mikroskil Members Onboarding 2023.pdf
gdscmikroskil72 views
Ansari: Practical experiences with an LLM-based Islamic Assistant by M Waleed Kadous
Ansari: Practical experiences with an LLM-based Islamic AssistantAnsari: Practical experiences with an LLM-based Islamic Assistant
Ansari: Practical experiences with an LLM-based Islamic Assistant
M Waleed Kadous12 views
Programmable Logic Devices : SPLD and CPLD by Usha Mehta
Programmable Logic Devices : SPLD and CPLDProgrammable Logic Devices : SPLD and CPLD
Programmable Logic Devices : SPLD and CPLD
Usha Mehta27 views
Integrating Sustainable Development Goals (SDGs) in School Education by SheetalTank1
Integrating Sustainable Development Goals (SDGs) in School EducationIntegrating Sustainable Development Goals (SDGs) in School Education
Integrating Sustainable Development Goals (SDGs) in School Education
SheetalTank113 views
MongoDB.pdf by ArthyR3
MongoDB.pdfMongoDB.pdf
MongoDB.pdf
ArthyR351 views
Field Programmable Gate Arrays : Architecture by Usha Mehta
Field Programmable Gate Arrays : ArchitectureField Programmable Gate Arrays : Architecture
Field Programmable Gate Arrays : Architecture
Usha Mehta23 views

[DoKDayNA2022] - Architecting Your First Event Driven Serverless Streaming Applications on K8

  • 1. Tim Spann “Serverless Streaming” “Architecting Your First Event Driven Serverless Streaming Application on K8”
  • 2. Tim Spann “Serverless Streaming” Tim Spann DoK Day North America 2022 @ KubeCon “Serverless Streaming” Timothy Spann Developer Advocate FLiP(N) Stack = Flink, Pulsar and NiFi Stack Streaming Systems & Data Architecture Expert Experience: 15+ years of experience with streaming technologies including Pulsar, Flink, Spark, NiFi, Kafka, Big Data, Cloud, MXNet, IoT and more. Today, he helps to grow the Pulsar community sharing rich technical knowledge and experience at both global conferences and through individual conversations.
  • 3. Tim Spann “Serverless Streaming” Tim Spann DoK Day North America 2022 @ KubeCon “Serverless Streaming” FLiP Stack Weekly This week in Apache Flink, Apache Pulsar, Apache NiFi, Apache Spark, Elasticsearch and open source friends. https://bit.ly/32dAJft
  • 4. Tim Spann “Serverless Streaming” Tim Spann “Serverless Streaming” Apache Pulsar is a Cloud-Native Messaging and Event-Streaming Platform. DoK Day North America 2022 @ KubeCon
  • 5. Tim Spann “Serverless Streaming” Apache Pulsar adoption is being driven by organizations seeking cloud-native architectures and new uses cases. Microservices Apache Pulsar - Built for Containers / Modern Cloud Cloud Native Hybrid & Multi-Cloud Containers DoK Day North America 2022 @ KubeCon
  • 6. Tim Spann “Serverless Streaming” Key Pulsar Concepts: Messaging vs Streaming Message Queueing - Queueing systems are ideal for work queues that do not require tasks to be performed in a particular order. Streaming - Streaming works best in situations where the order of messages is important. DoK Day North America 2022 @ KubeCon
  • 7. Tim Spann “Serverless Streaming” Pulsar Cluster CLUSTER Global ZooKeeper Configuration Store ZK3 ZK2 ZK1 Local ZooKeeper Quorum Bookie 0 Bookie 1 Bookie 2 BookKeeper Bookie Ensemble Pulsar Broker 0 Pulsar Broker 1 Pulsar Broker 2 DoK Day North America 2022 @ KubeCon
  • 8. Tim Spann “Serverless Streaming” ● Consume messages from one or more Pulsar topics. ● Apply user-supplied processing logic to each message. ● Publish the results of the computation to another topic. ● Support multiple programming languages (Java, Python, Go) ● Can leverage 3rd-party libraries Pulsar Functions DoK Day North America 2022 @ KubeCon
  • 9. Tim Spann “Serverless Streaming” import java.util.function.Function; public class MyFunction implements Function<String, String> { public String apply(String input) { return doBusinessLogic(input); } } Entire Function Why Pulsar Functions?
  • 10. Tim Spann “Serverless Streaming” from pulsar import Function from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer import json class Chat(Function): def __init__(self): pass def process(self, input, context): fields = json.loads(input) sid = SentimentIntensityAnalyzer() ss = sid.polarity_scores(fields["comment"]) row = { } row['id'] = str(msg_id) if ss['compound'] < 0.00: row['sentiment'] = 'Negative' else: row['sentiment'] = 'Positive' row['comment'] = str(fields["comment"]) json_string = json.dumps(row) return json_string Entire Function Pulsar Python NLP Function https://github.com/tspannhw/pulsar-pychat-function
  • 11. Tim Spann “Serverless Streaming” Function Mesh Pulsar Functions, along with Pulsar IO/Connectors, provide a powerful API for ingesting, transforming, and outputting data. Function Mesh, another StreamNative project, makes it easier for developers to create entire applications built from sources, functions, and sinks all through a declarative API.
  • 14. Tim Spann “Serverless Streaming” Apache Pulsar Kafka K8 https://docs.streamnative.io/platform/v1.3.0/quickstart DoK Day North America 2022 @ KubeCon
  • 15. Tim Spann “Serverless Streaming” Webinar Series: Building Microservices with Pulsar Watch now Read more Learn how Pulsar Functions, can be leveraged to build a message bus for event-driven microservices DoK Day North America 2022 @ KubeCon
  • 16. Tim Spann “Serverless Streaming” Apache Pulsar Resources ● https://pulsar.apache.org/docs/en/kubernetes-helm/ ● https://pulsar.apache.org/charts ● https://streamnative.io/streamnativecloud/ ● https://medium.com/@_oleksii_/how-to-deploy-apache-puls ar-cluster-in-kubernetes-808ecdfd87 Academy.StreamNative.io
  • 17. Tim Spann “Serverless Streaming” Thank you! DoK Day North America 2022 @ KubeCon Tim Spann “Serverless Streaming”