StreamNative FLiP into scylladb - scylla summit 2022

•

1 like•528 views

StreamNative FLiP into scylladb - scylla summit 2022 Utilizing Apache Pulsar with Apache NiFi, Apache Flink, Apache Spark and Scylla for fast IoT application with MQTT and beyond.

Technology

FLiP Into Apache
Pulsar Apps with
ScyllaDB
Timothy Spann
Developer Advocate

Tim Spann
■ FLiP/FLiPN/FLiPS Stacks
■ Ex-Principal Field Engineer - Cloudera
■ Ex-Senior Solutions Engineer - Hortonworks
■ Ex-Field Engineer - Pivotal
Developer Advocate

Apache Pulsar
■ Serverless computing framework
■ Unbounded storage, multi-tiered
architecture, and tiered-storage
■ Streaming & Pub/Sub messaging
semantics
■ Multi-protocol support

Why Apache Pulsar?
Uniﬁed
Messaging
Platform
Guaranteed
Message
Delivery
Resiliency Inﬁnite
Scalability

Use Cases
■ Uniﬁed Messaging Platform
■ AdTech
■ Fraud Detection
■ Connected Car
■ IoT Analytics
■ Microservices Development

<-> Events <->
Streaming FLiPS Apps
StreamNative Hub
StreamNative Cloud
Uniﬁed Batch and Stream COMPUTING
Batch
(Batch + Stream)
Uniﬁed Batch and Stream STORAGE
Offload
(Queuing + Streaming)
Tiered Storage
Pulsar
---
KoP
---
MoP
---
Websocket
Pulsar
Sink
Streaming
Edge Gateway
Protocols
<-> Events <->
CDC
Apps

Connectivity
• Functions - Lightweight Stream
Processing (Java, Python, Go)
• Connectors - Sources & Sinks
(Cassandra, Kafka, …)
• Protocol Handlers - AoP (AMQP), KoP
(Kafka), MoP (MQTT)
• Processing Engines - Flink, Spark,
Presto/Trino via Pulsar SQL
• Data Oﬄoaders - Tiered Storage - (S3)
hub.streamnative.io

Scylla Compatible Sink Connector
https://pulsar.apache.org/docs/en/io-quickstart/
Scylla Compatible Sink

Your App https://github.com/tspannhw/ScyllaFLiPSTheStream

What’s Next?
Continue your journey with Apache Pulsar:
■ Get started with Pulsar Training from
StreamNative Academy
■ Watch the 3-part on-demand webinar
series - Building Event-driven
Microservices with Pulsar
■ Join the Pulsar community on Slack

Thank you!
Stay in touch
Timothy Spann
@PaaSDev
tim@streamnative.io
https://www.linkedin.com/in/timothyspann
tspannhw

What's hot

Spark optimizationAnkit Beohar

Big data conference europe real-time streaming in any and all clouds, hybri...Timothy Spann

Osacon 2021 hello hydrate! from stream to clickhouse with apache pulsar and...Timothy Spann

Music city data Hail Hydrate! from stream to lakeTimothy Spann

Real time cloud native open source streaming of any data to apache solrTimothy Spann

Data minutes #2 Apache Pulsar with MQTT for Edge Computing Lightning - 2022Timothy Spann

Devfest uk & ireland using apache nifi with apache pulsar for fast data on-r...Timothy Spann

Using the flipn stack for edge ai (flink, nifi, pulsar)Timothy Spann

Scenic City Summit (2021): Real-Time Streaming in any and all clouds, hybrid...Timothy Spann

Distributed Crypto-Currency Trading with Apache PulsarStreamlio

Python web conference 2022 apache pulsar development 101 with python (f li-...Timothy Spann

ApacheCon 2021: Apache NiFi 101- introduction and best practicesTimothy Spann

Real-time Streaming Pipelines with FLaNKData Con LA

Hail hydrate! from stream to lake using open sourceTimothy Spann

Automation + dev ops summit hail hydrate! from stream to lakeTimothy Spann

Matt Franklin - Apache Software (Geekfest)W2O Group

Ai dev world utilizing apache pulsar, apache ni fi and minifi for edgeai io...Timothy Spann

Using Apache Spark with IBM SPSS ModelerGlobal Knowledge Training

Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europ...StreamNative

Architecting for ScalePooyan Jamshidi

What's hot (20)

Spark optimization

Big data conference europe real-time streaming in any and all clouds, hybri...

Osacon 2021 hello hydrate! from stream to clickhouse with apache pulsar and...

Music city data Hail Hydrate! from stream to lake

Real time cloud native open source streaming of any data to apache solr

Data minutes #2 Apache Pulsar with MQTT for Edge Computing Lightning - 2022

Devfest uk & ireland using apache nifi with apache pulsar for fast data on-r...

Using the flipn stack for edge ai (flink, nifi, pulsar)

Scenic City Summit (2021): Real-Time Streaming in any and all clouds, hybrid...

Distributed Crypto-Currency Trading with Apache Pulsar

Python web conference 2022 apache pulsar development 101 with python (f li-...

ApacheCon 2021: Apache NiFi 101- introduction and best practices

Real-time Streaming Pipelines with FLaNK

Hail hydrate! from stream to lake using open source

Automation + dev ops summit hail hydrate! from stream to lake

Matt Franklin - Apache Software (Geekfest)

Ai dev world utilizing apache pulsar, apache ni fi and minifi for edgeai io...

Using Apache Spark with IBM SPSS Modeler

Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europ...

Architecting for Scale

Similar to StreamNative FLiP into scylladb - scylla summit 2022

Cloud lunch and learn real-time streaming in azureTimothy Spann

DBCC 2021 - FLiP Stack for Cloud Data LakesTimothy Spann

Using FLiP with InfluxDB for EdgeAI IoT at Scale 2022Timothy Spann

Big mountain data and dev conference apache pulsar with mqtt for edge compu...Timothy Spann

Big Data Analytics Platforms by KTH and RISE SICSBig Data Value Association

Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...Timothy Spann

ApacheCon2022_Citizen Streaming Engineer - A How ToTimothy Spann

Timothy Spann [StreamNative] | Using FLaNK with InfluxDB for EdgeAI IoT at Sc...InfluxData

Using FLiP with influxdb for EdgeAI IoT at ScaleTimothy Spann

Near real-time anomaly detection at Lyftmarkgrover

OSSNA Building Modern Data Streaming AppsTimothy Spann

Processing Real-Time Data at Scale: A streaming platform as a central nervous...confluent

Citizen Streaming Engineer - A How ToTimothy Spann

Apache Pulsar: Why Unified Messaging and Streaming Is the Future - Pulsar Sum...StreamNative

Data Stream Processing with Apache FlinkFabian Hueske

Budapest Data/ML - Building Modern Data Streaming Apps with NiFi, Flink and K...Timothy Spann

Apache Pulsar with MQTT for Edge Computing - Pulsar Summit Asia 2021StreamNative

ApacheCon2022_Deep Dive into Building Streaming Applications with Apache PulsarTimothy Spann

Codeless pipelines with pulsar and flinkTimothy Spann

Workshop híbrido: Stream Processing con Flinkconfluent

Similar to StreamNative FLiP into scylladb - scylla summit 2022 (20)

Cloud lunch and learn real-time streaming in azure

DBCC 2021 - FLiP Stack for Cloud Data Lakes

Using FLiP with InfluxDB for EdgeAI IoT at Scale 2022

Big mountain data and dev conference apache pulsar with mqtt for edge compu...

Big Data Analytics Platforms by KTH and RISE SICS

Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...

ApacheCon2022_Citizen Streaming Engineer - A How To

Timothy Spann [StreamNative] | Using FLaNK with InfluxDB for EdgeAI IoT at Sc...

Using FLiP with influxdb for EdgeAI IoT at Scale

Near real-time anomaly detection at Lyft

OSSNA Building Modern Data Streaming Apps

Processing Real-Time Data at Scale: A streaming platform as a central nervous...

Citizen Streaming Engineer - A How To

Apache Pulsar: Why Unified Messaging and Streaming Is the Future - Pulsar Sum...

Data Stream Processing with Apache Flink

Budapest Data/ML - Building Modern Data Streaming Apps with NiFi, Flink and K...

Apache Pulsar with MQTT for Edge Computing - Pulsar Summit Asia 2021

ApacheCon2022_Deep Dive into Building Streaming Applications with Apache Pulsar

Codeless pipelines with pulsar and flink

Workshop híbrido: Stream Processing con Flink

Recently uploaded

Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst

Install Stable Diffusion in windows machinePadma Pradeep

08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls

My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar

E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxnull - The Open Security Community

Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your BudgetEnjoy Anytime

WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure servicePooja Nehwal

AI as an Interface for Commercial BuildingsMemoori

Presentation on how to chat with PDF using ChatGPT code interpreternaman860154

Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...shyamraj55

GenCyber Cyber Security Day PresentationMichael W. Hawkins

Maximizing Board Effectiveness 2024 Webinar.pptxOnBoard

Swan(sea) Song – personal research during my six years at Swansea ... and bey...Alan Dix

Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada

Snow Chain-Integrated Tire for a Safe Drive on Winter RoadsHyundai Motor Group

Vulnerability_Management_GRC_by Sohang Sengupta.pptxnull - The Open Security Community

Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptxnull - The Open Security Community

Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j

FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhisoniya singh

Recently uploaded (20)

Human Factors of XR: Using Human Factors to Design XR Systems

Install Stable Diffusion in windows machine

08448380779 Call Girls In Civil Lines Women Seeking Men

My Hashitalk Indonesia April 2024 Presentation

E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx

Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your Budget

WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service

AI as an Interface for Commercial Buildings

Presentation on how to chat with PDF using ChatGPT code interpreter

Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...

GenCyber Cyber Security Day Presentation

Maximizing Board Effectiveness 2024 Webinar.pptx

Swan(sea) Song – personal research during my six years at Swansea ... and bey...

Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024

Snow Chain-Integrated Tire for a Safe Drive on Winter Roads

Vulnerability_Management_GRC_by Sohang Sengupta.pptx

Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx

Breaking the Kubernetes Kill Chain: Host Path Mount

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...

FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi

StreamNative FLiP into scylladb - scylla summit 2022

1. FLiP Into Apache Pulsar Apps with ScyllaDB Timothy Spann Developer Advocate

2. Tim Spann ■ FLiP/FLiPN/FLiPS Stacks ■ Ex-Principal Field Engineer - Cloudera ■ Ex-Senior Solutions Engineer - Hortonworks ■ Ex-Field Engineer - Pivotal Developer Advocate

3. Cloud Streaming

4. Apache Pulsar ■ Serverless computing framework ■ Unbounded storage, multi-tiered architecture, and tiered-storage ■ Streaming & Pub/Sub messaging semantics ■ Multi-protocol support

5. Why Apache Pulsar? Uniﬁed Messaging Platform Guaranteed Message Delivery Resiliency Inﬁnite Scalability

6. Use Cases ■ Uniﬁed Messaging Platform ■ AdTech ■ Fraud Detection ■ Connected Car ■ IoT Analytics ■ Microservices Development

7. <-> Events <-> Streaming FLiPS Apps StreamNative Hub StreamNative Cloud Uniﬁed Batch and Stream COMPUTING Batch (Batch + Stream) Uniﬁed Batch and Stream STORAGE Offload (Queuing + Streaming) Tiered Storage Pulsar --- KoP --- MoP --- Websocket Pulsar Sink Streaming Edge Gateway Protocols <-> Events <-> CDC Apps

8. Connect

9. Connectivity • Functions - Lightweight Stream Processing (Java, Python, Go) • Connectors - Sources & Sinks (Cassandra, Kafka, …) • Protocol Handlers - AoP (AMQP), KoP (Kafka), MoP (MQTT) • Processing Engines - Flink, Spark, Presto/Trino via Pulsar SQL • Data Oﬄoaders - Tiered Storage - (S3) hub.streamnative.io

10. Scylla Compatible Sink Connector https://pulsar.apache.org/docs/en/io-quickstart/ Scylla Compatible Sink

11. Build and Run

12. Conﬁg

13. Deﬁne

14. Deploy

15. Deploy ML Function

16. Your App https://github.com/tspannhw/ScyllaFLiPSTheStream

17. Future

18. Shard-Aware Pulsar Connector for Scylla

19. What’s Next? Continue your journey with Apache Pulsar: ■ Get started with Pulsar Training from StreamNative Academy ■ Watch the 3-part on-demand webinar series - Building Event-driven Microservices with Pulsar ■ Join the Pulsar community on Slack

20. Thank you! Stay in touch Timothy Spann @PaaSDev tim@streamnative.io https://www.linkedin.com/in/timothyspann tspannhw

21. Demo chat pulsar_test_table

22. Sink Status

StreamNative FLiP into scylladb - scylla summit 2022

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to StreamNative FLiP into scylladb - scylla summit 2022

Similar to StreamNative FLiP into scylladb - scylla summit 2022 (20)

More from Timothy Spann

More from Timothy Spann (20)

Recently uploaded

Recently uploaded (20)

StreamNative FLiP into scylladb - scylla summit 2022