Building event streaming pipelines using Apache Pulsar

•

1 like•300 views

StreamNative

"Event Streaming with Apache Pulsar" by Sijie Guo in open source community data #ossday

Data & Analytics

streamnative.io
Who am I?
● Sijie Guo (@sijieg)
● Co-Founder & CEO, StreamNative
● PMC Member of Pulsar/BookKeeper
● Ex Co-Founder, Streamlio
● Ex-Twitter, Ex-Yahoo
● Work on messaging and streaming data
technologies for many years

streamnative.io
About StreamNative
● San Francisco based company
● Founded by the original developers of
Apache Pulsar and Apache BookKeeper
● Offer a cloud-native event streaming
platform powered by Pulsar
● Provide fully managed Pulsar services from
public cloud to on-premises

streamnative.io
Event Streaming with
Apache Pulsar
Sijie Guo (@sijie)
streamnative.io

streamnative.io
Event Streaming Platform
● Connect
○ Publish and subscribe to event streams
● Store
○ Store inﬁnite event streams reliably
● Process
○ Process event streams in real-time at scale

streamnative.io
Apache Pulsar
● Publish-subscribe: uniﬁed messaging model (streaming + queueing)
● Inﬁnite event stream storage: Apache BookKeeper + Tiered Storage
● Connectors: ingest events without writing code
● Process events in real-time
○ Pulsar Functions for serverless / lightweight computation
○ Spark / Flink for uniﬁed data processing
○ Presto for interactive queries

streamnative.io
Event Streaming Platform

streamnative.io
● Connect
○ Protocol Handler
■ Kafka-on-Pulsar
■ AMQP-on-Pulsar
● Process
○ Uniﬁed data processing
■ Spark-Pulsar Integration
■ Flink-Pulsar Integration
StreamNative Extensibilities

streamnative.io
Why Kafka-on-Pulsar
● Kafka - Rich ecosystem of connectors and integrations
● Pulsar
○ Inﬁnite stream storage
○ Pulsar Functions
● Adoption & Migration
● Real-time data pipelines
○ Ingesting data using Kafka Connect
○ Processing data using Pulsar Functions

streamnative.io
Kafka-on-Pulsar Protocol Handler

streamnative.io
Kafka-on-Pulsar
● Support Kafka 2.0+ protocols
● Support official java client, librdkafka, and non-java clients
● Multi-tenancy support
● SSL + SASL support
● It is a plugin and you can install it to your existing Pulsar
cluster!

streamnative.io
Demo
● K/P-Producer -> K/P-Consumer
○ TLS & SASL-PLAIN
● Kafka Connect -> Pulsar Functions
* All demos run with TLS and SASL-PLAIN
https://hackmd.io/nLj5M9BEQIacKcZsNrDxmQ

streamnative.io
Demo 1: K-Producer -> K-Consumer

streamnative.io
Demo 2: P-Producer -> K-Consumer

streamnative.io
Demo 3: K-Producer -> P-Consumer

streamnative.io
Demo 5: Pulsar Functions

streamnative.io
2020 Pulsar User Survey Report
https://streamnative.io/whitepaper/sn-apache-pulsar-user-survey-report-2020/

streamnative.io
TGI Pulsar Weekly Live Stream
https://www.youtube.com/channel/UCywxUI5HlIyc0VEKYR4X9Pg/live

streamnative.io
Follow us!
● KoP - https://github.com/streamnative/kop
● Webinar - https://www.youtube.com/watch?v=gL6hzRtij8M
● Follow us at Twitter
○ Sijie Guo (@sijieg)
○ Apache Pulsar (@apache_pulsar)
○ StreamNative (@streamnativeio)
● Join us at Pulsar slack!

What's hot

Infinite event streams are the core data abstraction in Apache Pulsar. Pulsar provides two-level reading APIs for accessing events in Pulsar topics, one is pub/sub and the other one is segment readers. The pub/sub API provides a unified messaging API for accessing events in a streaming way. People can choose different subscription modes for consuming events. The segment API provides a way to access events directly from Apache BookKeeper and tiered storage, which is more suitable for batch-oriented workloads. You can combine both pub/sub API and segment API to create a unified data processing experience as well. In the past year, we at StreamNative have been helping with many customers running Pulsar for different use cases from online queuing, event sourcing to stream and batch processing. We also worked on integrating Pulsar with different components in the big data ecosystem. In this talk, we will share our experiences and best practices of choosing the right API for accessing your event streams in Pulsar for different use cases.

Stream or segment : what is the best way to access your events in Pulsar_Neng

StreamNative

Code the docs-yu liu

StreamNative

Apache Pulsar at Yahoo! Japan

StreamNative

How Orange Financial combat financial frauds over 50M transactions a day usin...

JinfengHuang3

Transaction preview of Apache Pulsar

StreamNative

Apache Pulsar: A borderless community

StreamNative

Architecture of a Kafka camus infrastructure

mattlieber

SQL is eating the world (again!), and stream processing is no exception. As Flink SQL evolves to power business-critical applications at companies like Yelp, Airbnb or Uber, the Flink and Pulsar communities have been working in close collaboration to bring you the best of both worlds. But where do we stand today? In this talk, we’ll get you up to speed with the latest in streaming SQL with Flink and demo how you can integrate with Apache Pulsar to build unified, elastic data processing pipelines.

Select Star: Flink SQL for Pulsar Folks - Pulsar Summit NA 2021

StreamNative

Open keynote_carolyn&matteo&sijie

StreamNative

(VIRTUAL) Hail Hydrate! From Stream to Lake Using Open Source - Timothy J Spann, StreamNative https://osselc21.sched.com/event/lAPi?iframe=no A cloud data lake that is empty is not useful to anyone. How can you quickly, scalably and reliably fill your cloud data lake with diverse sources of data you already have and new ones you never imagined you needed. Utilizing open source tools from Apache, the FLiP stack enables any data engineer, programmer or analyst to build reusable modules with low or no code. FLiP utilizes Apache NiFi, Apache Pulsar, Apache Flink and MiNiFi agents to load CDC, Logs, REST, XML, Images, PDFs, Documents, Text, semistructured data, unstructured data, structured data and a hundred data sources you could never dream of streaming before. I will teach you how to fish in the deep end of the lake and return a data engineering hero. Let's hope everyone is ready to go from 0 to Petabyte hero. https://osselc21.sched.com/event/lAPi/virtual-hail-hydrate-from-stream-to-lake-using-open-source-timothy-j-spann-streamnative

Hail hydrate! from stream to lake using open source

Timothy Spann

Biography Tim Spann is a Principal DataFlow Field Engineer at Cloudera where he works with Apache NiFi, MiniFi, Pulsar, Apache Flink, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a senior solutions architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science. Talk Real-Time Streaming in Any and All Clouds, Hybrid and Beyond Today, data is being generated from devices and containers living at the edge of networks, clouds and data centers. We need to run business logic, analytics and deep learning at the scale and as events arrive. Tools: Apache Flink, Apache Pulsar, Apache NiFi, MiNiFi, DJL.ai Apache MXNet. References: https://www.datainmotion.dev/2019/11/introducing-mm-flank-apache-flink-stack.html https://www.datainmotion.dev/2019/08/rapid-iot-development-with-cloudera.html https://www.datainmotion.dev/2019/09/powering-edge-ai-for-sensor-reading.html https://www.datainmotion.dev/2019/05/dataworks-summit-dc-2019-report.html https://www.datainmotion.dev/2019/03/using-raspberry-pi-3b-with-apache-nifi.html Source Code: https://github.com/tspannhw/MmFLaNK FLiP Stack StreamNative

Big data conference europe real-time streaming in any and all clouds, hybri...

Timothy Spann

Five years of operating a large scale globally replicated Pulsar installation...

StreamNative

Real time cloud native open source streaming of any data to apache solr Utilizing Apache Pulsar and Apache NiFi we can parse any document in real-time at scale. We receive a lot of documents via cloud storage, email, social channels and internal document stores. We want to make all the content and metadata to Apache Solr for categorization, full text search, optimization and combination with other datastores. We will not only stream documents, but all REST feeds, logs and IoT data. Once data is produced to Pulsar topics it can instantly be ingested to Solr through Pulsar Solr Sink. Utilizing a number of open source tools, we have created a real-time scalable any document parsing data flow. We use Apache Tika for Document Processing with real-time language detection, natural language processing with Apache OpenNLP, Sentiment Analysis with Stanford CoreNLP, Spacy and TextBlob. We will walk everyone through creating an open source flow of documents utilizing Apache NiFi as our integration engine. We can convert PDF, Excel and Word to HTML and/or text. We can also extract the text to apply sentiment analysis and NLP categorization to generate additional metadata about our documents. We also will extract and parse images that if they contain text we can extract with TensorFlow and Tesseract.

Real time cloud native open source streaming of any data to apache solr

Timothy Spann

Suppose you want to know analytics on your Pulsar topics, or you want to debug those hard corner cases that fail to be sent, or even you want to monitor your Pulsar deployment: how do you do it? A tool exists to do this and more: Pulsar SQL. Since the 2.2.0 release, Pulsar SQL provides an abstraction layer to run any SQL query we may want against Pulsar effortlessly and without affecting performance. There is nothing like it on the pub-sub ecosystem. In this short session, we will revisit what Pulsar SQL is, how to make the best out of it, how to deploy it, and how to use it!

Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europ...

StreamNative

Kafka on Pulsar

StreamNative

Building a distributed Key-Value store with Cassandra

aaronmorton

Matt Franklin - Apache Software (Geekfest)

W2O Group

Introducing the FLiPN stack which combines Apache Flink, Apache NiFi, Apache Pulsar and other Apache tools to build fast applications for IoT, AI, rapid ingest. FLiPN provides a quick set of tools to build applications at any scale for any streaming and IoT use cases. Tools Apache Flink, Apache Pulsar, Apache NiFi, MiNiFi, Apache MXNet, DJL.AI References https://www.datainmotion.dev/2019/08/... https://www.datainmotion.dev/2019/09/... https://www.datainmotion.dev/2019/05/... https://www.datainmotion.dev/2019/03/... Get the presentation slides: https://www.slideshare.net/streamnati... Subscribe to the StreamNative Newsletter for Apache Pulsar for more Pulsar content: https://share.hsforms.com/1IS56E-RvSV... Get started with the on-demand Pulsar training by StreamNative Academy: https://www.academy.streamnative.io/

Using the FLiPN Stack for Edge AI (Flink, NiFi, Pulsar) - Pulsar Summit Asia ...

StreamNative

Cloud lunch and learn real-time streaming in azure

Timothy Spann

Big mountain data and dev conference apache pulsar with mqtt for edge compu...

Timothy Spann

What's hot (20)

Stream or segment : what is the best way to access your events in Pulsar_Neng

Code the docs-yu liu

Apache Pulsar at Yahoo! Japan

How Orange Financial combat financial frauds over 50M transactions a day usin...

Transaction preview of Apache Pulsar

Apache Pulsar: A borderless community

Architecture of a Kafka camus infrastructure

Select Star: Flink SQL for Pulsar Folks - Pulsar Summit NA 2021

Open keynote_carolyn&matteo&sijie

Hail hydrate! from stream to lake using open source

Big data conference europe real-time streaming in any and all clouds, hybri...

Five years of operating a large scale globally replicated Pulsar installation...

Real time cloud native open source streaming of any data to apache solr

Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europ...

Kafka on Pulsar

Building a distributed Key-Value store with Cassandra

Matt Franklin - Apache Software (Geekfest)

Using the FLiPN Stack for Edge AI (Flink, NiFi, Pulsar) - Pulsar Summit Asia ...

Cloud lunch and learn real-time streaming in azure

Big mountain data and dev conference apache pulsar with mqtt for edge compu...

Similar to Building event streaming pipelines using Apache Pulsar

Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Spark DevNexus 2022 Atlanta https://devnexus.com/presentations/7150/ This talk is a quick overview of the How, What and WHY of Apache Pulsar, Apache Flink and Apache NiFi. I will show you how to design event-driven applications that scale the cloud native way. This talk was done live in person at DevNexus across from the booth in room 311 Tim Spann Tim Spann is a Developer Advocate for StreamNative. He works with StreamNative Cloud, Apache Pulsar, Apache Flink, Flink SQL, Apache NiFi, MiniFi, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.

Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...

Timothy Spann

Using FLiP with InfluxDB for EdgeAI IoT at Scale 2022 https://adtmag.com/webcasts/2021/12/influxdata-february-10.aspx?tc=page0 Using FLiP with InfluxDB for EdgeAI IoT at Scale Date: Thursday, February 10th at 11am PT / 2pm ET Join this webcast as Timothy from StreamNative takes you on a hands-on deep-dive using Pulsar, Apache NiFi + Edge Flow Manager + MiniFi Agents with Apache MXNet, OpenVino, TensorFlow Lite, and other Deep Learning Libraries on the actual edge devices including Raspberry Pi with Movidius 2, Google Coral TPU and NVidia Jetson Nano. The team runs deep learning models on the edge devices, sends images, and captures real-time GPS and sensor data. Their low-coding IoT applications provide easy edge routing, transformation, data acquisition and alerting before they decide what data to stream in real-time to their data space. These edge applications classify images and sensor readings in real-time at the edge and then send Deep Learning results to Flink SQL and Apache NiFi for transformation, parsing, enrichment, querying, filtering and merging data to InfluxDB. In this session you will learn how to: Build an end-to-end streaming edge app Pull messages from Pulsar topics and persists the messages to InfluxDB Build a data stream for IoT with NiFi and InfluxDB Use Apache Flink + Apache Pulsar Timothy Spann, Developer Advocate, StreamNative Tim Spann is a Developer Advocate at StreamNative where he works with Apache NiFi, MiniFi, Kafka, Apache Flink, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a senior solutions architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.

Using FLiP with InfluxDB for EdgeAI IoT at Scale 2022

Timothy Spann

Using FLiP with influxdb for edgeai iot at scale 2022

Timothy Spann

DBCC 2021 - FLiP Stack for Cloud Data Lakes

Timothy Spann

[AI Dev World 2022] Build ML Enhanced Event Streaming 3:00 PM - 3:25 PM PDT [#VIRTUAL] OPEN TALK (AI): BUILD ML ENHANCED EVENT STREAMING APPLICATIONS WITH JAVA MICROSERVICES VIRTUAL ONLY API WORLD & AI DEVWORLD STAGE VIRTUAL ONLY VIRTUAL APPLIED MACHINE LEARNING Timothy Spann Timothy Spann StreamNative, Developer Advocate In this talk we will walk through how to build event streaming applications as functions running in with cloud native messaging via Apache Pulsar that run on near infinite scale in any cloud, docker or K8. We will show you have to deploy ML functions to transform real-time data for IoT, Streaming Analytics and many other use cases. After this talk you will be able to build Java microservices with ease and deploy them anywhere utilizing the open source unified streaming and messaging platform, Apache Pulsar. Finally, we will show you have to add dashboards with Web Sockets, no code data sinks, integrate with Apache NiFi data pipelines, SQL Reports with Apache Spark and finally continuous ETL with Apache Flink. I have built many of these applications for many organizations as part of the FLiPN Stack. Let's build next generation applications today regardless if your data is REST APIs, Sensors, Logs, NoSQL Sources, Events or Database tables. https://aidevworld.com/conference/conference-tracks/applied-machine-learning/ https://github.com/tspannhw/pulsar-transit-function https://github.com/tspannhw/FLiPStackWeekly/blob/main/README.md

[AI Dev World 2022] Build ML Enhanced Event Streaming

Timothy Spann

Data science online camp using the flipn stack for edge ai (flink, nifi, pu...

Timothy Spann

Using FLaNK with InfluxDB for EdgeAI IoT at Scale Timothy from StreamNative take you on a hands-on deep-dive on using Pulsar, Apache NiFi + Edge Flow Manager + MiniFi Agents with Apache MXNet, OpenVino, TensorFlow Lite, and other Deep Learning Libraries on the actual edge devices including Raspberry Pi with Movidius 2, Google Coral TPU and NVidia Jetson Nano. The team run deep learning models on the edge devices and send images, and capture real-time GPS and sensor data. Their low-coding IoT applications provide easy edge routing, transformation, data acquisition and alerting before they decide what data to stream real-time to their data space. These edge applications classify images and sensor readings real-time at the edge and then send Deep Learning results to Flink SQL and Apache NiFi for transformation, parsing, enrichment, querying, filtering and merging data to InfluxDB.

Timothy Spann [StreamNative] | Using FLaNK with InfluxDB for EdgeAI IoT at Sc...

InfluxData

Using FLiP with influxdb for EdgeAI IoT at Scale

Timothy Spann

Pulsar summit asia 2021 apache pulsar with mqtt for edge computing

Timothy Spann

FLiP Into Trino FLiP into Trino. Flink Pulsar Trino Pulsar SQL (Trino/Presto) Remember the days when you could wait until your batch data load was done and then you could run some simple queries or build stale dashboards? Those days are over, today you need instant analytics as the data is streaming in real-time. You need universal analytics where that data is. I will show you how to do this utilizing the latest cloud native open source tools. In this talk we will utilize Trino, Apache Pulsar, Pulsar SQL and Apache Flink to analyze instantly data from IoT, sensors, transportation systems, Logs, REST endpoints, XML, Images, PDFs, Documents, Text, semistructured data, unstructured data, structured data and a hundred data sources you could never dream of streaming before. I will teach how to use Pulsar SQL to run analytics on live data. Tim Spann Developer Advocate StreamNative David Kjerrumgaard Developer Advocate StreamNative https://www.starburst.io/info/trinosummit/ https://github.com/tspannhw/FLiP-Into-Trino/blob/main/README.md https://github.com/tspannhw/StreamingAnalyticsUsingFlinkSQL/tree/main/src/main/java select * from pulsar."public/default"."weather"; Apache Pulsar plus Trio = fast analytics at scale

FLiP Into Trino

Timothy Spann

Devfest uk & ireland using apache nifi with apache pulsar for fast data on-ramp 2022 As the Pulsar communities grows, more and more connectors will be added. To enhance the availability of sources and sinks and to make use of the greater Apache Streaming community, joining forces between Apache NiFi and Apache Pulsar is a perfect fit. Apache NiFi also adds the benefits of ELT, ETL, data crunching, transformation, validation and batch data processing. Once data is ready to be an event, NiFi can launch it into Pulsar at light speed. I will walk through how to get started, some use cases and demos and answer questions. https://www.devfest-uki.com/schedule https://linktr.ee/tspannhw

Devfest uk & ireland using apache nifi with apache pulsar for fast data on-r...

Timothy Spann

Apache Pulsar is a distributed and open-source pub-sub messaging system. It offers many advantages over Kafka, such as multi-tenant, geo-replication, decoupled storage or even SQL and FaaS directly integrated. The only thing missing for wide adoption is support for the de-facto standard for streaming: Kafka. And this is how our story begins. In this talk, Sijie Guo from StreamNative and Pierre Zemb from OVHcloud will share the journey on building Kafka-on-Pulsar (KoP) to bring native Kafka protocol support to Pulsar. Before joining the force on building KoP, OVHcloud implemented a Kafka proxy in Rust capable of transforming the Kafka protocol to that Pulsar on the fly and encountered some challenges. After realizing that StreamNative was working on bringing the Kafka protocol natively to Pulsar broker via a pluggable protocol handler mechanism. OVHCloud joined forces with StreamNative to work on brining Kafka protocol support to Pulsar brokers. At the end of this talk, you will know more about the inner workings of Kafka and Pulsar. You'll also get feedback from both companies from their initial proofs of concepts and the current implementation.

Kafka on Pulsar:bringing native Kafka protocol support to Pulsar_Sijie&Pierre

StreamNative

Automation + dev ops summit hail hydrate! from stream to lake

Timothy Spann

Current and Future of Apache Kafka

Joe Stein

Activity feeds (and more) at mate1

Hisham Mardam-Bey

Today we will span from edge to any and all clouds to support data collection, real-time streaming, sensor ingest, edge computing, IoT use cases and edge AI. Apache Pulsar allows us to build computing at the edge and produce and consume messages at scale in any IoT, hybrid or cloud environment. Apache Pulsar supports MoP which allows for MQTT protocol to be used for high speed messaging. We will teach you to quickly build scalable open source streaming applications regardless of if you are running in containers, pods, edge devices, VMs, on-premise servers, moving vehicles and any cloud.

Apache Pulsar with MQTT for Edge Computing - Pulsar Summit Asia 2021

StreamNative

The Netflix Way to deal with Big Data Problems

Monal Daxini

Building an Event Streaming Architecture with Apache Pulsar

ScyllaDB

Ai dev world utilizing apache pulsar, apache ni fi and minifi for edgeai io...

Timothy Spann

Netflix keystone streaming data pipeline @scale in the cloud-dbtb-2016

Monal Daxini

Similar to Building event streaming pipelines using Apache Pulsar (20)

Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...

Using FLiP with InfluxDB for EdgeAI IoT at Scale 2022

Using FLiP with influxdb for edgeai iot at scale 2022

DBCC 2021 - FLiP Stack for Cloud Data Lakes

[AI Dev World 2022] Build ML Enhanced Event Streaming

Data science online camp using the flipn stack for edge ai (flink, nifi, pu...

Timothy Spann [StreamNative] | Using FLaNK with InfluxDB for EdgeAI IoT at Sc...

Using FLiP with influxdb for EdgeAI IoT at Scale

Pulsar summit asia 2021 apache pulsar with mqtt for edge computing

FLiP Into Trino

Devfest uk & ireland using apache nifi with apache pulsar for fast data on-r...

Kafka on Pulsar:bringing native Kafka protocol support to Pulsar_Sijie&Pierre

Automation + dev ops summit hail hydrate! from stream to lake

Current and Future of Apache Kafka

Activity feeds (and more) at mate1

Apache Pulsar with MQTT for Edge Computing - Pulsar Summit Asia 2021

The Netflix Way to deal with Big Data Problems

Building an Event Streaming Architecture with Apache Pulsar

Ai dev world utilizing apache pulsar, apache ni fi and minifi for edgeai io...

Netflix keystone streaming data pipeline @scale in the cloud-dbtb-2016

More from StreamNative

So, you are a responsible software engineer building microservices for Apache Kafka, and life is good. Eventually, you hear the community talking about the outstanding experience they are having with Apache Pulsar features. They talk about infinite event stream retention, a rebalance-free architecture, native support for event processing, and multi-tenancy. Exciting, right? Most people would want to migrate their code to Pulsar. Especially when you know that Pulsar also supports Kafka clients natively via the protocol handler known as KoP — which enables the Kafka client APIs on Pulsar. But, as said before, you are responsible; and you don't believe in fairy tales, just like you don't believe that migrations like this happen effortlessly. This session will discuss the architecture behind protocol handlers, what it means having one enabled on Pulsar, and how the KoP works. It will detail the effort required to migrate a microservice written for Kafka to Pulsar, and whether the code need to change for this.

Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022

StreamNative

This talk describes Klaviyo’s internal messaging system, an asynchronous application framework built around Pulsar that provides a set of high-quality tools for building business-critical asynchronous data flows in unreliable environments. This framework includes: a pulsar ORM and schema migrator for topic configuration; a retry/replay system; a versioned schema registry; a consumer framework oriented around preventing message loss and in hostile environments while maximizing observability; an experimental “online schema change” for topics; and more. Development of this system was informed by lessons learned during heavy use of datastores like RabbitMQ and Kafka, and frameworks like Celery, Spark, and Flink. In addition to the capabilities of this system, this talk will also cover (sometimes painful) lessons learned about the process of converting a heterogenous async-computing environment onto Pulsar and a unified model.

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

StreamNative

In this talk, learn how Toast leverages our Envoy control-plane to manage blue-green deploys of Pulsar consumers, and how this has helped drive adoption across the engineering organization. Dive into the history of Pulsar at Toast, starting from its introduction in 2019 to provide event-driven architecture across a rapidly scaling restaurant software platform. We will detail some of the hurdles that we encountered gaining buy-in across a diverse set of teams, and dive deep into how we enforce best practices and integrate with our service control plane.

Blue-green deploys with Pulsar & Envoy in an event-driven microservice ecosys...

StreamNative

Event streaming architectures launched a reexamination of applications and systems architectures across the board. We live in a world where answers are needed now in a constant real-time flow. Yet beyond the event streaming system itself, what are the corequisites to ensure our large scale distributed database systems can keep pace with this always-on, always-current real time flow of data? What are the requirements and expectations for this next tech cycle?

Distributed Database Design Decisions to Support High Performance Event Strea...

StreamNative

Pulsar Functions is a succinct framework provided by Apache Pulsar to conduct real-time data processing. Its use cases include ETL pipeline, event-driven applications, and simple data analytics. While Pulsar Functions already provides an extremely simple programming interface, we want to further lower the barrier for users to access real-time data. Since SQL is one of the universal languages in the technology world and well accepted by the vast majority of data engineers, we decided to add a SQL expressing layer on top of Pulsar Functions runtime. In this talk, we will discuss the architecture and implementation of this new service. We will see how SQL syntax, Pulsar Functions, and Function Mesh can work together to deliver a unique user development experience for real-time data jobs in the cloud environment. We will also walk through use cases like filtering, routing, and projecting messages as well as integrating with the Pulsar IO Connectors framework.

Simplify Pulsar Functions Development with SQL - Pulsar Summit SF 2022

StreamNative

Starting with version 2.10, the Apache ZooKeeper dependency has been eliminated and replaced with a pluggable framework that enables you to reduce the infrastructure footprint of Apache Pulsar by leveraging alternative metadata and coordination systems based on your deployment environment. In this talk, walk through the steps required to utilize the existing etcd service running inside Kubernetes to act as Pulsar's metadata store, thereby eliminating the need to run ZooKeeper entirely, leaving you with a Zookeeper-less Pulsar.

Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022

StreamNative

Apache Pulsar is a highly available, distributed messaging system that provides guarantees of no message loss and strong message ordering with predictable read and write latency. In this talk, learn how this can be validated for Apache Pulsar Kubernetes deployments. Various failures are injected using Chaos Mesh to simulate network and other infrastructure failure conditions. There are many questions that are asked about failure scenarios, but it could be hard to find answers to these important questions. When a failure happens, how long does it take to recover? Does it cause unavailability? How does it impact throughput and latency? Are the guarantees of no message loss and strong message ordering kept, even when components fail? If a complete availability zone fails, is the system configured correctly to handle AZ failures? This talk will help you find answers to these questions and apply the tooling and practices to your own testing and validation.

Validating Apache Pulsar’s Behavior under Failure Conditions - Pulsar Summit ...

StreamNative

Despite what the Ghostbusters said, we’re going to go ahead and cross (or, join) the streams. This session covers getting started with streaming data pipelines, maximizing Pulsar’s messaging system alongside one of the most flexible streaming frameworks available, Apache Flink. Specifically, we’ll demonstrate the use of Flink SQL, which provides various abstractions and allows your pipeline to be language-agnostic. So, if you want to leverage the power of a high-speed, highly customizable stream processing engine without the usual overhead and learning curves of the technologies involved (and their interconnected relationships), then this talk is for you. Watch the step-by-step demo to build a unified batch and streaming pipeline from scratch with Pulsar, via the Flink SQL client. This means you don’t need to be familiar with Flink, (or even a specific programming language). The examples provided are built for highly complex systems, but the talk itself will be accessible to any experience level.

Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...

StreamNative

Apache Pulsar depends upon message acknowledgments to provide at-least-once or exactly-once processing guarantees. With these guarantees, any transmission between the broker and its producers and consumers requires an acknowledgment. But what happens if an acknowledgment is not received? Resending the message introduces the potential of duplicate processing and increases the likelihood of out or order processing. Therefore, it is critical to understand the Pulsar message redelivery semantics in order to prevent either of these conditions. In this talk, we will walk you through the redelivery semantics of Apache Pulsar, and highlight some of the control mechanisms available to application developers to control this behavior. Finally, we will present best practices for configuring message redelivery to suit various use cases.

Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022

StreamNative

Lakehouses are quickly growing in popularity as a new approach to Data Platform Architecture bringing some of the long-established benefits from OLTP world to OLAP, including transactions, record-level updates/deletes, and changes streaming. In this talk, we will discuss Apache Hudi and how it unlocks possibilities of building your own fully open-source Lakehouse featuring a rich set of integrations with existing technologies, including Apache Pulsar. In this session, we will present: - What Lakehouses are, and why they are needed. - What Apache Hudi is and how it works. - Provide a use-case and demo that applies Apache Hudi’s DeltaStreamer tool to ingest data from Apache Pulsar.

Unlocking the Power of Lakehouse Architectures with Apache Pulsar and Apache ...

StreamNative

Pulsar is a horizontally scalable messaging system, so the traffic in a logical cluster must be balanced across all the available Pulsar brokers as evenly as possible, in order to ensure full utilization of the broker layer. You can use multiple settings and tools to control the traffic distribution which requires a bit of context to understand how the traffic is managed in Pulsar. In this talk, we will walk you through the load balancing capabilities of Apache Pulsar, and highlight some of the control mechanisms available to control the distribution of load across the Pulsar brokers. Finally, we will discuss the various loading shedding strategies that are available. At the end of the talk, you will have a better understanding of how Pulsar's broker level auto-balancing works, and how to properly configure it to meet your workload demands.

Understanding Broker Load Balancing - Pulsar Summit SF 2022

StreamNative

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

StreamNative

In today’s world, we are seeing a big shift toward the Cloud. With this shift comes a big shift in the expectations we have for a messaging system, especially when the messaging system is presented as managed service in a large-scale, multi-tenant environment. For any large-scale enterprise, it’s very important to evaluate messaging system and be confident before expanding complex distributed data systems like Apache Pulsar from on-premise to elastically scalable, fully managed services on cloud services. We must consider aspects such as: migration from and integration with large-scale on-premise clusters, security, cost efficiency, and the cloud friendliness of the architecture, modeling cost and capacity, tenant isolation, deployment robustness, availability, monitoring, etc. Not every messaging system is built to be cloud-native and run as a managed service with cost efficiency. We have been running large-scale Apache Pulsar at Yahoo for the last 8 years on various platforms and hardware configurations while meeting application SLAs and serving more than 1M topics in a cluster. In this talk, we will talk about Pulsar’s journey in Yahoo! from an on-premise platform to a hybrid cloud and on-premise system. We will talk about Pulsar’s architecture and features that make Pulsar a good cloud-native messaging-system choice for any enterprise.

Pulsar's Journey in Yahoo!: On-prem, Cloud and Hybrid - Pulsar Summit SF 2022

StreamNative

Pulsar Summit San Francisco is the event dedicated to Apache Pulsar. This one-day, action-packed event will include 5 keynotes, 12 breakout sessions, and 1 amazing happy hour. Speakers are from top companies, including Google, AWS, Databricks, Onehouse, StarTree, Intel, ScyllaDB, and more! It’s the perfect opportunity to network with Pulsar thought leaders in person. Join developers, architects, data engineers, DevOps professionals, and anyone who wants to learn about messaging and event streaming for this one-day, in-person event. Pulsar Summit San Francisco brings the Apache Pulsar Community together to share best practices and discuss the future of streaming technologies.

Event-Driven Applications Done Right - Pulsar Summit SF 2022

StreamNative

Our services team creates, builds, and maintains the as a service offering for base platform services within our organization. Several thousand applications use these custom services daily generating more than 700 million requests per minute. One of these services was our publish / subscriber offering, BQ with custom SDK and custom metrics based on Apache Pulsar. BQ is the core communication service within our organization, having more 200M RPM. All the core processes of the organization depend on this service for operation: the CDC of any of our RDBMS or NoSQL offering, all the eventing efforts of the organization, async communication between apps, notification systems, etc. The backend of the solution was Apache Pulsar running on EC2 on AWS and on top of that we built several components as wrappers of the actual backend, creating our own SDKs and abstractions and in many ways extending the features provided by Pulsar. We had a multi-cluster setup 100% on AWS, with custom Pulsar Docker images running on large ASG setups, along with our own wrapping and admin APIs and DBs. All of this in turn transformed the solution into a volatile solution.

Pulsar @ Scale. 200M RPM and 1K instances - Pulsar Summit SF 2022

StreamNative

There is an increasing need to unleash analytical capabilities directly to the end-users to democratize decision-making. User-Facing Analytics is a new frontier that will shape the products of tomorrow and push the limits of existing technology. It demands a solution that will scale to millions of users to provide fast, real-time insights. In this session, Xiang will talk about his journey to build Apache Pinot to tackle the analytics problem space with the architectural changes and technology inventions made over the past decade. He will also talk about how other big data companies such as LinkedIn, Uber, and Stripe power their user-facing analytical applications.

Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022

StreamNative

Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022

StreamNative

Welcome and Opening Remarks - Pulsar Summit SF 2022

StreamNative

Milvus is an open-source vector database that leverages a novel data fabric to build and manage vector similarity search applications. As the world's most popular vector database, it has already been adopted in production by thousands of companies around the world, including Lucidworks, Shutterstock, and Cloudinary. With the launch of Milvus 2.0, the community aims to introduce a cloud-native, highly scalable and extendable vector similarity solution, and the key design concept is log as data. Milvus relies on Pulsar as the log pub/sub system. Pulsar helps Milvus to reduce system complexity by loosely decoupling each micro service, making the system stateless by disaggregating log storage and computation, which also makes the system further extendable. We will introduce the overview design, the implementation details of Milvus and its roadmap in this topic. Takeaways: 1) Get a general idea about what is a vector database and its real-world use cases. 2) Understand the major design principles of Milvus 2.0. 3) Learn how to build a complex system with the help of a modern log system like Pulsar.

Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...

StreamNative

MQTT (Message Queuing Telemetry Transport,) is a message protocol based on the pub/sub model with the advantages of compact message structure, low resource consumption, and high efficiency, which is suitable for IoT applications with low bandwidth and unstable network environments. This session will introduce MQTT on Pulsar, which allows developers users of MQTT transport protocol to use Apache Pulsar. I will share the architecture, principles and future planning of MoP, to help you understand Apache Pulsar's capabilities and practices in the IoT industry.

MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...

StreamNative

More from StreamNative (20)