Download as PDF, PPTX
Uploaded byTimothy Spann
The Next Generation of Streaming
The document provides an overview of next-generation streaming technologies, focusing on key developers such as Tim Spann and David Kjerrumgaard. It highlights the capabilities and features of the Apache Nifi and Pulsar platforms, including data delivery, buffering, quality of service, and support for various data protocols. Additionally, it references resources for further learning and emphasizes the Apache Software Foundation's trademarks.
More Related Content
![[March sn meetup] apache pulsar + apache nifi for cloud data lake](https://cdn.slidesharecdn.com/ss_thumbnails/marchsnmeetupapachepulsarapachenififorclouddatalake-220311125943-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to The Next Generation of Streaming
![[AerospikeRoadshow] Apache Pulsar Unifies Streaming and Messaging for Real-Ti...](https://cdn.slidesharecdn.com/ss_thumbnails/aerospikeroadshowapachepulsarunifiesstreamingandmessagingforreal-timedata2-221103174214-18ce3fa2-thumbnail.jpg?width=640&height=640&fit=bounds)
![[AI Dev World 2022] Build ML Enhanced Event Streaming](https://cdn.slidesharecdn.com/ss_thumbnails/aidevworld2022buildmlenhancedeventstreaming-221102182253-edfc66ae-thumbnail.jpg?width=640&height=640&fit=bounds)
The Next Generation of Streaming
- 1. The Next Generation ofStreaming Tim Spann Developer Advocate
- 2. Tim Spann Developer Advocate ●FLiP(N) Stack = Flink, Pulsar and NiFi Stack ● Streaming Systems/ Data Architect ● Experience: ○ 15+ years of experience with batch and streaming technologies including Pulsar, Kafka, Flink, Spark, NiFi, Spring, Java, Big Data, Cloud, MXNet, Hadoop, Datalakes, IoT and more.
- 3. David Kjerrumgaard ● ApachePulsar Committer | Author of Pulsar In Action ● Former Principal Software Engineer on Splunk’s messaging team that is responsible for Splunk’s internal Pulsar-as-a-Service platform. ● Former Director of Solution Architecture at Streamlio. Your photo goes here, smile :)
- 5. • Guaranteed delivery •Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Supports push and pull models • Hundreds of processors • Visual command and control • Over a sixty sources • Flow templates • Pluggable/multi-role security • Designed for extension • Clustering • Version Control DataFlows for Data Ingest, Movement and Routing
- 6. • Moving Binary,Unstructured, Image and Tabular Data • Enrichment • Universal Visual Processor • Simple Event Processor • Routing • Feeding data to Central Messaging • Support for modern protocols • Kafka Protocol Source/Sink • Pulsar Protocol Source/Sink The Power of Apache NiFi
- 7.
- 10.
- 11. Streaming Consumer Consumer Consumer Subscription Shared Failover Consumer Consumer Subscription In case offailure in Consumer B-0 Consumer Consumer Subscription Exclusive X Consumer Consumer Key-Shared Subscription Pulsar Topic/Partition Messaging 11
- 12. Kafka On Pulsar (KoP) Kafka Protocol(Binary Data over TCP)
- 13. Schema Registry forSchema Support Schema Registry schema-1 (value=Avro/Protobuf/JSON) schema-2 (value=Avro/Protobuf/JSON) schema-3 (value=Avro/Protobuf/JSON) Schema Data ID Local Cache for Schemas + Schema Data ID + Local Cache for Schemas Send schema-1 (value=Avro/Protobuf/JSON) data serialized per schema ID Send (register) schema (if not in local cache) Read schema-1 (value=Avro/Protobuf/JSON) data deserialized per schema ID Get schema by ID (if not in local cache) Producers Consumers
- 14. Data Offloaders (Tiered Storage) ClientLibraries Apache Pulsar ecosystem hub.streamnative.io Connectors (Sources & Sinks) Protocol Handlers Pulsar Functions (Lightweight Stream Processing) Processing Engines … and more! … and more!
- 15. Apache NiFi -Apache Pulsar Connector 15 https://github.com/streamnative/pulsar-nifi-bundle
- 16.
- 17.
- 18. FLiP Stack Weekly Thisweek in Apache Kafka, Apache Flink, Apache Pulsar, Apache NiFi, Apache Spark, Java and Open Source friends. https://bit.ly/32dAJft
- 19. 19 Apache Pulsar in Action Please enjoyDavid’s complete book which is the ultimate guide to Pulsar.
- 20.
- 21. Notices Apache Pulsar™ Apache®, ApachePulsar™, Pulsar™, Apache Flink®, Flink®, Apache Spark®, Spark®, Apache NiFi®, NiFi® and the logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks. Copyright © 2021-2022 The Apache Software Foundation. All Rights Reserved. Apache, Apache Pulsar and the Apache feather logo are trademarks of The Apache Software Foundation.