[Meetup] Unified Messaging and Streaming
https://medium.com/@tspann/tracking-aircraft-in-real-time-with-open-source-554124125011 Title: Unified Messaging & Data Streaming Utilizing the Open-Source FLiP Stack we can track aircraft overhead with ease! It does require a little bit of hardware and some Python magic. In this talk, you will learn how to build your own at home with a few bits of hardware and some easy open-source software." Tim Spann is a Developer Advocate for StreamNative. He works with StreamNative Cloud, Apache Pulsar, Apache Flink, Flink SQL, Apache NiFi, MiniFi, Apache MXNet, TensorFlow, Apache Spark, Big Data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science. David Kjerrumgaard is Developer Advocate for StreamNative & Author of "Pulsar in Action" David is a committer on the Apache Pulsar project, and also the author of "Pulsar in Action" and co-author of "Practical Hive". He currently serves as a Developer Advocate for StreamNative where he focuses on strengthening the Apache Pulsar community through education and evangelization. Prior to that he was a principal software engineer on the messaging team at Splunk, and Director of Solutions for two Big Data startups; Streamlio and Hortonworks.
Recommended
Recommended
More Related Content
More from Timothy Spann
More from Timothy Spann (20)
[Meetup] Unified Messaging and Streaming
- 2. ● Apache Pulsar Committer | Author of Pulsar In Action ● Former Principal Software Engineer on Splunk’s messaging team that is responsible for Splunk’s internal Pulsar-as-a-Service platform. ● Former Director of Solution Architecture at Streamlio. David Kjerrumgaard Developer Advocate
- 3. Tim Spann Developer Advocate Tim Spann, Developer Advocate at StreamNative ● FLiP(N) Stack = Flink, Pulsar and NiFI Stack ● Streaming Systems & Data Architecture Expert ● Experience: ○ 15+ years of experience with streaming technologies including Pulsar, Flink, Spark, NiFi, Big Data, Cloud, MXNet, IoT, Python and more. ○ Today, he helps to grow the Pulsar community sharing rich technical knowledge and experience at both global conferences and through individual conversations.
- 4. Apache Pulsar is a Cloud-Native Messaging and Event-Streaming Platform.
- 6. API Model Pulsar’s core API design supports both messaging and streaming use cases with a unified messaging model. It combines the capabilities of systems like RabbitMQ or ActiveMQ, but is built on top and exposes the capabilities of a distributed log, like Apache Kafka or Kinesis.
- 7. Messaging Ideal for work queues that do not require tasks to be performed in a particular order—for example, sending one email message to many recipients. RabbitMQ and Amazon SQS are examples of popular queue-based message systems. Pulsar: Unified Messaging + Data Streaming
- 8. Messaging Ideal for work queues that do not require tasks to be performed in a particular order—for example, sending one email message to many recipients. RabbitMQ and Amazon SQS are examples of popular queue-based message systems. Pulsar: Unified Messaging + Data Streaming .. and Streaming Works best in situations where the order of messages is important—for example, data ingestion. Kafka and Amazon Kinesis are examples of messaging systems that use streaming semantics for consuming messages.
- 9. Messaging vs. Streaming #1 Message Queuing #2 Data Streaming ● Not built for the cloud ● Single tenant systems ● Monolithic architecture couples compute with storage ● Lack of geo replication support
- 10. Pulsar Subscription Modes Different subscription modes have different semantics: Exclusive/Failover - guaranteed order, single active consumer Shared - multiple active consumers, no order Key_Shared - multiple active consumers, order for given key Producer 1 Producer 2 Pulsar Topic Subscription D Consumer D-1 Consumer D-2 Key-Shared < K 1, V 10 > < K 1, V 11 > < K 1, V 12 > < K 2 ,V 2 0 > < K 2 ,V 2 1> < K 2 ,V 2 2 > Subscription C Consumer C-1 Consumer C-2 Shared < K 1, V 10 > < K 2, V 21 > < K 1, V 12 > < K 2 ,V 2 0 > < K 1, V 11 > < K 2 ,V 2 2 > Subscription A Consumer A Exclusive Subscription B Consumer B-1 Consumer B-2 In case of failure in Consumer B-1 Failover
- 11. Streaming Consumer Consumer Consumer Subscription Shared Failover Consumer Consumer Subscription In case of failure in Consumer B-0 Consumer Consumer Subscription Exclusive X Consumer Consumer Key-Shared Subscription Pulsar Topic/Partition Messaging
- 13. ● “Bookies” ● Stores messages and cursors ● Messages are grouped in segments/ledgers ● A group of bookies form an “ensemble” to store a ledger ● “Brokers” ● Handles message routing and connections ● Stateless, but with caches ● Automatic load-balancing ● Topics are composed of multiple segments ● ● Stores metadata for both Pulsar and BookKeeper ● Service discovery Store Messages Metadata & Service Discovery Metadata & Service Discovery Pulsar Architecture MetaData Storage
- 15. Pulsar is built for easy scale-out. *Illustrations by Jack Vanlightly
- 16. Connectivity • Functions - Lightweight Stream Processing (Java, Python, Go) • Connectors - Sources & Sinks (InfluxDB, Kafka, S3, Kinesis, Lambda, …) • Protocol Handlers - AoP (AMQP), KoP (Kafka), MoP (MQTT) • Processing Engines - Flink, Spark, Presto/Trino via Pulsar SQL • Data Offloaders - Tiered Storage - (S3) hub.streamnative.io
- 17. Moving Data In and Out of Pulsar IO/Connectors are a simple way to integrate with external systems and move data in and out of Pulsar. ● Built on top of Pulsar Functions ● Built-in connectors - hub.streamnative.io Source Sink
- 18. Aerospike <-> Apache Pulsar
- 19. Aerospike ← Apache Pulsar
- 20. Aerospike -> Apache Pulsar
- 21. Today’s Quick Aircraft Data Demo
- 27. FLiP Stack Weekly This week in Apache Flink, Apache Pulsar, Apache NiFi, Apache Spark and open source friends. https://bit.ly/32dAJft
- 28. ● https://docs.aerospike.com/connect/pulsar/from-asdb ● https://docs.aerospike.com/connect/pulsar/to-asdb/from-pu lsar-to-asdb-overview ● https://pulsar.apache.org/docs/io-aerospike-sink/ ● https://aerospike.com/products/connect-for-pulsar/ Aerospike <-> Apache Pulsar Resources
- 31. StreamNative: By the Creators Of Apache Pulsar ✓ Original creators of Apache Pulsar & BookKeeper ✓ Operated the largest Pulsar/BookKeeper cluster ✓ Data veterans with extensive industry experience CONFIDENTIAL. DO NOT SHARE. ASF Member Pulsar/BookKeeper PMC Founder and CEO Sijie Guo ASF Member Pulsar/BookKeeper PMC CTO Matteo Merli Pulsar/BookKeeper PMC Co-Founder Jia Zhai
- 32. Pulsar ranked as a Top 5 ASF Project > 580+ Contributors > 10,000+ Commits > 8,500+ Slack Members > 1,000+ Organizations using Pulsar
- 33. Pulsar has already been deployed by thousands of companies across the globe
- 34. Q&A Additional Resources ● For a first look at Pulsar benchmark report, share your email in the chat ● Join the Pulsar Slack channel - Apache-Pulsar.slack.com ● Follow @streamnativeio and @apache_pulsar on Twitter ● Contact StreamNative Sales - doug@streamnative.io