Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Kafka on Pulsar (KoP) 翟佳
Who am I? Apache Pulsar Committer & PMC Member Apache BookKeeper Committer & PMC Member EMC -> StreamNative StreamNative C...
What is Apache Pulsar? Flexible Pub/Sub Messaging backed by Durable log/stream Storage
Barrier for user? Unified Messaging Protocol Apps Build on old systems
How Pulsar handles it? Pulsar Kafka Wrapper on Kafka Java API https://pulsar.apache.org/docs/en/adaptors-kafka/ Pulsar IO ...
Kafka on Pulsar (KoP)
KoP Feasibility — Log Topic
KoP Feasibility — Log Topic Producer Consumer
KoP Feasibility — Log Topic Producer Consumer Kafka
KoP Feasibility — Log Topic Producer Consumer Pulsar
KoP Feasibility — Others Producer Consumer Topic Lookup Produce Consume Offset Consumption State
KoP Overview Kafka lib Broker Pulsar Consumer Pulsar lib Load Balancer Pulsar Protocol handler Kafka Protocol handler Puls...
KoP Implementation Topic flat map: Broker sets `kafkaNamespace` Message ID and Offset: LedgerId + EntryId Message: Convert...
KoP Implementation — Topic Map
KoP Implementation — Offset Kafka lib Kafka Producer entryId LedgerId entryIdLedgerId Offset
KoP Implementation — Message Map
KoP Implementation — Topic Lookup
KoP Implementation — Pro/Con
KoP Implementation — Pro/Con
KoP Now Kafka lib Broker Pulsar Consumer Pulsar lib Load Balancer Pulsar Protocol handler Kafka Protocol handler Pulsar Pr...
KoP Now Layered Architecture Independent Scale Instant Recovery Balance-free expand
Ordering Guaranteed ordering Multi-tenancy A single cluster can support many tenants and use cases High throughput Can rea...
Demo https://kafka.apache.org/quickstart Demo1: Kafka Producer / Consumer Demo2: Kafka Connect https://archive.apache.org/...
Demo Kafka lib Broker Pulsar Consumer Pulsar lib Load Balancer Pulsar Protocol handler Kafka Protocol handler Pulsar Produ...
Demo1: K-Producer -> K-Consumer Kafka lib Kafka Consumer Kafka libKafka lib Kafka Producer Broker Pulsar Protocol handler ...
Demo1: P-Producer -> K-Consumer Pulsar Consumer Pulsar lib Pulsar Producer Pulsar lib Kafka lib Kafka Consumer Kafka libKa...
Demo1: P-Producer -> K-Consumer Pulsar Consumer Pulsar lib Pulsar Producer Pulsar lib Kafka lib Kafka Consumer Kafka libKa...
Demo2: Kafka Connect
Demo2: Kafka Connect Kafka lib Kafka File
 Source Broker Pulsar Protocol handler Kafka Protocol handler Pulsar Topic InPut...
Demo2: Pulsar Functions https://pulsar.apache.org/docs/en/functions-overview/
Demo2: Pulsar Functions Kafka lib Kafka File
 Source Broker Pulsar Protocol handler Kafka Protocol handler Pulsar Topic In...
Apache Pulsar & Apache Kafka
Thanks!Stream Native
Thanks!Stream NativeWe are hiring
Kafka on Pulsar
Upcoming SlideShare
Loading in …5
×

Kafka on Pulsar

517 views

Published on

In Apache Pulsar Meetup, Jia Zhai from StreamNative presents KoP (Kafka-on-Pulsar) which bring native Kafka protocol support on Pulsar broker. He gave a demo about how to use Kafka clients and Pulsar clients can work seamlessly on same data, and how Kafka Connectors can work on a Pulsar cluster.

Published in: Internet
no profile picture user

  • Login to see the comments

  • Be the first to like this

Kafka on Pulsar

  1. 1. Kafka on Pulsar (KoP) 翟佳
  2. 2. Who am I? Apache Pulsar Committer & PMC Member Apache BookKeeper Committer & PMC Member EMC -> StreamNative StreamNative Core Engineer HUST -> ICT Jia Zhai / 翟佳
  3. 3. What is Apache Pulsar? Flexible Pub/Sub Messaging backed by Durable log/stream Storage
  4. 4. Barrier for user? Unified Messaging Protocol Apps Build on old systems
  5. 5. How Pulsar handles it? Pulsar Kafka Wrapper on Kafka Java API https://pulsar.apache.org/docs/en/adaptors-kafka/ Pulsar IO Connect https://pulsar.apache.org/docs/en/io-overview/
  6. 6. Kafka on Pulsar (KoP)
  7. 7. KoP Feasibility — Log Topic
  8. 8. KoP Feasibility — Log Topic Producer Consumer
  9. 9. KoP Feasibility — Log Topic Producer Consumer Kafka
  10. 10. KoP Feasibility — Log Topic Producer Consumer Pulsar
  11. 11. KoP Feasibility — Others Producer Consumer Topic Lookup Produce Consume Offset Consumption State
  12. 12. KoP Overview Kafka lib Broker Pulsar Consumer Pulsar lib Load Balancer Pulsar Protocol handler Kafka Protocol handler Pulsar Producer Pulsar lib Kafka Producer Kafka lib Kafka Consumer Kafka lib Kafka Producer Managed Ledger BK Client Geo- Replicator Pulsar Topic ZooKeeper Bookie Pulsar
  13. 13. KoP Implementation Topic flat map: Broker sets `kafkaNamespace` Message ID and Offset: LedgerId + EntryId Message: Convert Key/value/timestamp/headers(properties) Topic Lookup: Pulsar admin topic lookup -> owner broker Produce: Convert, then call PulsarTopic.publishMessage Consume: Convert, then call non-durable-cursor.readEntries Group Coordinator: Keep in topic `public/__kafka/__offsets`
  14. 14. KoP Implementation — Topic Map
  15. 15. KoP Implementation — Offset Kafka lib Kafka Producer entryId LedgerId entryIdLedgerId Offset
  16. 16. KoP Implementation — Message Map
  17. 17. KoP Implementation — Topic Lookup
  18. 18. KoP Implementation — Pro/Con
  19. 19. KoP Implementation — Pro/Con
  20. 20. KoP Now Kafka lib Broker Pulsar Consumer Pulsar lib Load Balancer Pulsar Protocol handler Kafka Protocol handler Pulsar Producer Pulsar lib Kafka Producer Kafka lib Kafka Consumer Kafka lib Kafka Producer Managed Ledger BK Client Geo- Replicator Pulsar Topic ZooKeeper Bookie Pulsar
  21. 21. KoP Now Layered Architecture Independent Scale Instant Recovery Balance-free expand
  22. 22. Ordering Guaranteed ordering Multi-tenancy A single cluster can support many tenants and use cases High throughput Can reach 1.8 M messages/s in a single partition Durability Data replicated and synced to disk Geo-replication Out of box support for geographically distributed applications Unified messaging model Support both Streaming and Queuing Delivery Guarantees At least once, at most once and effectively once Low Latency Low publish latency of 5ms Highly scalable & available Can support millions of topics HA KoP Now
  23. 23. Demo https://kafka.apache.org/quickstart Demo1: Kafka Producer / Consumer Demo2: Kafka Connect https://archive.apache.org/dist/kafka/2.0.0/ kafka_2.12-2.0.0.tgz
  24. 24. Demo Kafka lib Broker Pulsar Consumer Pulsar lib Load Balancer Pulsar Protocol handler Kafka Protocol handler Pulsar Producer Pulsar lib Kafka Producer Kafka lib Kafka Consumer Kafka lib Kafka Producer Managed Ledger BK Client Geo- Replicator Pulsar Topic ZooKeeper Bookie Pulsar
  25. 25. Demo1: K-Producer -> K-Consumer Kafka lib Kafka Consumer Kafka libKafka lib Kafka Producer Broker Pulsar Protocol handler Kafka Protocol handler Pulsar Topic bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
  26. 26. Demo1: P-Producer -> K-Consumer Pulsar Consumer Pulsar lib Pulsar Producer Pulsar lib Kafka lib Kafka Consumer Kafka libKafka lib Kafka Producer Broker Pulsar Protocol handler Kafka Protocol handler Pulsar Topic bin/pulsar-client produce test -n 1 -m “Hello from Pulsar Producer, Message 1” bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
  27. 27. Demo1: P-Producer -> K-Consumer Pulsar Consumer Pulsar lib Pulsar Producer Pulsar lib Kafka lib Kafka Consumer Kafka libKafka lib Kafka Producer Broker Pulsar Protocol handler Kafka Protocol handler Pulsar Topic bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test bin/pulsar-client consume -s sub-name test -n 0
  28. 28. Demo2: Kafka Connect
  29. 29. Demo2: Kafka Connect Kafka lib Kafka File
 Source Broker Pulsar Protocol handler Kafka Protocol handler Pulsar Topic InPut File Kafka File
 Sink OutPut File TOPIC bin/connect-standalone.sh 
 config/connect-standalone.properties 
 config/connect-file-source.properties 
 config/connect-file-sink.properties
  30. 30. Demo2: Pulsar Functions https://pulsar.apache.org/docs/en/functions-overview/
  31. 31. Demo2: Pulsar Functions Kafka lib Kafka File
 Source Broker Pulsar Protocol handler Kafka Protocol handler Pulsar Topic InPut File Kafka File
 Sink OutPut File TOPIC Kafka lib Pulsar 
 Functions OutPut Topic bin/pulsar-admin functions localrun --name pulsarExclamation
 --jar pulsar-functions-api-examples.jar 
 --classname org…ExclamationFunction
 --inputs connect-test-partition-0 --output out-hello
  32. 32. Apache Pulsar & Apache Kafka
  33. 33. Thanks!Stream Native
  34. 34. Thanks!Stream NativeWe are hiring

×