Kafka on Pulsar

1. Kafka on Pulsar (KoP) 翟佳

2. Who am I? Apache Pulsar Committer & PMC Member Apache BookKeeper Committer & PMC Member EMC -> StreamNative StreamNative Core Engineer HUST -> ICT Jia Zhai / 翟佳

3. What is Apache Pulsar? Flexible Pub/Sub Messaging backed by Durable log/stream Storage

4. Barrier for user? Unified Messaging Protocol Apps Build on old systems

5. How Pulsar handles it? Pulsar Kafka Wrapper on Kafka Java API https://pulsar.apache.org/docs/en/adaptors-kafka/ Pulsar IO Connect https://pulsar.apache.org/docs/en/io-overview/

6. Kafka on Pulsar (KoP)

7. KoP Feasibility — Log Topic

8. KoP Feasibility — Log Topic Producer Consumer

9. KoP Feasibility — Log Topic Producer Consumer Kafka

10. KoP Feasibility — Log Topic Producer Consumer Pulsar

11. KoP Feasibility — Others Producer Consumer Topic Lookup Produce Consume Offset Consumption State

12. KoP Overview Kafka lib Broker Pulsar Consumer Pulsar lib Load Balancer Pulsar Protocol handler Kafka Protocol handler Pulsar Producer Pulsar lib Kafka Producer Kafka lib Kafka Consumer Kafka lib Kafka Producer Managed Ledger BK Client Geo- Replicator Pulsar Topic ZooKeeper Bookie Pulsar

13. KoP Implementation Topic flat map: Broker sets `kafkaNamespace` Message ID and Offset: LedgerId + EntryId Message: Convert Key/value/timestamp/headers(properties) Topic Lookup: Pulsar admin topic lookup -> owner broker Produce: Convert, then call PulsarTopic.publishMessage Consume: Convert, then call non-durable-cursor.readEntries Group Coordinator: Keep in topic `public/__kafka/__offsets`

14. KoP Implementation — Topic Map

15. KoP Implementation — Offset Kafka lib Kafka Producer entryId LedgerId entryIdLedgerId Offset

16. KoP Implementation — Message Map

17. KoP Implementation — Topic Lookup

18. KoP Implementation — Pro/Con

19. KoP Implementation — Pro/Con

20. KoP Now Kafka lib Broker Pulsar Consumer Pulsar lib Load Balancer Pulsar Protocol handler Kafka Protocol handler Pulsar Producer Pulsar lib Kafka Producer Kafka lib Kafka Consumer Kafka lib Kafka Producer Managed Ledger BK Client Geo- Replicator Pulsar Topic ZooKeeper Bookie Pulsar

21. KoP Now Layered Architecture Independent Scale Instant Recovery Balance-free expand

22. Ordering Guaranteed ordering Multi-tenancy A single cluster can support many tenants and use cases High throughput Can reach 1.8 M messages/s in a single partition Durability Data replicated and synced to disk Geo-replication Out of box support for geographically distributed applications Uniﬁed messaging model Support both Streaming and Queuing Delivery Guarantees At least once, at most once and effectively once Low Latency Low publish latency of 5ms Highly scalable & available Can support millions of topics HA KoP Now

23. Demo https://kafka.apache.org/quickstart Demo1: Kafka Producer / Consumer Demo2: Kafka Connect https://archive.apache.org/dist/kafka/2.0.0/ kafka_2.12-2.0.0.tgz

24. Demo Kafka lib Broker Pulsar Consumer Pulsar lib Load Balancer Pulsar Protocol handler Kafka Protocol handler Pulsar Producer Pulsar lib Kafka Producer Kafka lib Kafka Consumer Kafka lib Kafka Producer Managed Ledger BK Client Geo- Replicator Pulsar Topic ZooKeeper Bookie Pulsar

25. Demo1: K-Producer -> K-Consumer Kafka lib Kafka Consumer Kafka libKafka lib Kafka Producer Broker Pulsar Protocol handler Kafka Protocol handler Pulsar Topic bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

26. Demo1: P-Producer -> K-Consumer Pulsar Consumer Pulsar lib Pulsar Producer Pulsar lib Kafka lib Kafka Consumer Kafka libKafka lib Kafka Producer Broker Pulsar Protocol handler Kafka Protocol handler Pulsar Topic bin/pulsar-client produce test -n 1 -m “Hello from Pulsar Producer, Message 1” bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

27. Demo1: P-Producer -> K-Consumer Pulsar Consumer Pulsar lib Pulsar Producer Pulsar lib Kafka lib Kafka Consumer Kafka libKafka lib Kafka Producer Broker Pulsar Protocol handler Kafka Protocol handler Pulsar Topic bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test bin/pulsar-client consume -s sub-name test -n 0

28. Demo2: Kafka Connect

29. Demo2: Kafka Connect Kafka lib Kafka File  Source Broker Pulsar Protocol handler Kafka Protocol handler Pulsar Topic InPut File Kafka File  Sink OutPut File TOPIC bin/connect-standalone.sh   config/connect-standalone.properties   config/connect-file-source.properties   config/connect-file-sink.properties

30. Demo2: Pulsar Functions https://pulsar.apache.org/docs/en/functions-overview/

31. Demo2: Pulsar Functions Kafka lib Kafka File  Source Broker Pulsar Protocol handler Kafka Protocol handler Pulsar Topic InPut File Kafka File  Sink OutPut File TOPIC Kafka lib Pulsar   Functions OutPut Topic bin/pulsar-admin functions localrun --name pulsarExclamation  --jar pulsar-functions-api-examples.jar   --classname org…ExclamationFunction  --inputs connect-test-partition-0 --output out-hello

32. Apache Pulsar & Apache Kafka

33. Thanks!Stream Native

34. Thanks!Stream NativeWe are hiring

Kafka on Pulsar

You are reading a preview.

Check these out next

Kafka on Pulsar

More Related Content

Slideshows for you (20)

Similar to Kafka on Pulsar (20)

More from StreamNative (20)

Recently uploaded (20)

Kafka on Pulsar

Kafka on Pulsar

You are reading a preview.

Kafka on Pulsar

Recommended

More Related Content

Slideshows for you (20)

Similar to Kafka on Pulsar (20)

More from StreamNative (20)

Recently uploaded (20)

Kafka on Pulsar