Download to read offline
Uploaded byStreamNative
How Zhaopin contributes to Pulsar community
This document discusses Apache Pulsar usage in Zhaopin and some key features: 1. It provides an overview of how Pulsar is used in Zhaopin and the increasing message throughput over time. 2. It describes several Pulsar features in detail, including key-shared subscriptions, schema versioning, HDFS offloading, and upcoming topics like policies and sticky consumers. 3. It discusses the Pulsar community contributions from the Zhaopin team, including details on key-shared subscriptions, schema version handling, HDFS offloader storage, and other improvements.
More Related Content
Similar to How Zhaopin contributes to Pulsar community
How Zhaopin contributes to Pulsar community
- 2. PHOTO Zhaopin in Pulsarcommunity Penghui Li 李李鹏辉 Messaging platform leader in zhaopin.com Apache Pulsar Committer
- 3. Our team Penghui LiBo Cong
- 4. Apache Pulsar inzhaopin.com 2018/08 First service for online applications 2018/10 1 billion / day 2019/02 6 billion / day 2019/08 20 billion / day 50+ Namespaces 5000+ Topics
- 5. 1. Features ofzhaopin contributing to the community 2. Details of Key_shared subscription 3. Release Pulsar 4. Details of Pulsar multiple schema version 5. Details of HDFS Offloader
- 6. Dead letter topic TopicTopic-DLQConsumer … 2 1 0 1 message 1 process failed so many times
- 7. Client interceptors TopicProducer Consumer Send SendAck Receive Acknowledge
- 8. Time partitioned un-ackmessage tracker p-4 p-3 p-2 p-1 p-0 Current partition Timeout partition Add messages to tracker Send redelivery request
- 9. Message redelivery optimization 73 2 06 5 4 1 08 Consumer internal queue ConsumerBroker 0 1 2 3 4 5 6 7 0
- 10. Key_shared subscription A newsubscription mode in 2.4.0 Producer 1 Producer 2 Pulsar topic <k1,v0> <k2,v1> <k3,v2> <k2,v3> <k1,v4> Subscription Consumer D-1 Consumer D-2 Consumer D-3 <k1,v0> <k1,v4> <k3,v2> <k2,v1> <k2,v3>
- 11. Start with Key_sharedsubscription Consumer consumer = client.newConsumer() .topic(“my-topic”) .subscriptionName(“my-sub”) .subscriptionType(SubscriptionType.Key_Shared) .subscribe()
- 12. How Key_shared subscriptionworks Sticky key dispatcher(auto split hash range) 0 65536 Consumer-1 0 1 2 3 4 5 6 7 8 9
- 13. How Key_shared subscriptionworks Sticky key dispatcher(auto split hash range) 0 65536 Consumer-1 0 1 2 3 4 5 6 7 8 9 Consumer-2
- 14. How Key_shared subscriptionworks Sticky key dispatcher(auto split hash range) 0 65536 Consumer-1 0 1 2 3 4 5 6 7 8 9 Consumer-2 Consumer-3
- 15. How Key_shared subscriptionworks Sticky key dispatcher(auto split hash range) 0 65536 Consumer-1 0 1 2 3 4 5 6 7 8 9 Consumer-4
- 16. How Key_shared subscriptionworks Sticky key dispatcher(auto split hash range) 0 65536 Consumer-1 0 1 2 3 4 5 6 7 8 9 Consumer-3Consumer-4
- 17. Key-based message batcher p-0p-1 p-2 <k3,v0> <k2,v0> <k2,v1> <k3,v1> <k1,v1> <k4,v0> <k5,v0> <k6,v0> <k6,v1>
- 18. Key-based message batcher p-0p-1 p-2 <k3,v0> <k2,v0> <k2,v1> <k3,v1> <k1,v1> <k4,v0> <k5,v0> <k6,v0> <k6,v1>
- 19. Use Key-based messagebatcher Consumer consumer = client.newConsumer() .topic(“my-topic”) .subscriptionName(“my-sub”) .subscriptionType(SubscriptionType.Key_Shared) .batcherBuilder(BatcherBuilder.KEY_BASED) .subscribe()
- 20. Pulsar SQL improvements ✓Namespace delimiter rewriter ✓ Partition as internal column ✓ Primitive schema handle ➡ Multiple version schemas handle
- 21. Some other improvements ✓Service URL provider ✓ Consumer reconnect limiter ➡ Messages batch receive
- 22. Next ★ Topic levelpolicy ★ Sticky consumer
- 23. 2.4.0 Release 1. Newbranch and tag 2. Stage release (check -> sign -> stage) 3. Move master to new version and write release notes 4. Start vote 5. Promote release and publish 6. Update site and announce the release
- 24. PHOTO Schema versioning &HDFS offloader Bo Cong 丛搏 Message platform engineer in zhaopin.com Apache Pulsar contributor
- 25. The meaning ofmulti-version schema message 1 message 2 message 3 message 4 message 5 Message's schema is not immutable version 0 version 1 version 2 version 3 version 4schema
- 26. Problems caused byversion changes Class Person { Int id; } Class Person { Int id; @AvroDefault(""Zhang San"") String name; } Class Person { Int id; String name; } Version 0 Version 2 Version 1 Can read Can readCan’t read
- 27. Change in compatibilitypolicy Back Ward Back Ward Transitive version 2 version 1 version 0 can read can read version 2 version 1 version 0 can read can read can read
- 28. Schema creation process adminclient api admin rest api producer create consumer subscribe schema data SchemaRegistryService new schema old schema version compatibility check Incompatible version
- 29. Multi-version use inpulsar Avro schema message 1 version 0 message 2 version 1 message 3 version 2 version 3 consumer
- 30. SchemaInfoProvider Message exist new AvroReader() Multi-version usein pulsar Avro schema new ReflectDatumReader<>(writerSchema, readerSchema) ReaderCache Version0 read not exist find schema by version 0 from broker read If the read and write schema is different in the Avro schema, the reader needs to generate the corresponding read and write schema.
- 31. Multi-version use inpulsar Auto consume schema only support AvroSchema and JsonSchema GenericAvroRecord GenericJsonRecord getField unlike JsonSchema or AvroSchema, the reader only needs writerSchema. Consumer<GenericRecord> consumer = client .newConsumer(Schema.AUTO_CONSUME()) .topic("test") .subscriptionName("test") .subscribe();
- 32. The use ofschema definition Class Person { @Nullable String name; } SchemaDefinition<Person> schemaDefinition = SchemaDefinition.<Person>builder() .withAlwaysAllowNull(false) .withPojo(Person.class).build(); Producer<Person> producer = null; producer = client .newProducer(Schema.AVRO(schemaDefinition)) .topic("persistent://public/default/test") .create();
- 33. Why do weneed HDFS offloader Bookeeper HDFS ManagedLedger (Broker) •Cold and Heat Data Separation SSD HDD High throughput Low latency Massive data storage
- 34. Offload topic ledgersto HDFS stored relative path tenant/namespace/topic/ledgerId + "-" + uuid.toString() topic ledger 1 ledger 2 ledger 3 index data index data index data
- 35. HDFS Offloader storagestructure •Storage mode use org.apache.hadoop.io.MapFile Index Data entryID entryData entryID entryData entryID entryData entryID entryID entryID entryID entryID entryID entryID
- 36.
- 37.