Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Zhaopin's enterprise event center on apache pulsar

186 views

Published on

Zhaopin.com is a Chinese online recruitment services provider. As a bilingual job board, Zhaopin.com has one of the largest selections of real-time job vacancies in China. All recruitment advertisements are provided by prominent Chinese and foreign companies and enterprises throughout China. Zhaopin.com provides professional HR services to over 2.2 million clients and its average daily page views are over 68 million.nApache Pulsar was developed to address several shortcomings of existing messaging systems including a lot of enterprise features, message durability, and lower message latency. Zhaopin.com had built its enterprise event bus By using RabbitMQ for years. As the company grows, the amount of its data is getting larger and larger, and the usage scenarios of message systems are also become varied. The original RabbitMQ based architecture was hard to afford. Ultimately, it chose Apache Pulsar to replace RabbitMQ based architecture in 2018. Since Apache Pulsar was deployed at August 2018, The data amount in Pulsar is increased from 30 million messages per day to 6 billion+ messages. By learning from a lot of practical experiences, they summarised the best practices to using Pulsar as an enterprise event bus. Penghui and Jia detail how Apache Pulsar meet the requirements from Zhaopin.com. How Apache Pulsar is used in Zhaoping.com, and What is the best practices to using Pulsar as an enterprise event bus. Along the way, they will highlight the advantages of Apache Pulsar over the old system, and how high durability, high throughput and low latency of Apache Pulsar make it ideally suited for enterprise event bus.

Published in: Internet
  • Be the first to comment

  • Be the first to like this

Building Zhaopin's enterprise event center on apache pulsar

  1. 1. Building Zhaopin’s enterprise Event Center on Apache Pulsar Penghui Li (@lipenghui6) Jia Zhai (@Jia_Zhai)
  2. 2. Zhaopin.com Zhaopin.com is the biggest online recruitment service provider in China Zhaopin.com provides job seekers a comprehensive resume service, latest deployment, and career development related information, as well as in-depth online job search for positions throughout China. Zhaopin.com provides professional HR services to over 2.2 million clients and its average daily pageviews are over 68 millions.
  3. 3. Who we are ❏ Penghui Li ❏ Tech lead of Infrastructure team at Zhaopin ❏ 5+ years of experiences in messaging and microservices ❏ Apache Pulsar Committer
  4. 4. Who we are ❏ Jia Zhai ❏ Pulsar PMC Member / Committer ❏ BookKeeper PMC Member / Committer ❏ Funding engineer at StreamNative
  5. 5. Agenda ❏ Why building an Event Center ❏ Why Apache Pulsar ❏ Apache Pulsar at Zhaopin ❏ Streaming Platform ❏ Zhaopin’s contributions to Apache Pulsar
  6. 6. Why building an Event Center Data Silos -> Unified Platform
  7. 7. Data Silos ❏ High Maintenance Cost ❏ Extremely hard to scale data cross teams ❏ Inconsistency between data silos ❏ Doesn’t scale ❏ No consistent SLA Pain Points To Enterprises MSMQ Data Processing Kafka To End Users RabbitMQ
  8. 8. Data Silos ❏ High Maintenance Cost ❏ Extremely hard to scale data cross teams ❏ Inconsistency between data silos ❏ Doesn’t scale ❏ No consistent SLA Pain Points To Enterprises MSMQ Data Processing Kafka To End Users RabbitMQ
  9. 9. Unification - MQService ❏ Simplified Operations ❏ Scale-out Service ❏ High Availability Problem Solved Problem Unsolved ❏ Keep messages for longer period ❏ Data rewind ❏ Order guarantee
  10. 10. Unification - MQService Online Services MQService Data Processing Kafka
  11. 11. Why building an Event Center
  12. 12. Why building an Event Center
  13. 13. Why building an Event Center
  14. 14. Why building an Event Center
  15. 15. Why Apache Pulsar Pulsar == Messaging + Storage
  16. 16. Why Apache Pulsar? Flexible Pub/Sub Messaging backed by scalable log storage
  17. 17. Why Apache Pulsar / Multi Tenancy
  18. 18. Why Apache Pulsar / Queuing + Streaming
  19. 19. Why Apache Pulsar / Cloud Native Architecture
  20. 20. Why Apache Pulsar
  21. 21. Apache Pulsar at Zhaopin 20+ core services, 20 billions events/day
  22. 22. Unification - MQService ❏ No Data Silos ❏ Queue + Streaming ❏ Disaster Recovery ❏ Infinite Stream Storage (via Tiered Storage) ❏ Data rewind Problem Solved
  23. 23. Milestones
  24. 24. Core Metrics ❏ 50+ Namespaces ❏ 5000+ Topics ❏ 20+ billions events/day ❏ 5TB storage per day ❏ 20+ core services
  25. 25. System Metrics
  26. 26. Pulsar at Zhaopin ❏ One copy of data, single source-of-truth ❏ Don’t worry about data consistency between RabbitMQ and Kafka ❏ Multi-tenancy makes topic management easier ❏ Strong data durability allows us to stop worrying about message loss
  27. 27. Event Streaming Platform Beyond Pub/Sub Messaging
  28. 28. Event Streaming Platform
  29. 29. Event Streaming Platform ❏ Pulsar Functions: lightweight computing ❏ Flink: streaming-first, unified data processing ❏ Pulsar SQL (presto): interactive queries on both historic and real-time data
  30. 30. More details Coming soon for Next ApacheCon :-)
  31. 31. Contribute to Apache Pulsar The Apache Way
  32. 32. Zhaopin’s Contributions to Apache Pulsar ❏ Client Interceptors ❏ Dead Letter Topic ❏ Time Partitioned Message Tracker ❏ Service Url Provider ❏ Key_Shared Subscription ❏ Pulsar SQL Improvements ❏ Multi-versions Schema Support ❏ HDFS Offloader
  33. 33. Community ❏ Pulsar Website: https://pulsar.apache.org ❏ Twitter: @apache_pulsar / @streamnativeio ❏ Slack: https://apache-pulsar.herokuapp.com ❏ Mailing Lists dev@pulsar.apache.org, users@pulsar.apache.org ❏ Github https://github.com/apache/pulsar ❏ Medium https://medium.com/streamnative
  34. 34. Thanks!

×