Elastic Data Processing with Apache Flink and Apache Pulsar More and more applications are using Flink for low-latency data processing. Flink unifies batch and stream processing using one computation engine. However in reality, in order to really unify batch and stream processing, it requires a data system offers one unified data representation for both batch and streaming data. Nowadays, streaming data is typically stored in a log storage or messaging system, while batch data is stored in distributed filesystem and object stores. That means that data scientists still need write two different computing jobs to access same data stored in different data systems. Apache Pulsar is the next generation messaging and streaming data system. It was originally built at Yahoo, and has graduated from Apache Incubator and become a Top-Level-Project. Pulsar separates messaging serving and data storage into two layers. Such layered architecture provides high throughput and low-latency while ensuring high availability and scalability. Pulsar’s segment centric storage design along with layered architecture makes Pulsar a perfect unbounded streaming data system, which can well fit into Flink’s computation model. In this talk, Sijie Guo from Apache Pulsar PMC, will introduce Pulsar and its layered architecture and segment-centric storage, detailing how this architecture can well integrate with Flink to provide elastic unified batch and stream processing.

1. Elastic Data Processing with
Apache Flink and Apache Pulsar
Sijie Guo (sijieg)
2019-04-02

2. Who am I
• Apache Pulsar PMC Member
• Apache BookKeeper PMC Member
• Interested in technologies around Event Streaming

3. Agenda
• What is Apache Pulsar?
• A Pulsar View on Data - Segmented Stream
• Pulsar - Access Pattern & Tiered Storage
• Pulsar - Schema
• When Flink meets Pulsar

4. What is Apache Pulsar?

5. 2010
2011
2012
2006
2003
Pub/Sub Messaging

6. “Flexible Pub/Sub messaging
backed by durable log/stream storage”

7. Pulsar - Pub/Sub

8. Pulsar - Multi Tenancy

9. Pulsar - Queue + Streaming

10. Pulsar - Cloud Native
• Independent Scalability
• Instant Failure Recovery
• Balance-free on cluster
expansions
Layered Architecture

11. A Pulsar View on Data

12. Batch - HDFS

13. Stream - Pub/Sub

14. A Flink View on Computing
“Batch processing is a special case of
Stream processing”

15. Pulsar = Segmented Stream

16. Topic
Topic
Producers
Consumers
Time

17. Partitions
P0
P1
P2
P3
Producers
Consumers
Time

18. Segments
Segment 1 Segment 2 Segment 3 Segment 4
Segment 1 Segment 2 Segment 3
Segment 1 Segment 2 Segment 3
Segment 1 Segment 2 Segment 3
P0
P1
P2
P3
Producers
Consumers
Time

19. Stream
Segment 1 Segment 2 Segment 3 Segment 4
Segment 1 Segment 2 Segment 3
Segment 1 Segment 2 Segment 3
Segment 1 Segment 2 Segment 3
P0
P1
P2
P3
Producers
Consumers
Time

20. Stream
Segment 1 Segment 2 Segment 3 Segment 4Stream
Producers
Consumers
Time

21. Segmented Stream
• Segmented Stream Systems
• Apache Pulsar, Twitter EventBus, EMC Pravega
• All Apache BookKeeper based
• Used BK in a different way
• Pulsar, EventBus - Uses BK as the segment store
• Pravega - Uses BK as the journal only

22. Access Patterns
✓ Write
写
✓ Tailing Read
追尾
✓ Catchup Read
追赶读
Segment 1 Segment 2 Segment 3 Segment 4Stream
Time

23. Access Patterns
✓ Write
写
✓ Tailing Read
追尾
✓ Catchup Read
追赶读
Segment 1 Segment 2 Segment 3 Segment 4Stream
Time
Write

24. Access Patterns
✓ Write
写
✓ Tailing Read
追尾
✓ Catchup Read
追赶读
Segment 1 Segment 2 Segment 3 Segment 4Stream
Time
Write
Tailing Read

25. Access Patterns
✓ Write
写
✓ Tailing Read
追尾
✓ Catchup Read
追赶读
Segment 1 Segment 2 Segment 3 Segment 4Stream
Time
Write
Tailing Read
Catchup Read

26. Write
PulsarKafka
Broker
Partition
(Leader)
Broker
Partition
(Follower)
Broker
Partition
(Follower)
Broker Broker Broker

27. Tailing Read
PulsarKafka
Broker
Partition
(Leader)
Broker
Partition
(Follower)
Broker
Partition
(Follower)
Broker Broker Broker

28. Catchup Read
PulsarKafka
Broker
Partition
(Leader)
Broker
Partition
(Follower)
Broker
Partition
(Follower)
Broker Broker Broker

29. IO Isolation
PulsarKafka
Broker
Partition
(Leader)
Broker
Partition
(Follower)
Broker
Partition
(Follower)
Broker Broker Broker
PulsarKafka
Broker
Partition
(Leader)
Broker
Partition
(Follower)
Broker
Partition
(Follower)
Broker Broker Broker
PulsarKafka
Broker
Partition
(Leader)
Broker
Partition
(Follower)
Broker
Partition
(Follower)
Broker Broker Broker

30. Infinite Stream
✓ Write
写
✓ Tailing Read
追尾
✓ Catchup Read
追赶读
Segment 1 Segment 2 Segment 3 Segment 4Stream
Time
Bookies Brokers

31. Infinite Stream
✓ Write
写
✓ Tailing Read
追尾
✓ Catchup Read
追赶读
Segment 1 Segment 2 Segment 3 Segment 4Stream
Time
Tiered Storage Bookies Brokers

32. Tiered Storage
• Offloader
• When: size-based, time-based, or triggered by pulsar-admin
• How: copy a segment to tiered storage, and delete it from bookkeeper
• Access: broker knows how to read the data back, or bypass read
the offloaded segments directly
• Available Offloaders
• Cloud Offloder : AWS, GCS, Azure, …
• HDFS, Ceph, …

33. Stream as a Unified View on Data
Segment 1 Segment 2 Segment 3 Segment 4Stream
Producers
Consumers
Time
Segment 6Segment 5
Segment
Readers

34. Data Processing on Pulsar
Segment 1 Segment 2 Segment 3 Segment 4Stream Segment 6Segment 5
Time
Bounded Stream Bounded Stream
Unbounded Stream
Unbounded Stream

35. When Flink meets Pulsar

36. Goals
• Flink + Pulsar
• Streaming Connectors
• Source Connectors
• PulsarCatalog: Schema Integration
• PulsarStateBackend
• Pulsar for the unified view of Data, Flink for the unified view of Computing

37. Done

38. Streaming Source -> Streaming Sink

39. Streaming Source -> Streaming Table Sink

40. Batch Sink

41. Case Study - Zhaopin.com

42. Zhaopin.com
Zhaopin.com is the biggest online recruitment service provider
in China
Zhaopin.com provides job seekers a comprehensive resume service, latest
employment, and career development related information, as well as in-depth online
job search for positions throughout China
Zhaopin.com provides professional HR services to over 2.2 million clients and its
average daily page views are over 68 million.

43. Job Search

44. Data Processing

45. Metrics
50+ Namespaces
3000+ Topics
6+ billion Messages per day
3TB Storage per day
20+ Core Services

46. Roadmap

47. Batch Source
• Read Segments in Parallel
• Bypass Brokers
• Access tiered storage directly
• Scan Trimmer
• Select Segments by Publish Time

48. Schema Integration
• Pulsar has builtin schema registry
• Primitive types, Avro, Json, Protobuf, …
• Schema Evolution & Multi-versioning schemas
• PulsarCatalog

49. State Backend
• BookKeeperStateBackend
• Save State as Segments to BookKeeper

50. Unified Data Processing
Segment 1 Segment 2 Segment 3 Segment 4Stream Segment 6Segment 5
Time
Query
Past Now Future
Parallel Segment Reads Pub-Sub Streaming Reads
Segment 1
Segment 3
Segment 2
Segment 4 Segment 5 Segment 6

51. ✓ Twitter: @apache_pulsar
✓ Wechat Subscription: ApachePulsar
✓ Mailing Lists
dev@pulsar.apache.org, users@pulsar.apache.org
✓ Slack
https://apache-pulsar.slack.com
✓ Localization
https://crowdin.com/project/apache-pulsar
✓ Github
https://github.com/apache/pulsar
https://github.com/apache/bookkeeper
Community