Downloaded 19 times
Uploaded byDevin Bost
Real-World Pulsar Architectural Patterns
This document discusses real-world architectural patterns for implementing Apache Pulsar, focusing on distributed caching, tracing, and strategies for data ingestion and processing. It highlights various patterns and approaches for enhancing system reliability, data flow management, and message guarantee, while addressing challenges such as message loss, complex query logic, and tracing issues. The author, Devin Bost, shares insights from practical applications developed at Overstock, providing valuable tips for leveraging Pulsar in real-time messaging contexts.
More Related Content
Similar to Real-World Pulsar Architectural Patterns
![20260201 [FOSDEM] gomodjail - library sandboxing for Go modules.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/20260201fosdemgomodjail-librarysandboxingforgomodules-260201225659-76609ec4-thumbnail.jpg?width=640&height=640&fit=bounds)
Real-World Pulsar Architectural Patterns
- 1. Real-World Pulsar ArchitecturalPatterns Every pattern shown here has been developed and implemented with my team at Overstock Email: dbost@overstock.com Twitter: DevinBost LinkedIn: https://www.linkedin.com/in/devinbost/ By Devin Bost, Senior Data Engineer at Overstock Distributed Caching + Distributed Tracing +
- 2. Missing Stephen Bourkein Ireland and Sam Lowen, our new product manager.
- 3. Disclaimer: I’m assumingyou already know that Apache Pulsar is the future of real-time messaging
- 4. = Pulsar topic =Kafka topic = Apache Bookkeeper Tech Legend
- 5. Starting simple withPub/Sub
- 6. Starting simple withPub/Sub
- 7. Starting simple withPub/Sub
- 8. Starting simple withPub/Sub
- 9. Producer Consumer Alerts to endusers (e.g. Email, SMS, Twilio call, etc.) Passthrough function /ingest /feeds
- 10. Diagram courtesy of ThorSigurjonsson
- 11. Producer Producer Producer ConsumerConsumer Consumer Alerts to end users (e.g. Email, SMS, Twilio call, etc.) Passthrough function Higher-availability: /ingest /feeds
- 12. Producer Producer Producer ConsumerConsumer Consumer Passthrough function Is this a safe approach to caching? /ingest /feeds
- 13. Producer Producer Producer ConsumerConsumer Consumer Passthrough function What happens if Redis goes down? X /ingest /feeds
- 14. Producer Producer Producer ConsumerConsumer Consumer Passthrough function It’s much safer to use a distributed cache technology like Ignite /ingest /feeds
- 15. Producer Producer Producer ConsumerConsumer Consumer Passthrough function It’s much safer to use a distributed cache technology like Ignite Smart Persistence /ingest /feeds
- 16. Producer Producer Producer ConsumerConsumer Consumer Passthrough function It’s much safer to use a distributed cache technology like Ignite Smart Persistence Faster than Redis /ingest /feeds
- 17. Producer Producer Producer ConsumerConsumer Consumer Passthrough function It’s much safer to use a distributed cache technology like Ignite Smart Persistence Faster than Redis Supports tables with backing cache /ingest /feeds
- 18. Producer Producer Producer ConsumerConsumer Consumer Passthrough function It’s much safer to use a distributed cache technology like Ignite Smart Persistence Faster than Redis Supports tables with backing cache Supports transactions /ingest /feeds
- 19. What if youhave a business- critical service that can’t lose messages?
- 20. “But, doesn’t Pulsaralready guarantee that messages won’t be lost?”
- 21. There’s a differencebetween Pulsar losing messages and your application losing messages!
- 22. So, we introducea backfill path.
- 23. Notice I avoidedputting a SQL database on the list Replication Persistent storage Producer Producer Producer Consumer Consumer Consumer Alerts to end users (e.g. Email, SMS, Twilio call, etc.) Passthrough function Batch job Backfill Topic Consumer Consumer Consumer Alerts to end users (e.g. Email, SMS, Twilio call, etc.) OR Etc Message delivered yet? Message delivered yet? Kappa + Lambda architectures /ingest /feeds
- 24.
- 25. Replication Persistent storage Pulsar Function Alertsto end users (e.g. Email, SMS, Twilio call, etc.) Alerts to end users (e.g. Email, SMS, Twilio call, etc.) Batch job OR Etc Producer Producer Producer Passthrough function Backfill Topic Pulsar Function Message delivered yet? Message delivered yet? You could add another passthrough function and topic if you want more isolation. /ingest /feeds
- 26. How about justfor ingesting data into a cache with a backfill?
- 27. Option 1: Web Service Passthrough Function PersistentAppend Only Storage Cache Sink Batch Engine (e.g. Spark, NiFi, etc.) Read All Data Loads into existing topic OR Starts Job Etc Replication /ingest /feeds
- 28. Option 2: Achieves separationof concerns and prevents QoS problems with live traffic when running a backfill Web Service Passthrough Function Cache Sink Batch Engine (e.g. Spark, NiFi, etc.) Loads into backfill topic OR Starts Job Cache Sink Persistent Append Only Storage Read All Data Etc Replication /ingest /feeds
- 29. Topic with Retention Function CacheSink Function (stopped until needing to backfill) Exclusive Mode (Subscription stores in Bookkeeper automatically.) Tiered Storage in S3 or Google Cloud Backfill Cache Sink Starts Function Passthrough FunctionOption 3: Note: You need to ensure the Bookkeeper cluster is fast enough to keep up with the brokers or your brokers’ memory will fill up Also, this approach will only give you a single backfill run unless you have additional replication. /feeds
- 30. What about addingcaching to a legacy application?
- 31. Legacy SQL DB Website Enrich fromcached data Extract Relevant Data Filter to desired clicks (Raw Clicks) Store in cache Web Application Emitting specific events (Omitting passthrough details for simplicity.)
- 32. Legacy SQL DB Website Enrich fromcached data Extract Relevant Data Filter to desired clicks Store in cache Web Application Emitting specific events You can also emit directly to Pulsar as a producer. It’s simpler if you have the ability to touch the website code.
- 33. Legacy SQL DB Website Enrich fromcached data Extract Relevant Data Filter to desired clicks Store in cache Web Application Emitting specific events You can also emit directly to Pulsar as a producer. It’s simpler if you have the ability to touch the website code. However, the raw clicks flow still gets messy.
- 34. Legacy SQL DB Website Enrich fromcached data Store in cache Web Application Emitting specific events Event A Event B Event C Cleaner & better separation of concerns to have purposeful topics... Easier to debug & maintain.
- 35. What if you’reusing a graph engine for more complex query logic but need that data in real-time?
- 36. If you don’tmake it synchronous, you will get race conditions when updating and querying the graph! Web Application Emitting specific deltas (e.g. state change, increment, etc.) (2) Return on completion (1) Write Change, Wait for Success Get full record Complex graph query Synchronous Update Function
- 37. What if youneed a more robust verification that deltas are in order and aren’t duplicates? (e.g. financially impacting increment/decrement values or state variables)
- 38. Usually, it’s bestto separate concerns into separate functions, like this. Web Application Emitting specific deltas (e.g. state change, increment, etc.) (2) Return on completion (1) Write Change, Wait for Success Get full record Complex graph query Synchronous Update Function Gate Keeper Filter Check if message has been seen already Duplicate or late? Drop messageYes No But, is that the right approach here?
- 39. What happens whenyou turn up parallelism on the functions?
- 40. Web Application Emitting specificdeltas (e.g. state change, increment, etc.) (2) Return on completion (1) Write Change, Wait for Success Get full record Complex graph query Synchronous Update Function Synchronous Update Function Synchronous Update Function Synchronous Update Function Synchronous Update Function Synchronous Update Function Gate Keeper Filter Gate Keeper Filter Gate Keeper Filter Gate Keeper Filter Gate Keeper Filter Gate Keeper Filter
- 41. In this case,the right approach is to consolidate your logic to leverage the transactional guarantees of your database. Web Application (2) Return on completion (1) Check if duplicate or outdated. If not, write Change, Wait for Success Get full record Complex graph query Synchronous Update Function
- 42. Leverage transactional guaranteesof your database! (Your function will need to retry if a transaction fails.) Web Application Get full record Complex graph query Synchronous Update Function Synchronous Update Function Synchronous Update Function Synchronous Update Function
- 43. Leverage transactional guaranteesof your database! (Your function will need to retry if a transaction fails.) Web Application Get full record Complex graph query Synchronous Update Function Synchronous Update Function Synchronous Update Function Synchronous Update Function Always be mindful of how the behavior might change when function parallelism is turned up!
- 44. Leverage transactional guaranteesof your database! (Your function will need to retry if a transaction fails.) Web Application Get full record Complex graph query Synchronous Update Function Synchronous Update Function Synchronous Update Function Synchronous Update Function Always be mindful of how the behavior might change when function parallelism is turned up! If making state changes, be sure that you get timestamps on your upstream data contract so you can verify if the messages are in order!
- 45. Now, what happenswhen you need to debug a large or complex function flow?
- 46. FunctionFunction Function Web Application Function WebService Web Application Function Function Web Service Web Application FunctionFunction Function Function Function Function
- 47. FunctionFunction Function Web Application Function WebService Web Application Function Function Web Service Web Application FunctionFunction Function Function Function Function What happens if some messages seem to not be reaching their destination?
- 48. FunctionFunction Function Web Application Function WebService Web Application Function Function Web Service Web Application FunctionFunction Function Function Function Function What happens if some messages seem to not be reaching their destination? What happens if a message isn’t getting transformed correctly at some point or null values are appearing?
- 49. FunctionFunction Function Web Application Function WebService Web Application Function Function Web Service Web Application FunctionFunction Function Function Function Function What happens if some messages seem to not be reaching their destination? What happens if a message isn’t getting transformed correctly at some point or null values are appearing? What if we can’t modify the function code (since it’s a multi-tenant application)?
- 50. FunctionFunction Function Web Application Function WebService Web Application Function Function Web Service Web Application FunctionFunction Function Function Function Function What happens if some messages seem to not be reaching their destination? What happens if a message isn’t getting transformed correctly at some point or null values are appearing? What if we can’t modify the data contracts either for the same reason? What if we can’t modify the function code (since it’s a multi-tenant application)?
- 51. Function Function FunctionFunction Span 1 Span 2 Span 3 Span 4 Span 1 Trace
- 52. Function Function FunctionFunction Span 1 Span 2 Span 3 Span 4 Span 1 Trace
- 53. Function Function FunctionFunction Span 1 Span 2 Span 3 Span 4 Span 1 Trace
- 54. Jaeger is basedon the OpenTracing standard Check out the book, “Mastering Distributed Tracing” (2019) by Yuri Shkuro Trace Span Span Span Span Span Span Span Span Span Span SpanSpan Tags (Key/Value) Logs (Key/Value + Timestamp) Tags (Key/Value) Logs (Key/Value + Timestamp) Tags (Key/Value) Logs (Key/Value + Timestamp) . . .Span Span Tags (Key/Value) Logs (Key/Value + Timestamp)
- 57. 4 5 7 3 Count By Key e.g.Over a session-based window
- 58. + + +OR
- 59. Metadata parameters for the TapFunction’s envelopeare specified in its config Message1 Message2 Message3 Message1’ Message2’ Message3’ Message1’’ Message2’’ Message3’’ . . .Function1 Function1 JoinerFunction Message2 Message2’ Message2’’ CorrelationId=productId-784 Message1 Message1’ Message1’’ CorrelationId=productId-142 Message3 Message3’ Message3’’CorrelationId=productId-923 Jaeger Sink Taps wrap message with header containing: CorrelationId, Tenant, Namespace, Name, timestamp, etc. TapFunction TapFunction TapFunction SamplerFunction CorrelationId=productId-784 Message1 Message1’ Message1’’ StartTimestamp EndTimestamp Span CorrelationId=productId-784 StartTimestamp EndTimestamp Span Message1 StartTimestamp EndTimestamp Span Message1’ StartTimestamp EndTimestamp Span Message1’’Message1’ Message1’’ Message1’’’
- 60. CorrelationId is derived andput into the envelope produced by the TapFunction. { "correlationKey": "productId", "correlationValue": "20603199", "correlationId": "productId-20603199”, . . . } The TapFunction defines the correlationKey in its Pulsar Config. You can tap ANY topic! Message1 Message2 Message3 Message1’ Message2’ Message3’ Message1’’ Message2’’ Message3’’ . . .Function1 Function1 Taps wrap message with header containing: CorrelationId, Tenant, Namespace, Name, timestamp, etc. TapFunction TapFunction TapFunction
- 61. Uses Flink’s statefuljoin capability. It all happens in a keyed stream! JoinerFunction Message2 Message2’ Message2’’ CorrelationId=productId-784 Message1 Message1’ Message1’’ CorrelationId=productId-142 Message3 Message3’ Message3’’CorrelationId=productId-923
- 62. Allows us tospecify rate to limit how many messages are sampled. For dev, this is set to 100% to allow all. This is just a simple Pulsar filter function. SamplerFunction CorrelationId=productId-784 Message1 Message1’ Message1’’
- 63. These Spans emitto Jaeger and can be stored in a Cassandra or Elasticsearch backend for production. OR Jaeger Sink StartTimestamp EndTimestamp Span CorrelationId=productId-784 StartTimestamp EndTimestamp Span Message1 StartTimestamp EndTimestamp Span Message1’ StartTimestamp EndTimestamp Span Message1’’Message1’ Message1’’ Message1’’’
- 65. If fields areomitted from the tap’s config, we capture all that we can.
- 66. Another trick isto provide alternative representations of a value to make search/analytics easier downstream.
- 67.
- 68. Real-World Pulsar ArchitecturalPatterns Every pattern shown here has been developed and implemented with my team at Overstock Email: dbost@overstock.com Twitter: DevinBost LinkedIn: https://www.linkedin.com/in/devinbost/ By Devin Bost, Senior Data Engineer at Overstock Distributed Caching + Distributed Tracing