Migrating a Billion Transactions a Day To Apache Pulsar - Pulsar Summit NA 2021
•Download as PPTX, PDF•
0 likes•451 views
Iterable is a marketing automation SaaS that operates at vast scale. Iterable processes a billion queue transactions per day representing the flow of users through programmatic marketing campaigns, which we call Workflow Campaigns. We're currently migrating this workload from RabbitMQ to Apache Pulsar. In this talk we'll go over how Workflow Campaigns work, and the features of Apache Pulsar that make it well-suited to this workload. Pulsar transactions ensure exactly once processing of these campaigns. Tiered storage allows us to keep a permanent record of every users' campaign history. The log of state transitions that we store in Pulsar topics allows us to create derived metrics for near-real-time analytics. Stream processing with Flink allows us to enforce simple count-based application limits on Workflows. Finally, we'll discuss the process of migration with no downtime.
Recommended
More Related Content
More from StreamNative
More from StreamNative (20)
Recently uploaded
Recently uploaded (20)
Migrating a Billion Transactions a Day To Apache Pulsar - Pulsar Summit NA 2021
- 1. Scalable Marketing Automation Thomas Kim Systems Architect, Iterable Tom Wang Staff Software Engineer, Iterable
- 2. OUR AGENDA 2 1. Background: Iterable’s Marketing Automation Tools 2. Background: Current Implementation 3. Design on Pulsar 4. Flink and Beyond
- 3. Your Subtitle Goes Here (Delete if not needed) Iterable’s Marketing Automation: “Workflows” ● Drag and drop interface to mix and match filters and actions to make marketing campaigns ● Both blast and real-time 3
- 4. Workflow Campaigns: Delay Nodes 4 Your Subtitle Goes Here (Delete if not needed)
- 5. Workflow Campaigns: Visitor Tracking 5 Your Subtitle Goes Here (Delete if not needed) visitor id 1234 is here
- 6. Workflow Campaigns: UI Observability 6 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 7. 7 Workflow Campaigns: Entrance Limits
- 8. RabbitMQ 8 Your Subtitle Goes Here (Delete if not needed)
- 9. RabbitMQ Flow Control 9 Your Subtitle Goes Here (Delete if not needed)
- 10. Scale 10 ● ~ 1B workflow node transitions every day ● ~ 100M new users entering into a workflow each day (double-check this?) ● At any point in time, hundreds of millions of users sitting somewhere in a workflow (in-flight, or in delay nodes) Your Subtitle Goes Here (Delete if not needed)
- 11. RabbitMQ + Redis + PG 11 Your Subtitle Goes Here (Delete if not needed)
- 12. Goal: Single Source of Truth 12 ● Improves performance and correctness Your Subtitle Goes Here (Delete if not needed)
- 13. ● Queue drives processing ● Implicitly holds truth of visitor tracking state Phase 1: Drop-in Replacement 13 Your Subtitle Goes Here (Delete if not needed)
- 14. ● Already some benefits: ● Built-in support for scheduled messages ● Higher throughput ● Costs, compliance Phase 1: Drop-in Replacement 14 Your Subtitle Goes Here (Delete if not needed)
- 15. Phase 2: Transactions 15 ● Ack and re-enqueue need to be atomic Your Subtitle Goes Here (Delete if not needed)
- 16. Phase 3: Derived UI State 16 Your Subtitle Goes Here (Delete if not needed)
- 17. Future: Flink and Beyond 17 ● Use Flink, Pulsar Flink Connector, and Pulsar to process entrance limits Your Subtitle Goes Here (Delete if not needed)
- 18. Entrance Limits with Flink 18 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 19. Recap 19 ● Phased replacement of RabbitMQ and other data stores ● Improved correctness through transactions Your Subtitle Goes Here (Delete if not needed)
- 20. Thanks 20 ● Team members ● Iterable Pulsar team: ● Streamnative Your Subtitle Goes Here (Delete if not needed)
- 21. 21 Questions?
- 22. YOUR TITLE GOES HERE 22 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 23. YOUR TITLE GOES HERE 23 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 24. YOUR TITLE GOES HERE 24 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 25. YOUR TITLE GOES HERE 25 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 26. YOUR TITLE GOES HERE 26 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 27. YOUR TITLE GOES HERE 27 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 28. YOUR TITLE GOES HERE 28 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 29. ● Are two columns more your style? Then use this. YOUR TITLE GOES HERE Your Subtitle Goes Here (Delete if not needed) 29 ● Are two columns more your style? Then use this.
- 30. Your Subtitle Goes Here (Delete if not needed) YOUR TITLE GOES HERE ● Do you need three columns instead of two? Then use this. ● Do you need three columns instead of two? Then use this. 30 ● Do you need three columns instead of two? Then use this.
- 31. YOUR DIVIDER SLIDE TITLE GOES HERE (MORE DIVIDER OPTIONS IN THE APPENDIX)
- 32. YOUR TITLE GOES HERE Your Subtitle Goes Here (Delete if not needed) ● Use this slide if you have a stat that you would like to use. 32 58% This is where any info pertaining to the stat would go. STAT CREDIT
- 33. Your Subtitle Goes Here (Delete if not needed) YOUR TITLE GOES HERE ● Have two stats you want to use? Then this is the slide for you. 33 58% This is where any info pertaining to the stat would go. STAT CREDIT 29% This is where any info pertaining to the stat would go. STAT CREDIT
- 34. Your Subtitle Goes Here (Delete if not needed) YOUR TITLE GOES HERE ● Have stats that need to be displayed a little differently? Use this slide if a bar graph is more your style. 34 1 2 3 4
- 35. Your Subtitle Goes Here (Delete if not needed) YOUR TITLE GOES HERE ● Have stats that need to be displayed a little differently? Use this slide if a table is more your style. 35 TITLE TITLE TITLE Stat Stat Stat Stat Stat Stat Stat Stat Stat
- 36. Your Subtitle Goes Here (Delete if not needed) YOUR TITLE GOES HERE ● Have an awesome image to go with your content? Not a problem. ● Only want to use an image? Use the slide after this! 36 REPLACE ME
- 37. REPLACE ME 37
- 38. Your Subtitle Goes Here (Delete if not needed) YOUR TITLE GOES HERE ● One screenshot to show? I’m the slide for you. 38 REPLACE ME
- 39. Your Subtitle Goes Here (Delete if not needed) YOUR TITLE GOES HERE ● What about two? 39 REPLACE ME REPLACE ME
- 40. Your Subtitle Goes Here (Delete if not needed) YOUR TITLE GOES HERE ● Is a laptop more your style? Don’t worry. 40 REPLACE ME
- 41. 41 REPLACE ME
- 42. 42 Your Subtitle Goes Here (Delete if not needed) YOUR TITLE GOES HERE ● Have some extra white space that you’d like to fill up? Use the Iterable nodes! ● You can move, rotate, and resize these images.
- 43. 43 Your Subtitle Goes Here (Delete if not needed) YOUR TITLE GOES HERE ● Have some extra white space that you’d like to fill up? Use the Iterable nodes! ● You can move, rotate, and resize these images.
- 44. 44 Your Subtitle Goes Here (Delete if not needed) YOUR TITLE GOES HERE ● Have some extra white space that you’d like to fill up? Use the Iterable nodes! ● You can move, rotate, and resize these images.
- 45. 45 Your Subtitle Goes Here (Delete if not needed) YOUR TITLE GOES HERE ● Have some extra white space that you’d like to fill up? Use the Iterable nodes! ● You can move, rotate, and resize these images.
- 46. 46 Your Subtitle Goes Here (Delete if not needed) YOUR TITLE GOES HERE ● Have some extra white space that you’d like to fill up? Use the Iterable nodes! ● You can move, rotate, and resize these images.
- 47. 47 Your Subtitle Goes Here (Delete if not needed) YOUR TITLE GOES HERE ● Have some extra white space that you’d like to fill up? Use the Iterable nodes! ● You can move, rotate, and resize these images.
- 48. 48 This is where your quote would go. You can adjust the height as necessary to accommodate for anything that’s longer than this. First Name Last Name Job Title, Company
- 49. Q&A / END SLIDE
- 50. APPENDIX
- 52. SEND TIME OPTIMIZATION MODERN ARCHITECTURE WEBINAR RE-ENGAGE MESSAGING ANNOUNCEMENTS CHANNEL OPTIMIZATION CAMPAIGNS RESPONSE FEED DATA TARGETED MARKETING TRANSACTIONAL FREQUENCY OPTIMIZATION PERSONALIZATION LOW-LEVEL PERSONALIZATION SEGMENTATION SMS CADENCE SINGLE DATA PLATFORM GROWTH AUDIENCE CONNECTION ERROR DESKTOP QUOTE ONLINE SHOPPING WORKFLOW STUDIO DATA SILOS AUDIENCE CONNECTION CONTENT CHECKED RE-ENGAGEMENT TRUE CROSS-CHANNEL EMAIL ERROR SEARCH MEDIUMS HIGHLIGHT AUTOMATION EASY TO USE EMAIL BRAND LOVE EVENT DATA WELCOME USER PROFILE DATA 1-TO-1 RELEVANCE SMART PHONE CAMPAIGN SUCCESS DIRECTION PROMOTIONS VALUE RED ICONS 52
- 53. SEND TIME OPTIMIZATION MODERN ARCHITECTURE WEBINAR RE-ENGAGE MESSAGING ANNOUNCEMENTS CHANNEL OPTIMIZATION CAMPAIGNS RESPONSE FEED DATA TARGETED MARKETING TRANSACTIONAL FREQUENCY OPTIMIZATION PERSONALIZATION LOW-LEVEL PERSONALIZATION SEGMENTATION SMS CADENCE SINGLE DATA PLATFORM GROWTH AUDIENCE CONNECTION ERROR DESKTOP QUOTE ONLINE SHOPPING WORKFLOW STUDIO DATA SILOS AUDIENCE CONNECTION CONTENT CHECKED RE-ENGAGEMENT TRUE CROSS-CHANNEL EMAIL ERROR SEARCH MEDIUMS HIGHLIGHT AUTOMATION EASY TO USE EMAIL BRAND LOVE EVENT DATA WELCOME USER PROFILE DATA 1-TO-1 RELEVANCE SMART PHONE CAMPAIGN SUCCESS DIRECTION PROMOTIONS VALUE PURPLE ICONS 53
- 54. SEND TIME OPTIMIZATION MODERN ARCHITECTURE WEBINAR RE-ENGAGE MESSAGING ANNOUNCEMENTS CHANNEL OPTIMIZATION CAMPAIGNS RESPONSE FEED DATA TARGETED MARKETING TRANSACTIONAL FREQUENCY OPTIMIZATION PERSONALIZATION LOW-LEVEL PERSONALIZATION SEGMENTATION SMS CADENCE SINGLE DATA PLATFORM GROWTH AUDIENCE CONNECTION ERROR DESKTOP QUOTE ONLINE SHOPPING WORKFLOW STUDIO DATA SILOS AUDIENCE CONNECTION CONTENT CHECKED RE-ENGAGEMENT TRUE CROSS-CHANNEL EMAIL ERROR SEARCH MEDIUMS HIGHLIGHT AUTOMATION EASY TO USE EMAIL BRAND LOVE EVENT DATA WELCOME USER PROFILE DATA 1-TO-1 RELEVANCE SMART PHONE CAMPAIGN SUCCESS DIRECTION PROMOTIONS VALUE COOL GREEN ICONS 54
- 55. SEND TIME OPTIMIZATION MODERN ARCHITECTURE WEBINAR RE-ENGAGE MESSAGING ANNOUNCEMENTS CHANNEL OPTIMIZATION CAMPAIGNS RESPONSE FEED DATA TARGETED MARKETING TRANSACTIONAL FREQUENCY OPTIMIZATION PERSONALIZATION LOW-LEVEL PERSONALIZATION SEGMENTATION SMS CADENCE SINGLE DATA PLATFORM GROWTH AUDIENCE CONNECTION ERROR DESKTOP QUOTE ONLINE SHOPPING WORKFLOW STUDIO DATA SILOS AUDIENCE CONNECTION CONTENT CHECKED RE-ENGAGEMENT TRUE CROSS-CHANNEL EMAIL ERROR SEARCH MEDIUMS HIGHLIGHT AUTOMATION EASY TO USE EMAIL BRAND LOVE EVENT DATA WELCOME USER PROFILE DATA 1-TO-1 RELEVANCE SMART PHONE CAMPAIGN SUCCESS DIRECTION PROMOTIONS VALUE CYAN ICONS 55
- 56. SEND TIME OPTIMIZATION MODERN ARCHITECTURE WEBINAR RE-ENGAGE MESSAGING ANNOUNCEMENTS CHANNEL OPTIMIZATION CAMPAIGNS RESPONSE FEED DATA TARGETED MARKETING TRANSACTIONAL FREQUENCY OPTIMIZATION PERSONALIZATION LOW-LEVEL PERSONALIZATION SEGMENTATION SMS CADENCE SINGLE DATA PLATFORM GROWTH AUDIENCE CONNECTION ERROR DESKTOP QUOTE ONLINE SHOPPING WORKFLOW STUDIO DATA SILOS AUDIENCE CONNECTION CONTENT CHECKED RE-ENGAGEMENT TRUE CROSS-CHANNEL EMAIL ERROR SEARCH MEDIUMS HIGHLIGHT AUTOMATION EASY TO USE EMAIL BRAND LOVE EVENT DATA WELCOME USER PROFILE DATA 1-TO-1 RELEVANCE SMART PHONE CAMPAIGN SUCCESS DIRECTION PROMOTIONS VALUE GREY ICONS 56
- 57. WHITE ICONS 57 SEND TIME OPTIMIZATION MODERN ARCHITECTURE WEBINAR RE-ENGAGE MESSAGING ANNOUNCEMENTS CHANNEL OPTIMIZATION CAMPAIGNS RESPONSE FEED DATA TARGETED MARKETING TRANSACTIONAL FREQUENCY OPTIMIZATION PERSONALIZATION LOW-LEVEL PERSONALIZATION SEGMENTATION SMS CADENCE SINGLE DATA PLATFORM GROWTH AUDIENCE CONNECTION ERROR DESKTOP QUOTE ONLINE SHOPPING WORKFLOW STUDIO DATA SILOS AUDIENCE CONNECTION CONTENT CHECKED RE-ENGAGEMENT TRUE CROSS-CHANNEL EMAIL ERROR SEARCH MEDIUMS HIGHLIGHT AUTOMATION EASY TO USE EMAIL BRAND LOVE EVENT DATA WELCOME USER PROFILE DATA 1-TO-1 RELEVANCE SMART PHONE CAMPAIGN SUCCESS DIRECTION PROMOTIONS VALUE
- 58. DIVIDER #1
- 59. DIVIDER #2
- 60. DIVIDER #3
- 61. DIVIDER #4
- 62. DIVIDER #5
- 63. DIVIDER #6
- 64. DIVIDER #7
- 65. DIVIDER #8
- 66. DIVIDER #9
- 67. DIVIDER #10
- 68. DIVIDER #11
- 69. DIVIDER #12
- 70. DIVIDER #13
- 71. THANK YOU!
Editor's Notes
- Workflow has entrance limit checks Simultaneous Limit: How many times a contact can be in a workflow Lifetime Limit: How many times a contact can enter a workflow for the lifetime limit, this means we have to store the history of all users who have entered a workflow indefinitely
- discuss the core loop here