Download to read offline
Uploaded byStreamNative
Migrating a Billion Transactions a Day To Apache Pulsar - Pulsar Summit NA 2021
The document presents an overview of Iterable's marketing automation tools, focusing on the architecture design involving systems like RabbitMQ and Redis for improved performance and user tracking. It outlines a phased replacement plan for these systems, emphasizing improvements in throughput and correctness, and introduces future enhancements using Flink and Pulsar. Key metrics include handling approximately 1 billion workflow node transitions daily and accommodating around 100 million new users entering workflows each day.
Migrating a Billion Transactions a Day To Apache Pulsar - Pulsar Summit NA 2021
- 1. Scalable Marketing Automation ThomasKim Systems Architect, Iterable Tom Wang Staff Software Engineer, Iterable
- 2. OUR AGENDA 2 1. Background:Iterable’s Marketing Automation Tools 2. Background: Current Implementation 3. Design on Pulsar 4. Flink and Beyond
- 3. Your Subtitle GoesHere (Delete if not needed) Iterable’s Marketing Automation: “Workflows” ● Drag and drop interface to mix and match filters and actions to make marketing campaigns ● Both blast and real-time 3
- 4. Workflow Campaigns: DelayNodes 4 Your Subtitle Goes Here (Delete if not needed)
- 5. Workflow Campaigns: VisitorTracking 5 Your Subtitle Goes Here (Delete if not needed) visitor id 1234 is here
- 6. Workflow Campaigns: UIObservability 6 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 7.
- 8. RabbitMQ 8 Your Subtitle GoesHere (Delete if not needed)
- 9. RabbitMQ Flow Control 9 YourSubtitle Goes Here (Delete if not needed)
- 10. Scale 10 ● ~ 1Bworkflow node transitions every day ● ~ 100M new users entering into a workflow each day (double-check this?) ● At any point in time, hundreds of millions of users sitting somewhere in a workflow (in-flight, or in delay nodes) Your Subtitle Goes Here (Delete if not needed)
- 11. RabbitMQ + Redis+ PG 11 Your Subtitle Goes Here (Delete if not needed)
- 12. Goal: Single Sourceof Truth 12 ● Improves performance and correctness Your Subtitle Goes Here (Delete if not needed)
- 13. ● Queue drivesprocessing ● Implicitly holds truth of visitor tracking state Phase 1: Drop-in Replacement 13 Your Subtitle Goes Here (Delete if not needed)
- 14. ● Already somebenefits: ● Built-in support for scheduled messages ● Higher throughput ● Costs, compliance Phase 1: Drop-in Replacement 14 Your Subtitle Goes Here (Delete if not needed)
- 15. Phase 2: Transactions 15 ●Ack and re-enqueue need to be atomic Your Subtitle Goes Here (Delete if not needed)
- 16. Phase 3: DerivedUI State 16 Your Subtitle Goes Here (Delete if not needed)
- 17. Future: Flink andBeyond 17 ● Use Flink, Pulsar Flink Connector, and Pulsar to process entrance limits Your Subtitle Goes Here (Delete if not needed)
- 18. Entrance Limits withFlink 18 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 19. Recap 19 ● Phased replacementof RabbitMQ and other data stores ● Improved correctness through transactions Your Subtitle Goes Here (Delete if not needed)
- 20. Thanks 20 ● Team members ●Iterable Pulsar team: ● Streamnative Your Subtitle Goes Here (Delete if not needed)
- 21.
- 22. YOUR TITLE GOESHERE 22 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 23. YOUR TITLE GOESHERE 23 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 24. YOUR TITLE GOESHERE 24 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 25. YOUR TITLE GOESHERE 25 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 26. YOUR TITLE GOESHERE 26 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 27. YOUR TITLE GOESHERE 27 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 28. YOUR TITLE GOESHERE 28 ● Only need to use one column? Then this is the slide for you. Your Subtitle Goes Here (Delete if not needed)
- 29. ● Are twocolumns more your style? Then use this. YOUR TITLE GOES HERE Your Subtitle Goes Here (Delete if not needed) 29 ● Are two columns more your style? Then use this.
- 30. Your Subtitle GoesHere (Delete if not needed) YOUR TITLE GOES HERE ● Do you need three columns instead of two? Then use this. ● Do you need three columns instead of two? Then use this. 30 ● Do you need three columns instead of two? Then use this.
- 31. YOUR DIVIDER SLIDE TITLEGOES HERE (MORE DIVIDER OPTIONS IN THE APPENDIX)
- 32. YOUR TITLE GOESHERE Your Subtitle Goes Here (Delete if not needed) ● Use this slide if you have a stat that you would like to use. 32 58% This is where any info pertaining to the stat would go. STAT CREDIT
- 33. Your Subtitle GoesHere (Delete if not needed) YOUR TITLE GOES HERE ● Have two stats you want to use? Then this is the slide for you. 33 58% This is where any info pertaining to the stat would go. STAT CREDIT 29% This is where any info pertaining to the stat would go. STAT CREDIT
- 34. Your Subtitle GoesHere (Delete if not needed) YOUR TITLE GOES HERE ● Have stats that need to be displayed a little differently? Use this slide if a bar graph is more your style. 34 1 2 3 4
- 35. Your Subtitle GoesHere (Delete if not needed) YOUR TITLE GOES HERE ● Have stats that need to be displayed a little differently? Use this slide if a table is more your style. 35 TITLE TITLE TITLE Stat Stat Stat Stat Stat Stat Stat Stat Stat
- 36. Your Subtitle GoesHere (Delete if not needed) YOUR TITLE GOES HERE ● Have an awesome image to go with your content? Not a problem. ● Only want to use an image? Use the slide after this! 36 REPLACE ME
- 37.
- 38. Your Subtitle GoesHere (Delete if not needed) YOUR TITLE GOES HERE ● One screenshot to show? I’m the slide for you. 38 REPLACE ME
- 39. Your Subtitle GoesHere (Delete if not needed) YOUR TITLE GOES HERE ● What about two? 39 REPLACE ME REPLACE ME
- 40. Your Subtitle GoesHere (Delete if not needed) YOUR TITLE GOES HERE ● Is a laptop more your style? Don’t worry. 40 REPLACE ME
- 41.
- 42. 42 Your Subtitle GoesHere (Delete if not needed) YOUR TITLE GOES HERE ● Have some extra white space that you’d like to fill up? Use the Iterable nodes! ● You can move, rotate, and resize these images.
- 43. 43 Your Subtitle GoesHere (Delete if not needed) YOUR TITLE GOES HERE ● Have some extra white space that you’d like to fill up? Use the Iterable nodes! ● You can move, rotate, and resize these images.
- 44. 44 Your Subtitle GoesHere (Delete if not needed) YOUR TITLE GOES HERE ● Have some extra white space that you’d like to fill up? Use the Iterable nodes! ● You can move, rotate, and resize these images.
- 45. 45 Your Subtitle GoesHere (Delete if not needed) YOUR TITLE GOES HERE ● Have some extra white space that you’d like to fill up? Use the Iterable nodes! ● You can move, rotate, and resize these images.
- 46. 46 Your Subtitle GoesHere (Delete if not needed) YOUR TITLE GOES HERE ● Have some extra white space that you’d like to fill up? Use the Iterable nodes! ● You can move, rotate, and resize these images.
- 47. 47 Your Subtitle GoesHere (Delete if not needed) YOUR TITLE GOES HERE ● Have some extra white space that you’d like to fill up? Use the Iterable nodes! ● You can move, rotate, and resize these images.
- 48. 48 This is whereyour quote would go. You can adjust the height as necessary to accommodate for anything that’s longer than this. First Name Last Name Job Title, Company
- 49. Q&A / ENDSLIDE
- 50.
- 51.
- 52. SEND TIME OPTIMIZATION MODERN ARCHITECTURE WEBINAR RE-ENGAGE MESSAGING ANNOUNCEMENTS CHANNEL OPTIMIZATION CAMPAIGNS RESPONSE FEED DATA TARGETED MARKETING TRANSACTIONAL FREQUENCY OPTIMIZATION PERSONALIZATION LOW-LEVEL PERSONALIZATION SEGMENTATION SMS CADENCE SINGLE DATA PLATFORM GROWTH AUDIENCE CONNECTIONERROR DESKTOP QUOTE ONLINE SHOPPING WORKFLOW STUDIO DATA SILOS AUDIENCE CONNECTION CONTENT CHECKED RE-ENGAGEMENT TRUE CROSS-CHANNEL EMAIL ERROR SEARCH MEDIUMS HIGHLIGHT AUTOMATION EASY TO USE EMAIL BRAND LOVE EVENT DATA WELCOME USER PROFILE DATA 1-TO-1 RELEVANCE SMART PHONE CAMPAIGN SUCCESS DIRECTION PROMOTIONS VALUE RED ICONS 52
- 53. SEND TIME OPTIMIZATION MODERN ARCHITECTURE WEBINAR RE-ENGAGE MESSAGING ANNOUNCEMENTS CHANNEL OPTIMIZATION CAMPAIGNS RESPONSE FEED DATA TARGETED MARKETING TRANSACTIONAL FREQUENCY OPTIMIZATION PERSONALIZATION LOW-LEVEL PERSONALIZATION SEGMENTATION SMS CADENCE SINGLE DATA PLATFORM GROWTH AUDIENCE CONNECTIONERROR DESKTOP QUOTE ONLINE SHOPPING WORKFLOW STUDIO DATA SILOS AUDIENCE CONNECTION CONTENT CHECKED RE-ENGAGEMENT TRUE CROSS-CHANNEL EMAIL ERROR SEARCH MEDIUMS HIGHLIGHT AUTOMATION EASY TO USE EMAIL BRAND LOVE EVENT DATA WELCOME USER PROFILE DATA 1-TO-1 RELEVANCE SMART PHONE CAMPAIGN SUCCESS DIRECTION PROMOTIONS VALUE PURPLE ICONS 53
- 54. SEND TIME OPTIMIZATION MODERN ARCHITECTURE WEBINAR RE-ENGAGE MESSAGING ANNOUNCEMENTS CHANNEL OPTIMIZATION CAMPAIGNS RESPONSE FEED DATA TARGETED MARKETING TRANSACTIONAL FREQUENCY OPTIMIZATION PERSONALIZATION LOW-LEVEL PERSONALIZATION SEGMENTATION SMS CADENCE SINGLE DATA PLATFORM GROWTH AUDIENCE CONNECTIONERROR DESKTOP QUOTE ONLINE SHOPPING WORKFLOW STUDIO DATA SILOS AUDIENCE CONNECTION CONTENT CHECKED RE-ENGAGEMENT TRUE CROSS-CHANNEL EMAIL ERROR SEARCH MEDIUMS HIGHLIGHT AUTOMATION EASY TO USE EMAIL BRAND LOVE EVENT DATA WELCOME USER PROFILE DATA 1-TO-1 RELEVANCE SMART PHONE CAMPAIGN SUCCESS DIRECTION PROMOTIONS VALUE COOL GREEN ICONS 54
- 55. SEND TIME OPTIMIZATION MODERN ARCHITECTURE WEBINAR RE-ENGAGE MESSAGING ANNOUNCEMENTS CHANNEL OPTIMIZATION CAMPAIGNS RESPONSE FEED DATA TARGETED MARKETING TRANSACTIONAL FREQUENCY OPTIMIZATION PERSONALIZATION LOW-LEVEL PERSONALIZATION SEGMENTATION SMS CADENCE SINGLE DATA PLATFORM GROWTH AUDIENCE CONNECTIONERROR DESKTOP QUOTE ONLINE SHOPPING WORKFLOW STUDIO DATA SILOS AUDIENCE CONNECTION CONTENT CHECKED RE-ENGAGEMENT TRUE CROSS-CHANNEL EMAIL ERROR SEARCH MEDIUMS HIGHLIGHT AUTOMATION EASY TO USE EMAIL BRAND LOVE EVENT DATA WELCOME USER PROFILE DATA 1-TO-1 RELEVANCE SMART PHONE CAMPAIGN SUCCESS DIRECTION PROMOTIONS VALUE CYAN ICONS 55
- 56. SEND TIME OPTIMIZATION MODERN ARCHITECTURE WEBINAR RE-ENGAGE MESSAGING ANNOUNCEMENTS CHANNEL OPTIMIZATION CAMPAIGNS RESPONSE FEED DATA TARGETED MARKETING TRANSACTIONAL FREQUENCY OPTIMIZATION PERSONALIZATION LOW-LEVEL PERSONALIZATION SEGMENTATION SMS CADENCE SINGLE DATA PLATFORM GROWTH AUDIENCE CONNECTIONERROR DESKTOP QUOTE ONLINE SHOPPING WORKFLOW STUDIO DATA SILOS AUDIENCE CONNECTION CONTENT CHECKED RE-ENGAGEMENT TRUE CROSS-CHANNEL EMAIL ERROR SEARCH MEDIUMS HIGHLIGHT AUTOMATION EASY TO USE EMAIL BRAND LOVE EVENT DATA WELCOME USER PROFILE DATA 1-TO-1 RELEVANCE SMART PHONE CAMPAIGN SUCCESS DIRECTION PROMOTIONS VALUE GREY ICONS 56
- 57. WHITE ICONS 57 SEND TIME OPTIMIZATION MODERN ARCHITECTURE WEBINAR RE-ENGAGE MESSAGING ANNOUNCEMENTS CHANNEL OPTIMIZATION CAMPAIGNS RESPONSE FEED DATA TARGETED MARKETING TRANSACTIONAL FREQUENCY OPTIMIZATION PERSONALIZATION LOW-LEVEL PERSONALIZATION SEGMENTATION SMS CADENCE SINGLEDATA PLATFORM GROWTH AUDIENCE CONNECTION ERROR DESKTOP QUOTE ONLINE SHOPPING WORKFLOW STUDIO DATA SILOS AUDIENCE CONNECTION CONTENT CHECKED RE-ENGAGEMENT TRUE CROSS-CHANNEL EMAIL ERROR SEARCH MEDIUMS HIGHLIGHT AUTOMATION EASY TO USE EMAIL BRAND LOVE EVENT DATA WELCOME USER PROFILE DATA 1-TO-1 RELEVANCE SMART PHONE CAMPAIGN SUCCESS DIRECTION PROMOTIONS VALUE
- 58.
- 59.
- 60.
- 61.
- 62.
- 63.
- 64.
- 65.
- 66.
- 67.
- 68.
- 69.
- 70.
- 71.
Editor's Notes
- #8 Workflow has entrance limit checks Simultaneous Limit: How many times a contact can be in a workflow Lifetime Limit: How many times a contact can enter a workflow for the lifetime limit, this means we have to store the history of all users who have entered a workflow indefinitely
- #14 discuss the core loop here