Successfully reported this slideshow.
Your SlideShare is downloading. ×

Tackling Compliance When it Becomes Your Biggest Performance Bottleneck.pdf

Tackling Compliance When it Becomes Your Biggest Performance Bottleneck.pdf

Download to read offline

When developing software in heavily regulated markets, compliance is not an optional nicety. But compliance is every developer’s nightmare when trying to write performant software, especially when you need to communicate with 3rd party software or a Regulator’s services that can’t give you the performance guarantees you require.

This talk’s premise is to talk through such challenges we faced and how we used event-driven architecture and asynchronous events to handle different latencies to ensure that different market performance issues didn’t have system-wide effects.

When developing software in heavily regulated markets, compliance is not an optional nicety. But compliance is every developer’s nightmare when trying to write performant software, especially when you need to communicate with 3rd party software or a Regulator’s services that can’t give you the performance guarantees you require.

This talk’s premise is to talk through such challenges we faced and how we used event-driven architecture and asynchronous events to handle different latencies to ensure that different market performance issues didn’t have system-wide effects.

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Tackling Compliance When it Becomes Your Biggest Performance Bottleneck.pdf

  1. 1. Tackling Compliance When it Becomes Your Biggest Performance Bottleneck By Craig Saunders & Harsh Gautam
  2. 2. Classified as General Premise © Kindred Group plc 2022 2 “ When developing software in heavily regulated markets, compliance is not an optional nicety. But compliance is every developer’s nightmare when trying to write performant software, especially when you need to communicate with 3rd party software or a Regulator’s services that can’t give you the performance guarantees you require. This talk’s premise is to talk through such challenges we faced and how we used event-driven architecture and asynchronous events to handle different latencies to ensure that different market performance issues didn’t have system- wide effects.
  3. 3. Classified as General Our Opinions © Kindred Group plc 2022 3
  4. 4. Classified as General Who are we? © Kindred Group plc 2022 4
  5. 5. Classified as General Bio Cont… Craig Saunders © Kindred Group plc 2022 01 Software Engineer with 26 years experience 02 Work in the strategic side of software development 03 Work closely with Architects, Stakeholders and POs 04 Worked with Microservices for about 6 years 01 Huge tech advocate – love playing with the toys
  6. 6. Classified as General Bio Cont… Harsh Gautam © Kindred Group plc 2022 01 Software Engineer with 14 years experience 02 Passionate about solving complex technical challenges 03 Background with distributed systems and microservices 04 Love developing and designing new systems 05 Passion for learning new technologies and keeping myself current
  7. 7. Classified as General Bio Cont… © Kindred Group plc 2022 Ultimately, We’re geeks who likes playing with the toys!!
  8. 8. Classified as General Who is Kindred? © Kindred Group plc 2022 8
  9. 9. Classified as General Who is Kindred? © Kindred Group plc 2022 9 A tech company that operates in the online gambling space
  10. 10. Classified as General Tenet of Kindred’s Core Values © Kindred Group plc 2022 10 Player sustainability and compliance is non-negotiable
  11. 11. Classified as General © Kindred Group plc 2022 11 Kindred Core Values – Goes Beyond Kindred is about making gambling 100% enjoyable
  12. 12. Classified as General Set the Scene © Kindred Group plc 2022 12
  13. 13. Classified as General © Kindred Group plc 2022 13 Problem Domain? 01 Event-driven microservice landscape on Kubernetes At Kindred we have adopted the event-driven and microservice paradigms for a long time. This throws up may challenges with needing to be. fault-tolerant. 02 Zero downtime operation conditions We are a 24/7 product meaning customers are constantly engaging with our product; hence ensuring we have no occurrences of the site being down is critical. Deployments and migrations strategies need to bear this in mind. 03 50 million wallet transactions per day We have an average of 50 million transactions per day and can peak over 60 million; meaning we have to scale to a lot of customer activity – especially around large tournament such the World Cup. 04 Campaigns targeting upwards of 3 mil customers At certain times in the year Marketing can do large campaign send-outs targeting millions of customer; hence we need to scale to huge burst in traffic. 05 Real-time customer communication and engagement Modern marketing strategy is personalised content delivered at the right time to the customer, which can be onsite; hence execution speed is paramount. Couple this with large batch send- outs, we need to ensure that batch send-outs don’t impact the marketing strategy.
  14. 14. Classified as General Let’s Get Down to the Tech © Kindred Group plc 2022 14
  15. 15. Classified as General Platform Interactions © Kindred Group plc 2022 15 Campaign Back- office Platform Audience Integration Platform Communications Platform Player Account Platform Customer Verification Regulatory Integration Player Sustainability Execution Platform
  16. 16. Classified as General Execution Platform Raison D’Etre © Kindred Group plc 2022 16 “ The execution platform is a decision agnostic platform that enable safe, reliable and predictable execution of a marketing strategy to offer rewards and/or send communications. It enables 3rd party decision tools to be able to seamlessly integrate with Kindred platform
  17. 17. Classified as General Execution Platform © Kindred Group plc 2022 17 Send-out Execution Marketing Compliance Orchestration
  18. 18. Classified as General © Kindred Group plc 2022 18 Execution Platform Campaign Send-out Snapshot ADAPTER Campaign Association ADAPTER Test Accounts Adapter ADAPTER Campaign Send-out Executor ADAPTER Campaign Customer Details ADAPTER Campaign Orchestrator ADAPTER Campaign Scheduler ADAPTER Orchestration Marketing Compliance ADAPTER Marketing Compliance Profile ADAPTER Marketing Compliance Campaign Model ADAPTER Campaign Back-office Execution Rewards Notification Service ADAPTER
  19. 19. Classified as General © Kindred Group plc 2022 19 Challenges 01 Required to communicate with regulator components with little guarantees 02 Different markets require more checks that others 03 Real-time customer engagement whilst the customer is on-site 04 Large batch-based send-outs targeting up to 3 millions customers
  20. 20. Classified as General © Kindred Group plc 2022 20 Example Half time when we need to target customer quickly for second half opportunities
  21. 21. Classified as General Time to Cry? The latency from third parties makes this impossible How do we know we’ve achieve success? How’s this going to be possible? 20+ markets, how can we guarantee any consistency?
  22. 22. Classified as General Impossible Task? No © Kindred Group plc 2022 22 It’s just another challenge we developers dream about
  23. 23. Classified as General So What Do We Do About It? © Kindred Group plc 2022 23 Let’s Dive In
  24. 24. Classified as General © Kindred Group plc 2022 24 Technique 1: Microservices Principles 01 Single responsibly for each component 02 Optimize for the responsibility E.G., connections, memory, CPU, etc 03 Reduce points of failure. Can we continue when a service is down? 04 Reduced resource usage means key services can have more instances 01 Required to communicate with regulator components with little guarantees
  25. 25. Classified as General © Kindred Group plc 2022 25 Technique 1: Example 01 Heavy reads, low writes; hence can optimise for reading with caches 02 Low reads and caching means we can optimize for many connections 03 Because it’s standalone, it’s less likely to interfere with the Executor 04 If it falls over, it’s very quick to restart which reduces the impact 01 Smaller resource usage; hence easier to have more instances Campaign Send-out Snapshot ADAPTER
  26. 26. Classified as General © Kindred Group plc 2022 26 Focus Area for Potential Bottleneck Campaign Send-out Executor ADAPTER Marketing Compliance ADAPTER Marketing Compliance Profile ADAPTER Marketing Compliance Execution
  27. 27. Classified as General Let’s Define the Terms © Kindred Group plc 2022 27 Marketing Compliance Report Marketing Compliance Profile How can we engage with a customer in a compliant way. Typical action are can be target with rewards or what channel are allowed. Can we engage with the customer at all? Aggregates all the data points that are needed to build a compliance report. Continually monitors the other systems to ensure the profile is up- to-date.
  28. 28. Classified as General © Kindred Group plc 2022 29 Component Responsibilities Campaign Send-out Executor ADAPTER Execution Marketing Compliance Profile ADAPTER Marketing Compliance ADAPTER Marketing Compliance Request a compliance report Decides on what actions can be done based on the compliance report Returns a compliance report based on the rules Runs the compliance rules for the customer’s jurisdiction Returns a valid compliance profile Aggregates the compliance profile from a myriad of services
  29. 29. Classified as General © Kindred Group plc 2022 30 Technique 2: Asynchronous Hand-offs Used two queue for asynchronous communication to hand-off long running queries Hand-offs meant that slower queries were not holding up newer ones The compliance profile was asynchronously updated in the background Needed mechanism to know when the profile was consistent Simple requests, where the profile was already complete, are faster
  30. 30. Classified as General Technique 2 – Asynchronous Hand-offs © Kindred Group plc 2022 ADAPTER Compliance Profile Retrieval Update Compliance Profile Compliance Execution ADAPTER Regulatory Engine ADAPTER Regulatory Integration Rofus ADAPTER Third Party System Compliance Report Request Compliance Profile Request Marketing Compliance Profile Retrieve Batch of Customer to Check Request a Batch of Customer to Check Marketing Compliance Service Execution Platform Compliance Profile Response Obtain Profile From the DB Update Profile in the DB Compliance Report Response Request to Update Profile in the DB
  31. 31. Classified as General Technique 2: Sequence © Kindred Group plc 2022 32
  32. 32. Classified as General © Kindred Group plc 2022 33 Is This Over?
  33. 33. Classified as General © Kindred Group plc 2022 34 No! How do we know we’ve succeeded?
  34. 34. Classified as General © Kindred Group plc 2022 35 Inspiration from the Theory of Constraints the constraint In any journey, there’s only one bottleneck
  35. 35. Classified as General © Kindred Group plc 2022 36 Inspiration from the Theory of Constraints the constraint the constraint Everything else to the constraint Identify Exploit Subordinate Elevate Don’t stop now the constraint the constraint everything else to the constraint the constraint Find your next to constraint
  36. 36. Classified as General © Kindred Group plc 2022 37 Inspiration from the Theory of Constraints Resolve Protect If you can’t reduce the constraint, then protect it Ensure efficiency in and out of the constraint The constraint will define your throughput Any improvement either side of the bottleneck is pointless Understand the bottleneck, make the change and then check If the bottleneck has moved, rinse-repeat with the new location
  37. 37. Classified as General Technique 3: Performance Monitoring © Kindred Group plc 2022 01 Highlight where the bottleneck is in the stages of the send-out 02 Be able to slice and dice by campaign, market, and real-time vs batch 03 Understand throughput numbers by market 04 Understand where time is being spent that can affect end-to-end time 05 Understand common traffic patterns to replay in our PT environment
  38. 38. Classified as General © Kindred Group plc 2022 39 Performance Monitoring Examples
  39. 39. Classified as General © Kindred Group plc 2022 40 Performance Monitoring Examples
  40. 40. Classified as General So Where is the Constraint? © Kindred Group plc 2022 ADAPTER Compliance Profile Retrieval Update Compliance Profile Regulatory Engine ADAPTER Regulatory Integration Compliance Profile Request Marketing Compliance Profile Retrieve Batch of Customer to Check Request a Batch of Customers to Check Execution Platform Compliance Profile Response Obtain Profile From the DB Update Profile in the DB Request to Update Profile in the DB
  41. 41. Classified as General What Was the Issue? © Kindred Group plc 2022 42 Batches of requests for 1,000 customers are sent every 5 seconds per market Scenario: 100,000 customers for one market 100,000 divided by 1,000 multiplied by 5 seconds = 8 minutes 40 seconds Conclusion: The last few batches will have a natural 7 to 8 minutes delay
  42. 42. Classified as General Time to Fix? Can it be Fix? © Kindred Group plc 2022 43 Of course, we can
  43. 43. Classified as General © Kindred Group plc 2022 44 Technique 4: Non-blocking Queue Non-blocking Blocking Consumes customer selections one-by-one and stores in a DB Every 5 seconds create a lock Collates a batch and sends via a message to the Regulatory Integration component to process No batching, DB or locks required Reads the batch of customer directly from the queue up to the 1K limit Leveraging the power of Solace to poll and acknowledge message
  44. 44. Classified as General © Kindred Group plc 2022 45 Technique 4: Recap Sequence Area of focus
  45. 45. Classified as General Technique 4: Sequence © Kindred Group plc 2022 46
  46. 46. Classified as General What Did the Solution Bring? © Kindred Group plc 2022 47 Batches of requests for 1,000 customers are sent continuously for the queue per market Scenario: 100,000 customers for one market 100,000 divided by 1,000 multiplied by 100 milliseconds = 10 seconds Conclusion: Reduced 7 to 8 minutes delay to 10 seconds
  47. 47. Classified as General © Kindred Group plc 2022 48 Technique 5: Only Request What You Need Concept: Based on the campaign strategy and the rules that need to be executed for the respective market just request the necessary sections of the compliance profile Advantage: Based on the campaign strategy and the rules that need to be executed for the respective Example 1: A newsletter campaign strategy doesn’t require bonus block/opt-outs, etc. Example 2: A campaign strategy that rewards a customer without any communication doesn’t need to check
  48. 48. Classified as General © Kindred Group plc 2022 49 New Problem – Real-time vs Batch Real-time needs to be processed with very low latency Real-time send-outs are spread out over the day with occasional peaks Batch targets large quantities of customers at the same time - up to 3 mil Batch sends can block real-time by flooding the queue Batch sends can tolerate a slow throughput through the system
  49. 49. Classified as General © Kindred Group plc 2022 50 Technique 6: Bulkhead Pattern “ The bulkhead pattern is based on a concept from building ships. With a bulkhead design, a ship is divided into completely segregated and watertight compartments called bulkheads. Even if the ship’s hull is punctured, because the ship is divided into watertight compartments (bulkheads), the bulkhead will keep the water confined to the area of the ship where the puncture occurred and prevent the entire ship from filling with water and sinking.
  50. 50. Classified as General Variant of the Bulkhead Pattern © Kindred Group plc 2022 51 Campaign Send-out Executor ADAPTER Unica Adapter (batch sending) ADAPTER MMC (real- time sending) ADAPTER Audience Campaign Feeder (batch sending) ADAPTER Audience Integration Platform Execution Platform Batch Queue Real-time Queue The real-time queue is a constraint that needs to be protected. With the influence bulkhead pattern, we can build a second queue dedicated to real-time; hence large batch sends affect real-time less.
  51. 51. Classified as General Santa Claus’ Present to You © Kindred Group plc 2022 52 https://microservices.io/patterns/microservices.html
  52. 52. Classified as General © Kindred Group plc 2022 53 Keys to Success 01 Life throws up challenges that software can solve with the right mindset 02 Event-driven and asynchronous messaging is perfect to reduce delays 03 Performance monitoring in production is essential to understand if you’ve solved the problem 04 You are never finished. Constant monitoring to assess the constraint is important 05 These are challenges techies love to solve
  53. 53. Classified as General © Kindred Group plc 2022 54

×