Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale

1,440 views

Published on

Building a Highly Scalable File Processing Platform with NServiceBus
How do we move this data into our systems?

Published in: Technology

Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale

  1. 1. Building a Highly Scalable File Processing Platform with NServiceBus Sam Martindale
  2. 2. AI Introduction 2 Sam Martindale Architecting Innovation 14+ Years Development Experience Co-Founder & Managing Partner, AI NServiceBus adopter since v1.x
  3. 3. AI 3 The Problem
  4. 4. AI The Problem 4 How do we move this data into our systems? External Data Files Internal Systems
  5. 5. AI The Problem 5 Automated pickup of many file types Fixed Width
  6. 6. AI The Problem 6 Record Level Logic and Workflows New Account Account Payment Loan Application Account Payment Overdue Credit Account
  7. 7. AI The Problem 7 Many Integration Points
  8. 8. AI The Problem 8 Guaranteed Delivery
  9. 9. AI The Problem 9 Scalability
  10. 10. AI 10 Why NServiceBus?
  11. 11. AI Evaluated Technologies ! Sql Server Integration Services (SSIS) ! Workflow Integration Difficulty Systems Integration Difficulty Extensibility Not just ETL ! ! ! ! 11
  12. 12. AI Evaluated Technologies 12 BizTalk/Sterling Commerce/Axway/Others ! Licensing Costs Scalability Costs Difficult to “extend” functionality easily Requires development “specialists” ! ! ! ! ! !
  13. 13. AI Evaluated Technologies 13 NServiceBus ! Easy & inexpensive to scale ! Easy to integrate with existing systems ! Provides Message Durability ! Great API for experienced .NET developers
  14. 14. AI 14 System Design
  15. 15. AI The System 15 Alerts Endpoint Processing Endpoint Scheduler Endpoint Parsing Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint
  16. 16. Components Scheduler Manages scheduled “jobs” for picking up client files from FTP, SFTP, FileShare, etc Alerts Endpoint Processing Endpoint Scheduler Endpoint Parsing Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint 16 Why we needed it: ! 1. Needed the ability to schedule file pickups dynamically
  17. 17. Components FileRetrieval Performs the actual file pickup and decryption at the request of the scheduler Alerts Endpoint Processing Endpoint Scheduler Endpoint Parsing Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint 17 Why we needed it: ! 1. Needed the ability to pickup files 2. Needed the ability to decrypt files
  18. 18. Components Alerts Endpoint Processing Endpoint Scheduler Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint 18 FileParsing Performs the parsing of the file according to the configured client file format and layout. Divides the file into logical “record sets” for processing Why we needed it: ! Had to be able to dynamically parse any given file Parsing Endpoint
  19. 19. Components Processor Processes the record sets sent by the parser component by performing calculations and persisting records to the database Alerts Endpoint Processing Endpoint Scheduler Endpoint Parsing Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint 19 Why we needed it: ! Needed the ability to persist data Serves as an extension point for current/new workflows
  20. 20. Components FileManagement Tracks the completion of the file processing in a saga by listening to various parsing events Alerts Endpoint Processing Endpoint Scheduler Endpoint Parsing Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint 20 Why we needed it: ! Had to keep track of the overall status for file processing Needed the ability to monitor for failures or partial failures
  21. 21. Components Alerts Receives events from other components, sending notifications and emails to the end users for system monitoring and alerting Alerts Endpoint Processing Endpoint Scheduler Endpoint Parsing Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint 21 Why we needed it: ! Needed the ability to notify users of various events
  22. 22. Components Alerts Endpoint Processing Endpoint Scheduler Endpoint Parsing Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint 22 WebApi WebApi component with SignalR. Delivers real time events and updates to the web front end Why we needed it: ! Needed to notify users of created alerts in real-time Needed to allow users to quickly onboard clients !
  23. 23. AI The System 23 Alerts Endpoint Processing Endpoint Scheduler Endpoint Parsing Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint
  24. 24. AI 24 What did we learn along the way?
  25. 25. AI Division of Labor 25 Problem ! Monolithic Endpoints A naive implementation of our service placed far too many handlers within the same endpoint, making it impossible to prioritize messages and scale portions of the system ! Api Scheduler Alerts Everything
  26. 26. AI Division of Labor 26 Resolution ! A Service is NOT an endpoint In order to ensure our success, we had to split the system into many components, each with its own distinct responsibilities (and queues!) ! Granular Scaling By splitting into several disparate components, we were able to meet our scalability needs more appropriately. (e.g. scaling only the parsing endpoint)
  27. 27. AI Saga Responsibilities 27 Problem ! Scaling the File Processing Saga Our initial (read: naive) implementation was responsible for far too much business logic, making it difficult to scale
  28. 28. AI Saga Responsibilities 28 Resolution ! Keep it Simple! Sagas should be as clean and tight as possible ! Scaling Issues The more complicated the saga, the harder they are to scale ! “Observer" Implementation In our case, we found that a “passive” saga responding to events and simply keeping track of state was far easier to maintain and scale
  29. 29. AI Saga Methodology 29 Problem ! Saga Contention Our initial (read: naive) implementation responded to too many events, creating a large degree of contention and a high number of rollbacks Processing Endpoint File Management Endpoint 3 3 90 32 13 2
  30. 30. AI 30 Resolution ! “Check” Pattern In order to reduce contention, we implemented a “check” pattern using timeouts within the saga Processing Endpoint File Management Endpoint 3 90 32 13 2 Saga Methodology
  31. 31. AI Deployment Automation 31 Problem ! Deployment Challenges Hindering Proper Design The team (and client) were initially skeptical of breaking the system into many disparate deployable components, due to the increased difficulty of deployment ! ! Svc 1 Svc 2 Svc 4 Svc 3 Svc 4 Svc 5 Svc 1 Svc 4 Svc 5 Svc 1 Svc 2 Svc 4 Svc 1 Svc 2 Svc 3 Svc 1 Svc 3 Svc 5
  32. 32. AI Deployment Automation Resolution ! Simple Automation By leveraging the NServiceBus.Powershell bits along with our own extended Powershell scripts, we were able to automate the deployment of all of our NServiceBus projects 32 ! • Configuration Transformations • Distributed Deployments • Backup Strategy
  33. 33. AI Firewalls, Servers & Failure, Oh My! 33 Problem ! Deploying Into New Environments is Hard! Misconfiguration, network issues, security issues… All these make moving into a new environment cumbersome and error prone!
  34. 34. AI Firewalls, Servers & Failure, Oh My! 34 Resolution ! ServicePulse, ServiceControl & Custom Checks ! • MSDTC Firewall Checks • RavenDB Firewall Checks • FTP Accessibility Checks • Databus File Share Accessibility Checks
  35. 35. AI 35 The Result?
  36. 36. AI The Result 36 Business Impact ! Faster Processing Times > 10K records/sec parsed and processed, per worker machine ! Happier Users Full system visibility and no lost data ! Happier Developers Clean, maintainable system ! Happier Executives Faster onboarding times for new clients
  37. 37. Thank You

×