Successfully reported this slideshow.
Your SlideShare is downloading. ×

Tales from the trenches creating complex distributed systems

Loading in …3

Check these out next

1 of 39 Ad

More Related Content

More from Particular Software (20)

Recently uploaded (20)


Tales from the trenches creating complex distributed systems

  1. 1. Applying ADSD and NServiceBus 1
  2. 2. Introduction 2
  3. 3. Who we are? ● For around 20 years, Strasz Assessment Systems has provided solutions for the Assessments industry, so typically, our clients offer education, certification or licensure programs. ● The system we will be talking about today is the one we have built for one of our largest clients, the AICPA. ● The AICPA produces, delivers and scores the Uniform CPA exam. ● Our end to end solution for them includes content authoring of test questions, automated assembly into an exam, delivery in a test center, scoring, reporting and invoicing ● Some of our other clients include AAMC who offer the MCAT exam, Inteleos who offer sonography and physician certifications, and NCCPA who certify physician assistants. 3
  4. 4. Technology Stack 4
  5. 5. Technology Stack ● We are currently in the middle of a cloud migration effort. So our latest technology stack is NServiceBus, .Net core, Azure Functions, Azure Cosmos Db, Azure Service Bus, Azure Cognitive Search, Angular & Typescript and MonoRepo with NX. ● On the other hand, the CPA exam was computerised in the early 2000s so there are a lot of legacy technologies in play. ● Our on-prem stack includes NServiceBus, RabbitMQ, RavenDb and SQL Server ● We also use IBM CPLEX engine which uses linear programming and user defined constraints to automate exam creation. 5
  6. 6. Need for SOA 6
  7. 7. CPA Exam Administration 7
  8. 8. AICPA Systems and Data Flow 8
  9. 9. Example: Test Question (Item) 9
  10. 10. CMS: Need for SOA ● CPA exam computerized around 20 years ago ○ Model mirrors database structure ○ Business Rules Implicit in the system ● Example issue with CMS ○ Different aspects of an Item that CMS needs to support ■ Question ■ Scoring Rules ■ Inventory ■ Statistics ■ Reference management ■ Classification management ○ Technical coupling of separate business capabilities. 10
  11. 11. CMS: Need for SOA ● Example issue with CMS ○ Item development goes through various lifecycle stages from Conception to PreTest to Operational ■ Complex and inter-dependent workflows, reviews, validation rules ○ Snapshot for rescore or re-reporting ■ Candidate may request rescore for an exam they took in the past. ○ Increased cost of change 11
  12. 12. Duplication: Need for SOA ● Example issue with Adaptive Routing 12
  13. 13. Duplication: Need for SOA ● Example issue with Adaptive Routing ○ Routing rules and logic are imported into every system. ○ In SOA, a single service owns this data and behavior, not systems. 13
  14. 14. CMS: Scoring Rules (Before) 14 “Whatever the cause, software that lacks a concept at the foundation of its design is, at best, a mechanism that does useful things without explaining its actions.” ― Eric Evans, Domain-Driven Design: Tackling Complexity in the Heart of Software
  15. 15. CMS: Scoring Rules Service (After) 15
  16. 16. Challenges we faced 16
  17. 17. Challenges ● Getting buy in from business. ○ Though we have an excellent product team, we rightly needed to show that we were not further complicating the system. ○ External review of our codebase and SOA proposal by Paul Rayner who is one of the leading voices in the DDD community. ● Getting developer buy-in, thanks to our practice and commitment to Continuous Learning we were able to slowly but surely get the team on-board. ● Doing the actual work was challenging too. Unlike tactical DDD patterns, it was difficult to find a reference architecture for ADSD ○ Trust the team’s collective background in design and we engaged Particular software (Mauro Servienti) to consult with us I think a couple of hours every week during implementation 17
  18. 18. Finding Service Boundaries 18
  19. 19. Finding Service Boundaries ● Udi talks about slicing an entity based on which properties participate in a transaction and which ones never will. ● For folks with DDD background, you can pay attention to language. For example in the context of Questions, an item has response fields, stem, stimulus, widgets and exhibits. In the context of Scoring rules item has Measurement Opportunities, Keys, Assertions. ● Other ways to find boundaries if you have a large relational database is to look at all the tables and see which ones have more relationships between them. They are likely to be more cohesive. ● Remember the share nothing approach above all else. Only Ids can be shared. When correct service boundaries are established they will seem obvious. ● We even discovered a service boundary when deep into implementation. Routing rules between segments in an exam for example - intro, break, timeout, survey are different from routing rules between test questions. 19
  20. 20. ADSD Course: Benefits 20
  21. 21. ADSD Course: Benefits ● Our architects attended the course in NYC. ● Having watched Udi’s presentations over the years the course helped connect a lot of the concepts we had heard from him. ● We had extracted one bounded context/service from our Cms monolith, the one for Scoring rules. ● We were struggling with Composite UI questions in extracting more boundaries. ● Technical services like ItOps and Branding critical to eliminating data duplication ● Having our architects attend the course also meant that it was easier to bring the rest of the development team along. 21
  22. 22. Rolling your own messaging framework 22
  23. 23. Rolling your own messaging framework ● Our history is we had MSMQ on-prem, then RabbitMq on prem (since .Net core does not support DTC), then Azure Storage Queues (because it seemed ridiculously inexpensive) and now we are using Azure Service Bus. ● Easier to switch transport because it sits behind an NServicebus abstraction. ● Messaging design patterns like Sagas/ProcessManager, Outboxing and even behaviors like ReplyToOriginator are complex and not easy to roll on your own. ● Each transport offers different delivery guarantees and has different limitations, so you are likely going to write a lot of plumbing code. ● Time spent writing plumbing code = time not spent working on business logic. ● Best to source infrastructure code from the community. You cannot keep up yourself. for ex - even log4net does not support structured logging yet. 23
  24. 24. Applying NServiceBus 24
  25. 25. Sagas: Process Manager ● Compare with Point to Point interaction ● CQRS journey example (Before) - 25
  26. 26. Sagas: Process Manager ● CQRS journey example (After) - 26
  27. 27. Sagas: Process Manager Example ● Vendor Import 27
  28. 28. Sagas: Domain Model Example ● Percent Correct Scoring Policy 28
  29. 29. Service with Sagas as Aggregates 29
  30. 30. Sagas: Testability 30
  31. 31. Sagas: Benefits ● Model Business Process and manage Process State ● Handle out of order messages in the overall operation. ● Engage domain experts for compensating actions ● No need for data retrieval, save, concurrency checks ● Small aggregate boundaries ● Testable 31
  32. 32. Composite UI 32
  33. 33. Request Interceptor: Server 33
  34. 34. Request Interceptor: Server 34
  35. 35. Request Interceptor: Client 35
  36. 36. Other Code Samples 36
  37. 37. Request Interceptor (with Subscription): Server 37
  38. 38. Service with Functions As ACs 38
  39. 39. NServiceBus Style Handlers: Client 39