Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Integration daysBizTalk Used for Real Time Payments

4,586 views

Published on

Presented at Integrasjonsdagene 2012 in Halden, Norway. I gave the second part which is titled "Our Experiences and Challenges With BizTalk"

Published in: Technology, Economy & Finance
  • Be the first to comment

  • Be the first to like this

Integration daysBizTalk Used for Real Time Payments

  1. 1. Introducing Luup BizTalk Used for Real Time Payments Luup provides payment services for mobile phones in emerging markets in Asia and the Middle East. This presentation will discuss challenges in building a mobile payment platform, which supports high transaction volumes and a minimum of latency, using BizTalk for internal workflow services in a SOA architecture.© Luup International Ltd. #1
  2. 2. Introducing Luup Topics • Introduction to Luup – Real-time Payments – The Luup System Platform – The Main Architectural Driving Factors – How Luup Uses BizTalk – The Development Teams • Our Experiences and Challenges With BizTalk – Performance – Consistency – Maintainability – Automated Testing – Automated Deployment© Luup International Ltd. #2
  3. 3. Introducing Luup Luup • Started in Norway > 10 years ago • Development in Oslo, HQ in London, Sales Office in Dubai • Experience with several mobile payment products • Main product is a global remittance eco-system • Offers bank-grade mobile payment solutions • A managed service • An integrated banking platform • Our projects are centered around: - Integration towards new partners and customers - Make business processes work on our standard platform - UI: SMS, Mobile App and Web© Luup International Ltd. #3
  4. 4. Introducing Luup Real-time Payments Examples • Remittances - customers send money to their families conveniently, quickly and securely using their mobile devices • Domestic Payments - customers pay people, bills, charities and merchants • Disbursements - customers receive their wages or social benefits into an account linked to their mobile device© Luup International Ltd. #4
  5. 5. Introducing Luup Driving Factors Architectural Decisions • Decided to use standard software as much as possible (came from a history where everything was in-house and custom made) • Robust orchestration of services is important • Building a scalable platform is vital, since our business model relies on a high volume of transactions to be profitable© Luup International Ltd. #5
  6. 6. Introducing Luup The Luup Platform Luup Payment Processing Platform Employee UI End user UI SMS web site web site / web services server Telco Q Q Q of ser ration Q Q Q BizTalk vices st MsgBox Partner Partner Bank web Partner Orche Core/BPM Gateway services MsgBox Gateway Gateway http http In Ty teg Bi pic rat zT al io Luup al n Core banking k usa – system Services ge of© Luup International Ltd. #6
  7. 7. Introducing Luup The Luup Team • Our internal developers: – Had no experience with BizTalk 2 years ago – Were mostly experienced .Net developers – learned quickly – Now this background helps in deciding whether code should be made in BizTalk shapes, C# components or services • Offshore team with 3-5 developers: – Developed the Core/BPM application© Luup International Ltd. #7
  8. 8. Introducing Luup End of Introduction • Now over to challenges we have faced and experiences made during the last 2 years© Luup International Ltd. #8
  9. 9. IntroducingExperiences and Challenges Luup’s Luup in Using BizTalk • Introduction • Performance • Consistency • Maintainability • Automated Testing • Automated Deployment© Luup International Ltd. #9
  10. 10. Introducing Luup The Luup Platform - Recap Luup Payment Processing Platform Employee UI End user UI SMS web site web site / web services server Telco Q Q Q Q Q Q BizTalk MsgBox Partner Partner Bank web Partner Core/BPM Gateway Gateway services MsgBox Gateway http http Core banking Luup system Services© Luup International Ltd. #10
  11. 11. Introducing Luup PERFORMANCE© Luup International Ltd. #11
  12. 12. Introducing Luup Challenges • Complex business process orchestrations – Many (10-20) web service calls in each workflow • Important for our design choices • Low latency required for some scenarios – Serve requests from web and mobile app frontends • Performance tuning of BT is complicated! – Each system/setup is unique© Luup International Ltd. #12
  13. 13. Introducing Luup Current Configuration • Shared server for Partner Gateway and BPM • Active-Passive cluster – Due to licence costs, until the number of transactions increases • Separate hosts for receiving, sending and processing • We use the defaults for most settings© Luup International Ltd. #13
  14. 14. Introducing Luup Design Considerations • The database will be our bottleneck – The database is the hardest component to scale – We need to limit database roundtrips© Luup International Ltd. #14
  15. 15. Introducing Luup Latency Example • Message box round trips (polling every 0,5s): – If 10 service calls: • Adapter polls for messages to send, orchestration polls for response messages • 10 requests and 10 responses = 20 msg box roundtrips – 20 msg box roundtrips * 0,25 sec = 5 sec (for polling msg box only) – If 1 sec used on actual processing: 5 + 1 = 6 sec • How can we avoid this latency?© Luup International Ltd. #15
  16. 16. Introducing Luup Low Latency Considerations • Alternatives for calling web services from workflows – WCF Adapters • Polling interval can be tuned down to 50 ms – Use .Net code – «Service Caller» • Used in our solution© Luup International Ltd. #16
  17. 17. Introducing Luup Using Adapters • Pros – The default and recommended way when using BizTalk • Cons – Orchestration is persisted for each web service call – Message Box roundtrips for request and response message© Luup International Ltd. #17
  18. 18. Introducing Luup The «Service Caller» • Uses C# code to do SOAP calls • Two versions: – Old version using WebRequest – New version using WCF • We only use it in the CORE/BPM application – Only calls to locally deployed services • We use BAM for logging requests and responses© Luup International Ltd. #18
  19. 19. Introducing Luup Service Caller - Advantages • Low latency and DB load – No message box roundtrips – Orchestration is not persisted after the request is sent© Luup International Ltd. #19
  20. 20. Introducing Luup Service Caller - Disadvantages • Need to be careful about when the orchestration is persisted – When replaying request we need to use the original request ids – Generate request ids up-front and start using the service caller after the ids have been persisted© Luup International Ltd. #20
  21. 21. Introducing Luup Service Caller - Disadvantages • We take control of resource usage throttling away from BizTalk – Threads are used while waiting for web service replies – Use short timeouts (< 60s) and short retry intervals© Luup International Ltd. #21
  22. 22. Introducing Luup Summarizing Performance • Performance tuning BizTalk is hard and time consuming • Using .Net code to call web services has improved our latency • We need to focus on performance in the near future as the number transactions increases© Luup International Ltd. #22
  23. 23. Introducing Luup CONSISTENCY© Luup International Ltd. #23
  24. 24. Introducing Luup Consistency Luup Payment Processing Platform Employee UI End user UI SMS web site web site / web services server Telco Q Q Q Q Q Q BizTalk MsgBox Partner Partner Bank web Partner Core/BPM Gateway Gateway services MsgBox Gateway http http called with serv Inside circle: Id Core banking Luup system Services© Luup International Ltd. #24
  25. 25. Introducing Luup Consistency Luup Payment Processing Platform Employee UI End user UI SMS web site web site / web services server Telco Q Q Q Q Q Q BizTalk MsgBox Partner Partner Bank web Partner Core/BPM Gateway Gateway services MsgBox Gateway http http Throug Core banking Luup system Services h circle:© Luup International Ltd. #25 A
  26. 26. Introducing Luup MAINTAINABILITY© Luup International Ltd. #26
  27. 27. Introducing Luup Size of Orchestrations • Complex business logic gives complex orchestrations – Results in orchestrations which are hard to: • Understand • Change • Test© Luup International Ltd. #27
  28. 28. Introducing Luup Shapes vs C# Code • Use shapes as much as possible? – Results in huge orchestrations which are • Hard to test • Hard to read • Has duplicated logic – Pros • All the logic is visible in the orchestration designer and debugger – Call Orchestration can be used to split out orchestration parts© Luup International Ltd. #28
  29. 29. Introducing Luup Shapes Example© Luup International Ltd. #29
  30. 30. Introducing Luup Shapes vs C# Code • Or use C# code as much as possible? – Keeps the orchestrations dense and focused – Use the orchestration only for controlling the transaction (at a high level) • Which is the main reason why we needed BizTalk – C# code is easy to unit test© Luup International Ltd. #30
  31. 31. Introducing Luup Using Internal Contracts • Vital for having a clean modular architecture • Hide gory details from external systems – Internal contracts which are easier to understand • Mapping complex external contracts • Greatly improves maintainability – Anti-corruption layers • Keeps the core stable even when external services change • Small and reusable orchestrations© Luup International Ltd. #31
  32. 32. Introducing Luup Using Internal Contracts Luup Payment Processing Platform Employee UI End user UI SMS web site web site / web services server Telco Q Q Q Q Q Q BizTalk MsgBox Partner Partner Bank web Partner Core/BPM Gateway Gateway services MsgBox Gateway http http Core banking Luup system Services© Luup International Ltd. #32
  33. 33. Introducing Luup Using Internal Contracts Contracts Contracts Reusable orchestrations Contracts© Luup International Ltd. #33
  34. 34. Introducing Luup Summarizing Maintainability • Keep orchestrations small • Don’t be afraid of using C# code – But be careful about orchestration persistence when changing state in C# code • Use internal contracts© Luup International Ltd. #34
  35. 35. Introducing Luup I don’t usually test my code. But when I do, I do it in Production. AUTOMATED TESTS© Luup International Ltd. #35
  36. 36. Introducing Luup Unit and Integration Tests • Using unit tests for – Maps – C# code • Integration tests – Using BizUnit for testing orchestrations© Luup International Ltd. #36
  37. 37. Introducing Luup Testing Maps – The Built-in Test Func© Luup International Ltd. #37
  38. 38. Introducing Luup Testing Maps – The Built-in Test Func • Disadvantages of built-in test functionality: – Compiles maps inheriting from test base class • References test assembly Microsoft.BizTalk.TestTools.dll which must be deployed to production – Exceptions are swallowed • Hard to figure out why a test failed© Luup International Ltd. #38
  39. 39. Introducing Luup Testing Maps – Our Solution • Custom-made library • Calls XLANG component to do transformation • Fluent language for writing assertions© Luup International Ltd. #39
  40. 40. Introducing Luup Testing Maps – Our Solution Schema validation Field validations© Luup International Ltd. #40
  41. 41. Introducing Luup Testing Orchestrations • Complex workflows – Requires automated tests • Custom made fluent interface on top of BizUnit – Reuse of code – Terse tests which are refactor-friendly • Currently we have around 350 BizUnit orchestration tests© Luup International Ltd. #41
  42. 42. Introducing Luup Testing Orchestrations - Example[Test]public void RemittanceSend_WhenInvalidBankTransactionDate_ShouldReturnGeneralError(){ const string invalidTransactionDate = "12/24/2012"; var testCase = new BizUnitTestCaseBuilder() .WithTestCaseSteps(SharedTestSteps.CreateAndCleanupDirectories()) Initialize the test input file .WithCreateXmlFileStep( new CreateXmlFileStepBuilder(@"TestDataMtoRemittanceSendRequest_ToAccount.xml", SharedConstants.Pgw.ReceiveDirectory + TestContext.CurrentContext.Test.Name + ".xml") .WithUpdateXml("SenderBankTransactionDate", invalidTransactionDate, "//*[local-name()=SenderBankTransactionDate]") ) Wait for orchestration .WithWaitForFileStep(SharedConstants.Pgw.SendDirectory) execution to finish .WithXmlFileValidation(new XmlFileValidationBuilder(SharedConstants.Pgw.SendDirectory).WithSchemaValidation(@"......SharedBizTalkSchemasPartnerGatewayMtoRemittanceSendResponse.xsd", "http://www.luup.com/schemas/") .WithXPathValidation("ErrorId", MtoRemittanceSendResultCode.GeneralFailure.ToString(), "//*[local-name()=ErrorId]") ) .Build(); Verify the result message var bizUnit = new BizUnit.BizUnit(testCase); bizUnit.RunTest();} © Luup International Ltd. #42
  43. 43. Introducing Luup Build Server Integration • After each code checkin: – Build packages – Deploy all BizTalk applications to a dedicated BizUnit server – Run BizUnit integration tests© Luup International Ltd. #43
  44. 44. Introducing Luup Summarizing Automated Testing • The built-in unit testing functionality for maps is no good – Using BizUnit is a better choice • Both maps and orchestrations can and should have automated tests • Build server runs automated tests© Luup International Ltd. #44
  45. 45. Introducing Luup Test Environments TFS Build Source Servers Control Code Changes Deployment Production AUTOMATED DEPLOYMENT© Luup International Ltd. #45
  46. 46. Introducing Luup Our Deployment Challenges • Our solution consists of about 20 BizTalk applications • 9 test environments • Time consuming deployments to Production© Luup International Ltd. #46
  47. 47. Introducing Luup How We Do Deployments • Each application use BizTalk Deployment Framework (BTDF) • Using the Sprinkler framework to control deployment of multiple applications • The build server creates installation packages each time we check in code – Also deploys and runs BizUnit integration tests • Automated deployments to test environments© Luup International Ltd. #47
  48. 48. Introducing Luup The Sprinkler Framework • Open source on CodePlex • A set of PowerShell script built on top of BTDF • A bit immature – We had to make a couple of changes to the scripts in order to make it work with Active-Passive clusters • Great when having multiple applications and servers • Great for automated deployment© Luup International Ltd. #48
  49. 49. Introducing Luup Deploying to Production • Active-passive cluster • Using Sprinkler for parallel deployment to the nodes • IT Ops handle deployment – Developers should not touch the Production environment© Luup International Ltd. #49
  50. 50. Introducing Luup Summarizing Automated Deployment • BTDF and Sprinkler are good frameworks • Automate your deployments • Use the same binaries when deploying to test and production • Deploy early and often© Luup International Ltd. #50
  51. 51. Introducing Luup Main Summary • Flexible and scalable solution • Usage of BizTalk both as a traditional integration tool and orchestration of services • Reduce latency by reducing message box roundtrips • Improve maintainability by creating a modularized architecture with clean interfaces • Testing of maps and orchestrations can and should be automated • Automated deployment is useful when having many BizTalk applications© Luup International Ltd. #51
  52. 52. Introducing Luup Questions?© Luup International Ltd. #52

×