Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Capital One's experience with building a Reactive System with Akka


Published on

A newly assembled team. No reactive programming experience. A deadline that cannot be missed. In this session, we will describe the experience of learning reactive programming and using it to build an API for a credit card fulfillment process. Tight timelines and dependent system constraints increased the challenge of implementation. The team chose to implement the solution with Akka, Lagom and persistence using PostgreSQL. We will discuss the patterns and techniques leveraged including the promise pattern, Akka Clustering, Java Futures, and Error Handling / State Management. We will also demonstrate the challenges faced with upstream/downstream TPS constraint and resolution provided with Scheduler & Config driven Throttling Mechanism. We will review the options considered, learnings through this journey and what lies ahead for the service.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Capital One's experience with building a Reactive System with Akka

  1. 1. Andy Bonham, Raja Chattopadhyay, Thiaga Manian October 20, 2017 Reactive Summit 2017: Capital One's experience with building a Reactive System with Akka
  2. 2. 2 Capital One at a glance • A leading diversified bank with $357.0 billion in assets, $245.6 billion in loans and $236.8 billion in deposits1 • Major operations in 15 U.S. cities, Canada, U.K. • More than 65 million customer accounts and 45,000 associates • A FORTUNE 500 Company - #112 1) Source: Company reported data as of Q4’16 2
  3. 3. 2016 Acquires Critical Stack and Paribus 2015 Acquires GE Capital’s Healthcare Financial Services, Level Money and Monsoon 2014 Acquires Adaptive Path, a digital design leader and AmeriCommerce, an online e-commerce company 2013 Acquires Beech Street Capital, an originator, underwriter and servicer of multifamily commercial real estate loans 2012 Acquires ING DIRECT, HSBC US Card portfolio 2010 Enters into card partnerships with Kohl's and Sony in the US and Hudson's Bay Company and Delta in Canada 2009 Acquires Chevy Chase Bank in the Washington, DC area 2006 Acquires North Fork Bank, one of the largest banks in the New York metro area 2005 Acquires Hibernia National Bank, #1 bank in Louisiana 2002 Launches its Small Business credit card 2000 Introduces slogan, “What’s in your wallet?” 1998 Enters Auto Finance Market 1996 Expands into Canada and the U.K. 1995 Spins off from Signet Bank 1994 Initial Public Offering (IPO) We have transformed the company into a top 10 bank
  4. 4. 4 Background • Intent of the Credit Card Fulfillment Process is to aggregate data from different sources for triggering fulfillment process • Tight deadline was unmovable • Newly formed team – September 2016 • Team consisted of – Two associates joining from different teams – Two contractors • No reactive programming experience
  5. 5. 5 Mission: Execute Credit Card Fulfillment Process • Feature is to update Fulfillment System with details captured as part of the account decisioning process. • Many API calls were needed to retrieve the information from upstream systems and to update the downstream systems. • Challenge and Encouragement from leadership to evaluate and productionize the solution, but has to be done by project deadline. 5
  6. 6. 6 Why reactive ? • Evaluating frameworks for solving the broader use cases for Application Decisioning modernization, need to start somewhere small • Updating the System of Record was right for reactive work – Some of the calls can be made in parallel to reduce the processing time. – Need to be highly responsive. Applicant is waiting online. (Towards the middle or the project we decided to move from Real time to Batch)
  7. 7. 7 Monoliths are also hard to change Monoliths can have many dependencies*This illustration is from A monolithic application puts all its functionality into a single process… … and scales replicating the monolith on multiple servers A monolithic application can be challenging …
  8. 8. 8 A microservices architecture puts each element of functionality into a separate service… *This illustration is from … and scales by distributing these services across servers, replicating as needed. Microservices to the rescue! But are they the silver bullet?
  9. 9. 9 Using a reactive architecture with microservices can help achieve additional benefits Service A Service B Service C Orchestrator blocking Service A Service C Service B consume produce consume produce consume produce Event Stream
  10. 10. 10 The Reactive Manifesto highlights the key principles of a reactive architecture Responsive Elastic Message Driven Resilient
  11. 11. 11 A reactive architecture has both benefits and tradeoffs • Better resource utilization, saving cost – Can get higher efficiency out of CPUs (multi-core processors), doing more with less • More Agile – Decoupling enables services to be updated independently • Faster response times as requests can run in parallel – Back pressure can be used for flow control • Fast producers don’t overwhelm slower consumers • Enables a consumer to control queue bounds • Extensible – New components can be added that listen to the event stream without re-writing the system BENEFITS TRADEOFFS • Async programming is a mind shift • Complexity – The flow of the system is shifted from a central place to distributed services
  12. 12. 12 Four categories of tools you can use to build reactive apps Reactive UI Java script front-end libraries: Languages that support reactive models natively: Reactive Layers that run on top of the JDK & implement the Reactive-Streams spec: Reactive Extensions Java: RxJava JavaScript: RxJS C#: Rx.NET C#(Unity): UniRx Scala: RxScala Clojure: RxClojure C++: RxCpp Lua: RxLua Ruby: Rx.rb Python: RxPY Go: RxGo Groovy: RxGroovy JRuby: RxJRuby Kotlin: RxKotlin Swift: RxSwift PHP: RxPHP Elixir: reaxive Dart: RxDart ReactiveX for platforms and frameworks RxNetty RxAndroid RxCocoa
  13. 13. 13 Key technology components leveraged • Exposes Akka Actor as a REST API, accelerating development • Making async calls • Using a reactive hybrid approach – Saga Actor handles fanning & merging Java Future Interface • Enables REST API calls to be async and not wait for the response Akka Clustering Sharding • One node per AZ, 3 AZs total • Used in combination with ELB (round robins original request to a node) • Location transparency – actors can exist in any of the AZs ⎻ Leverage cluster sharding - actors are referenced by an ID instead of an ActorRef • Enables actors to scale and get automatically restarted if they die • Cluster sharding is using Akka Distributed Data ⎻ Shares data between Akka Cluster Nodes using a key-value store Constraints Throttling - Each Actor can only call an API at a certain rate Batch to real-time - Batch file triggers the process, which has ~50k records in it daily
  14. 14. 14 Advanced features using Lagom • Maximized development speed. • Created couple of POC with Lagom within short time span • REST end point developed using Lagom • With Lagom developed CQRS & Event Sourcing pattern
  15. 15. 15 Context Diagram – Credit Card Fulfillment Process PostGRE Retry Batch REST Client AppCapture API Identity API Encryption API Research API AppData API Datastore API Registry API Income API REST API AppCapture Actor Encryption Actor Research Actor AppData Actor Datastore Actor Registry Actor Income Actor Saga Actor Identity Actor
  16. 16. 16 Capabilities of Credit Card Fulfillment • Receive and validate request from batch – Batch processes calls the Credit Card Fulfillment process with the parameters (key fields) – Credit Card Fulfillment receives above parameters from batch and validate empty/null fields. • Retrieve application details from other microservices • Update SOR (System of Record) • Update Registry • Resiliency – Retry upstream/downstream API call incase of failure. – Max Retry frequency driven by configuration. – Handle and track error scenarios in PostgreSQL Database • Fulfill downstream systems TPS SLA using configuration driven throttling mechanism. – Maintain incoming message in a Java Queue – Control API call frequency based on configuration parameter.
  17. 17. 17 REST API Context Diagram – Correspondence App State AppDetails API Write Side Read Side Reporting Query API External Letter Dispatcher API Actor System Correspondence Actor AppDetails Actor SendLetter Actor REST API Event Sourcing LetterRequest LetterResponse AppId App Details Capabilities 1. Service Orchestration 2. Event History 3. Automatic API Call Retry 4. Real Time Status Check 5. Reporting 6. Restart Service Call From Last Failure 7. CQRS Pattern 8. Reporting Capability
  18. 18. 18 Lessons Learned • Learning curve • Unit testing and Performance Test metrics capturing was difficult • Using throttling was limiting Akka performance • Tight timeline caused few features to be dropped – Akka persistence - Recovery of objects during failure – Akka clustering - Providing resiliency and load balancing 18
  19. 19. 19 Questions