A newly assembled team. No reactive programming experience. A deadline that cannot be missed. In this session, we will describe the experience of learning reactive programming and using it to build an API for a credit card fulfillment process. Tight timelines and dependent system constraints increased the challenge of implementation. The team chose to implement the solution with Akka, Lagom and persistence using PostgreSQL. We will discuss the patterns and techniques leveraged including the promise pattern, Akka Clustering, Java Futures, and Error Handling / State Management. We will also demonstrate the challenges faced with upstream/downstream TPS constraint and resolution provided with Scheduler & Config driven Throttling Mechanism. We will review the options considered, learnings through this journey and what lies ahead for the service.

1. Andy Bonham, Raja Chattopadhyay, Thiaga Manian
October 20, 2017
Reactive Summit 2017: Capital One's experience with
building a Reactive System with Akka

2. 2
Capital One at a glance
• A leading diversified bank with
$357.0 billion in assets, $245.6
billion in loans and $236.8 billion
in deposits1
• Major operations in 15 U.S. cities,
Canada, U.K.
• More than 65 million customer
accounts and 45,000 associates
• A FORTUNE 500 Company - #112
1) Source: Company reported data as of Q4’16
2

3. 2016 Acquires Critical Stack and Paribus
2015 Acquires GE Capital’s Healthcare Financial Services, Level Money
and Monsoon
2014 Acquires Adaptive Path, a digital design leader and AmeriCommerce,
an online e-commerce company
2013 Acquires Beech Street Capital, an originator, underwriter and servicer of
multifamily commercial real estate loans
2012 Acquires ING DIRECT, HSBC US Card portfolio
2010 Enters into card partnerships with Kohl's and Sony in the US and
Hudson's Bay Company and Delta in Canada
2009 Acquires Chevy Chase Bank in the Washington, DC area
2006 Acquires North Fork Bank, one of the largest banks in the New York metro area
2005 Acquires Hibernia National Bank, #1 bank in Louisiana
2002 Launches its Small Business credit card
2000 Introduces slogan, “What’s in your wallet?”
1998 Enters Auto Finance Market
1996 Expands into Canada and the U.K.
1995 Spins off from Signet Bank
1994 Initial Public Offering (IPO)
We have transformed the company into a top 10 bank

4. 4
Background
• Intent of the Credit Card Fulfillment Process is to aggregate data from
different sources for triggering fulfillment process
• Tight deadline was unmovable
• Newly formed team – September 2016
• Team consisted of
– Two associates joining from different teams
– Two contractors
• No reactive programming experience

5. 5
Mission: Execute Credit Card Fulfillment Process
• Feature is to update Fulfillment System with details captured as
part of the account decisioning process.
• Many API calls were needed to retrieve the information from
upstream systems and to update the downstream systems.
• Challenge and Encouragement from leadership to evaluate and
productionize the solution, but has to be done by project deadline.
5

6. 6
Why reactive ?
• Evaluating frameworks for solving the broader use cases for Application Decisioning
modernization, need to start somewhere small
• Updating the System of Record was right for reactive work
– Some of the calls can be made in parallel to reduce the processing time.
– Need to be highly responsive. Applicant is waiting online. (Towards the middle or the project we decided to
move from Real time to Batch)

7. 7
Monoliths are also
hard to change
Monoliths can have
many dependencies*This illustration is from http://martinfowler.com/articles/microservices.html
A monolithic application puts all its
functionality into a single process…
… and scales
replicating the
monolith on
multiple
servers
A monolithic application can be challenging …

8. 8
A microservices architecture puts
each element of functionality into
a separate service…
*This illustration is from http://martinfowler.com/articles/microservices.html
… and scales by
distributing these
services across
servers, replicating
as needed.
Microservices to the rescue! But are they the silver bullet?

9. 9
Using a reactive architecture with microservices can help achieve
additional benefits
Service A
Service B
Service C
Orchestrator
blocking
Service A Service C
Service B
consume
produce
consume
produce
consume
produce
Event Stream

10. 10
The Reactive Manifesto highlights the key principles of a
reactive architecture
http://www.reactivemanifesto.org/
Responsive
Elastic
Message
Driven
Resilient

11. 11
A reactive architecture has both benefits and tradeoffs
• Better resource utilization, saving cost
– Can get higher efficiency out of CPUs (multi-core processors),
doing more with less
• More Agile
– Decoupling enables services to be updated independently
• Faster response times as requests can run in parallel
– Back pressure can be used for flow control
• Fast producers don’t overwhelm slower consumers
• Enables a consumer to control queue bounds
• Extensible
– New components can be added that listen to the event
stream without re-writing the system
BENEFITS TRADEOFFS
• Async programming is a mind shift
• Complexity
– The flow of the system is shifted from a
central place to distributed services

12. 12
Four categories of tools you can use to build reactive apps
Reactive UI
Java script
front-end libraries:
Languages that support
reactive models natively:
Reactive Layers that run on
top of the JDK & implement
the Reactive-Streams spec:
Reactive Extensions
Java: RxJava
JavaScript: RxJS
C#: Rx.NET
C#(Unity): UniRx
Scala: RxScala
Clojure: RxClojure
C++: RxCpp
Lua: RxLua
Ruby: Rx.rb
Python: RxPY
Go: RxGo
Groovy: RxGroovy
JRuby: RxJRuby
Kotlin: RxKotlin
Swift: RxSwift
PHP: RxPHP
Elixir: reaxive
Dart: RxDart
ReactiveX for platforms
and frameworks
RxNetty RxAndroid RxCocoa

13. 13
Key technology components leveraged
• Exposes Akka Actor as a REST API, accelerating development
• Making async calls
• Using a reactive hybrid approach – Saga Actor handles fanning & merging
Java Future Interface • Enables REST API calls to be async and not wait for the response
Akka
Clustering
Sharding
• One node per AZ, 3 AZs total
• Used in combination with ELB (round robins original request to a node)
• Location transparency – actors can exist in any of the AZs
⎻ Leverage cluster sharding - actors are referenced by an ID instead of an ActorRef
• Enables actors to scale and get automatically restarted if they die
• Cluster sharding is using Akka Distributed Data
⎻ Shares data between Akka Cluster Nodes using a key-value store
Constraints
Throttling - Each Actor can only call an API at a certain rate
Batch to real-time - Batch file triggers the process, which has ~50k records in it daily

14. 14
Advanced features using Lagom
• Maximized development speed.
• Created couple of POC with Lagom within short time span
• REST end point developed using Lagom
• With Lagom developed CQRS & Event Sourcing pattern

15. 15
Context Diagram – Credit Card Fulfillment Process
PostGRE
Retry
Batch REST
Client
AppCapture API
Identity API
Encryption API
Research API
AppData API
Datastore API
Registry API
Income API
REST
API
AppCapture
Actor
Encryption
Actor
Research
Actor
AppData
Actor
Datastore
Actor
Registry
Actor
Income
Actor
Saga
Actor
Identity
Actor

16. 16
Capabilities of Credit Card Fulfillment
• Receive and validate request from batch
– Batch processes calls the Credit Card Fulfillment process with the parameters (key fields)
– Credit Card Fulfillment receives above parameters from batch and validate empty/null fields.
• Retrieve application details from other microservices
• Update SOR (System of Record)
• Update Registry
• Resiliency
– Retry upstream/downstream API call incase of failure.
– Max Retry frequency driven by configuration.
– Handle and track error scenarios in PostgreSQL Database
• Fulfill downstream systems TPS SLA using configuration driven throttling mechanism.
– Maintain incoming message in a Java Queue
– Control API call frequency based on configuration parameter.

17. 17
REST API
Context Diagram – Correspondence
App State
AppDetails API
Write Side Read Side
Reporting
Query API
External Letter
Dispatcher API
Actor System
Correspondence
Actor
AppDetails
Actor
SendLetter
Actor
REST API
Event Sourcing
LetterRequest
LetterResponse
AppId
App Details
Capabilities
1. Service Orchestration
2. Event History
3. Automatic API Call Retry
4. Real Time Status Check
5. Reporting
6. Restart Service Call From Last Failure
7. CQRS Pattern
8. Reporting Capability

18. 18
Lessons Learned
• Learning curve
• Unit testing and Performance Test metrics capturing was difficult
• Using throttling was limiting Akka performance
• Tight timeline caused few features to be dropped
– Akka persistence - Recovery of objects during failure
– Akka clustering - Providing resiliency and load balancing
18

19. 19
Questions