In this presentation we look at the use of the Hortonworks Data Flow (HDF) platform application in the finance sector operations. We define the process of deployment and use of the distribution as part of the operations and how the different components are integrated to support real-time payments and banking functionality. Documenting the challenges faced to allow a high performance system that promotes data integrity and real-time visualisation using the Hortonworks Data Platform (HDP). Focusing specially on the use of Apache Zeppelin workbooks across the business as they main information management tool. Our specific approach focus on creating a flexible system that allows fast prototyping and integrated visualisation, monitoring and audit. Demonstrating the use of the HDF distribution on the creation of business domain abstractions to in a real life application of a Domain Driven Design. We implement a cross platform use of Avro, as future prof model and language that is understood by the business and IT areas. Due to the scalability of the platform we can execute payments operations at high rate even across countries at the same time reusing the same architecture to monitor the business operations.
Speaker
Luis Caldeira, Chief Architect, Orwell Group
Gian Marco Cabiato, Head of Engineering, Orwell Group
3. INTRODUCTION
BANKING SOLUTIONS AS A SERVICE
01
02
03
04
05
Regulated Cash Management &
Borderless Banking Model
Breakthrough Payments Technology
Open Banking and PSD2 compliant
White Label for Banks
Suite of Treasury, Payments and Loyalty
solutions for Corporates
4. INTRODUCTION
BANKING SOLUTIONS AS A SERVICE
Technology In Financial Services
Today's challenges and future thoughts
Complex estates that have been running since the late 60’s and
70’s.
How to manage change in these large estates to migrate to
newer technology.
Complexity is not just hardware and software but offered products
and lifespan of those products, some entities that number can be
more than 40000 different products.
Today’s institutions offer greater ranges
of products, some moved to other
areas like Insurance.
IT is the main enabler of the banking
offers and has large delivery cycles and
large costs as part of the processes
that must be followed.
Business have lost of knowledge even
from their internal operations by
outsourcing or large use of consultants.
5. INTRODUCTION
BANKING SOLUTIONS AS A SERVICE
How we look at a financial institution
Current accounts Cards
Bank transfer Standing orders
Direct Debits Overdrafts
Loans Mortgages
Savings Term Deposits
Investments Guaranties
INDIVIDUALS BUSINESS MERCHANTSCORPORATES
KYC
Overdrafts
Loans Mortgages
Savings
Term
Deposits
Investments Guaranties
Current accounts Cards
Bank transfer Standing orders
Direct Debits
INDIVIDUALS BUSINESS MERCHANTSCORPORATES
KYC
DISTRIBUTION
PRODUCTS
CASH
MANAGEMENT
DEPOSIT
TAKING
6. INTRODUCTION
BANKING SOLUTIONS AS A SERVICE
How can we address the challenges and
provide bleeding edge technology at
best possible cost ?
7. BANKING SOLUTIONS AS A SERVICE
What principles should we make use of ?
TECHNOLOGY
Micro-services.
Performance
Features
Data Quality
Operational
Analytical
API
Gateway
Stack Model.
Information is everything, like water,
flows and you want to store as much
as possible.
Models evolve don't think tables think
characteristics and relationships.
There is no one final answer but
some concepts are very constant so
we can employ some common sense.
8. BANKING SOLUTIONS AS A SERVICE
After some time and some exhausted brain cells….
INTRODUCTION The idea was to have a very high
performance, cloud based system that
allows us to create a core banking system
and financial supply chain in very short
period.
Use real-time approach to banking in a
distributed context that provides high
consistency, reliability at low operational
cost.
…. We mean a full bank, not just payments
or account/cash management.
9. BANKING SOLUTIONS AS A SERVICE
We have a list of technologies we can and a solution
approach but how do we design it and get it to market ?
TECHNOLOGY
We look at banking in a different way not
as a number on a database but as a set of
changes on accounts.
A balance is just the state at a certain
point in time that can vary faster or slower
depending on operations.
Not important for a process run faster but
how parallel can we run operations there
is where cost saving lies.
10. BANKING SOLUTIONS AS A SERVICE
Should we start from scratch or are there any other
options ?
TECHNOLOGY In low volumes the existing pattern’s are a
solution but as we increase the volume high
parallelism will impact the system.
NoSQL databases can be a solution to
increased performance but we have to look at
playing with availability and consistency.
Is trying to get the best of both worlds. Doesn't
violate CAP but makes things optimal in some
way.
We looked at the different areas to provide the most complete
support for all solutions element.
Operational Analytical
Hortonworks HDF is part of the core of the banking system. This
means we have a distributed, stream based banking system.
11. BANKING SOLUTIONS AS A SERVICE
As part of the design of the operational system we
looked at:
ARCHITECTURE
Streaming elements
that control the data
flows and DSL
control.
Security and One central control
unit that manages all baking
components.
Used to prototype
small flows before
building topologies
At Orwell we not only developed the best technology
models but had a very strong focus on information. The
platform model created from scratch is based upon
complex contractual law and financial concepts
We make use of the Party-Contract-Product Held
principles pioneered in other entities. This allows the
definitions of complex products with customized
characteristics that can be built and test in a matter of
days or weeks.
The model is design to allows complex relationships to be
defined between participants for example a company
having users that manage their own self-service banking
products
12. BANKING SOLUTIONS AS A SERVICE
The Principle of the design
ARCHITECTURE
Kappa Architecture is a software architecture
pattern. Rather than using a relational DB like
SQL or a key-value store like Cassandra, the
canonical data store in a Kappa Architecture
system is an Apache Kafka append-only
immutable log.
From the log, data is streamed through a
computational system and fed into auxiliary
stores for serving.
13. BANKING SOLUTIONS AS A SERVICE
How operations are tracked ?
ARCHITECTURE
The model supports:
Accounts that require a state that moves with time
and have limits (product limits, fees or facilities)
applied to it.
Accounts that record transaction, a direct log of
operations and not states
DSL Mapper is a domain boundary system that
generates a command with the necessary
datasets that allow a seamless execution of the
operation at next stages. These are used to
implement, for example, payment engines.
Monitoring is Built using the Same technologies,
Apache Storm, KAFKA Streams and Apache
Spark
The system centralises the Keys generation.
This means every operation can be tracked in real type across the
distributed system. A good example is the monitoring system.
14. BANKING SOLUTIONS AS A SERVICE
The Accounting system, a distributed, high performance
stream system
ARCHITECTURE • The Follower processors are consumers of
the processed log from the leader
Processor.
• Water marks from the original log itself,
each will have a full copy of the
information available from the lead. This
mean async replication that allows replay
without duplication and same model as the
processor start fail-over recovery model.
On a two node accounting system we can process
Process more than 12000 payments per second.
More than 3000 transactions per second on a single account.
15. BANKING SOLUTIONS AS A SERVICE
How to approach support and Platform management ?
MANAGEMENT
Not Only we make use of the HDF facilities we are
integrating all our services into Ambari to have one single
control console for all the banking operations.
Monitoring & Audit are business requirements
even when the business doesn't know. First
item a regulator looks at its at the audibility of
processes.
We want to know what , when, how and
actions we have taken. We don't want the
developers to spend lots of time thinking
about it.
Small support teams with all necessary
information operate more efficiently and a
lower operational cost.
16. BANKING SOLUTIONS AS A SERVICE
UK Faster Payments Model
The Faster payments system is fully integrated
with the platform architecture. There are 3
stages
• The Connectors that interact with the
gateways and external system. We have or
are building connectors for all payments
schemes.
• The DSL’s (domain subject language) that
understand the payments scheme as well
as banking. They transform the request into
a banking core language.
• The Processors, programmable engines
that can apply accounting rules and
manage accounts.
The FPS flow has an automated monitoring
monitoring system reusing the technology of
the core to monitor every step of execution for
full E2E tracking and audit.
ARCHITECTURE
17. Local Registry
Local Registry
Local Registry
System
System
System
Broadcast – 128bit Account
Number Hash
Broadcast – 128bit Account
Number Hash
DHT Based
Network
Make Payment
Make Payment
Communication between instances
The system is designed to have multiple instances across multiple public
clouds, this means each system must be instance aware.
UK
Germany
US
Instance ID
Instance ID
Instance ID
Account Key
(64Bit)
Instance
ID
Account Naming Structure to allow Routing
Automated Discovery of
Network participants
UUIDs:
That are
customized
identifiers
Like email
BANKING SOLUTIONS AS A SERVICE
TECHNOLOGY
18. DEPLOYMENT
• CICD is not just the latest IT buzz word. It effectively allows faster
delivery of solutions as well as faster validation of those solutions.
• Use of Apache Ambari for automated Application deployment and
control.
BANKING SOLUTIONS AS A SERVICE
Why use CICD (Continuous Integration Continuous Delivery) and
infrastructure as code?
Another important element is the
automation the process brings, e.g.
Automated Code validation.
Automated Builds
Proper version control
Multi-version support, multi-client
Infrastructure as code allow us to create
environments that simulate production
scenarios.
Better rate of development
Better quality of overall product.
Less time loss of manually creating and
configuring environments.
Deploy in every scenario, Private, Public
Clouds or on premise
19. BANKING SOLUTIONS AS A SERVICE
After some time and some exhausted brain cells….
DEPLOYMENT
20. BANKING SOLUTIONS AS A SERVICE
TECHNOLOGY
How can a banking platform improve customer experience ?
Its not just about gathering information but about creating a
unique experience, reducing the learning curve of learning to
use applications, reduce fraud and better target product to the
customer.
21. BANKING SOLUTIONS AS A SERVICE
TECHNOLOGY
How can a banking platform improve customer
experience ?
One of the design decisions of the platform is the
intention to allow a fast paced evolution of features
and facilities. From this principle we created two
rules:
1. Information should be used to learn and
improved how customers interact with their
applications.
1. The best developer experience is connected to
the best customer experience
The system is designed to record all internal
activities to learn from its on execution of processes
so we have taken the same approach for
customers.
Policy
Improve
Interactions
1. Create an enum-based list of events you want to track
enum LoginEvent {
case loginButtonTapped
case loginSuccess(username: String)
// More events...
}
2. Configure the instance of AnalyticsManager
extension AnalyticsManager {
static var shared: AnalyticsManager {
let cocoaMQTT = CocoaMQTT.shared
let client = MQTTClient(client: cocoaMQTT)
cocoaMQTT.delegate = client
client.connect()
let analytics = AnalyticsManager(engine:
MQTTAnalyticsEngine(client: client))
return analytics
}
}
22. BANKING SOLUTIONS AS A SERVICE
TECHNOLOGY
How can a banking platform improve customer experience ?
So how do I scale the system ?
Same way to scale HDF :)
Just add processors and request the
system to re-balance the account
ranges.
KAFKA can create topics on demand.
The processors will talk to each other
to agree who owns the account on re-
balance.
23. BANKING SOLUTIONS AS A SERVICE
TECHNOLOGY
How can a banking platform improve customer experience ?
24. BANKING SOLUTIONS AS A SERVICE
BUSINESS
3rd Party Integrations
• Addition of third party services is simple. Orwell can
integrate with any financial or other service with simplicity.
• Orwell is already connected to:
• CBILL (Italian bill payment service)
• Currency Cloud (FX and international FX payments)
• TransferTo (international payments and mobile phone
pre-payment)
• MAV/RAV (Italian taxes)
• Bolatino P (Public admin); Bolatino B (Billers)
• We are connecting to: ApplePay, Kantox
25. BANKING SOLUTIONS AS A SERVICE
BUSINESS
Orwell delivers instant payments (even cross border
and cross currency)
Instant
payments
even cross
border and
cross currency
Elimination of
card
payments
costs
Enables
innovation in
operations and
customer
relationships
Reduced needs of
working capital
by eliminating
“money in transit”
More capital
available for
investment of
distributions
Cheaper treasury
operations and
elimination of
interchange
Better control of
supply and
distribution
chains
RTP & VAS
enhances client
relations and
improves
transparency
Subscription
based revenue
model
26. BANKING SOLUTIONS AS A SERVICE
CONCLUSION
The use of Hortonworks Data Flow allows us to create a
banking system in less than 9 month including certifications.
Satisfy the support levels required by regulators.
No need for batch operations, the system can process
efficiently so that batching is not necessary.
No database limitations.
27. BANKING SOLUTIONS AS A SERVICE
CONCLUSION
Better customer experience. Specially for corporate banking.
Fast Prototyping with SAM
Allow processing time for other operations like sanctions ans
reduce risk.
Efficient capacity management.
Real-time MI, analytics, use of ML algorithms to manage flows,
etc