A banking system has some special constraints that are very hard to achieve together: High scalability, availability, auditing, and specially, dealing with parallelism respecting the invariances of a typical bank account. In this proposal, Kafka Platform is the backbone of this solution, but It’s not alone. To deal fully with the constraints, a combination with others technologies is necessary and Akka is the main chosen partner. The talk will contextualize the problem, how the Kafka Platform deal with it and the gaps filled by Akka.
1. A Core Banking System:
A Kafka Proposal
By: Diego Irismar da Costa
Brazil, São Paulo Apache Kafka® Meetup
August 22th, 2019
2. About Me
@diegoicosta
Current Manager of Infrastructure, sometimes also Chief Software Architect.
Worked mostly with java financial related backends since the early 2000s.
Always interested on Event Streaming, Event Oriented Systems, Distributed
Architectures, Cloud Computing, Observability, Data Bases
At Wirecard BR since 2010
• Previously Buscapé (2008-2010)
• Previously UOL (2000-2010)
3. Agenda
• Scope
• Kafka General Approach
• Kafka Applied to “A Core Bank System”
• Kafka Streams Topology and Strong Requirements
• Combining AKKA, Kafka and Kafka Streams
• Solution Scalability
• References
4. A Core Banking System ?
Let’s restrict the Scope
Apply Credit
KBank Account
Apply Debit
Balance
Credit/Debit/Read commands
KBank
5. An Old School Approach
The 60/70’s have gone
Get over it !
The Greater RDBMS
of the Universe
User Layer
Business Layer
8. Kafka Streams Possibilities
Stream Process Topologies
- Read Streams
- One Topic
- Join Topic Streams
- Process
- Map
- Sum, Count, Group, Merge
- Produce
- Branches
- Another Stream
- Change Log Stream (Ktable)
- Materialize Views
- Local State Store
?
9. Applying Kafka to a Core Bank System: How ?
?Apply Credit
KBank Account
Apply Debit
Balance
Credit/Debit/Read commands
10. The Core Topology
1. A account command stream is the topology input
2. The account have to accept the command. Does it have balance ?
3. The input stream is splitted: OK / NOK stream.
Both goes to different topics
?
4. A group by Account is applied to the OK stream
5. An aggregation summarizing the command value by account
6. The aggregation is materialized and persisted into a
changelog topic
11. The “Core” Requirements What is Needed to Build this ?
No Parallelism, No Multithread
ACID Transactions
Apply Credit
KBank Account
Apply Debit
Balance
Credit/Debit/Read commands
12. No Parallelism, No Multithread Partitions and Msg Keys
Topic
key:A1 | val:10 key:A2 | val:33 key:A3 | val:77
key:A1 | val:21
Conclusions
If AccountID is the key, each message from
that account will be processed by one
unique thread
13. ACID Transactions exactly-once “magic”
Kafka Streams Perspective
StreamsConfig.PROCESSING_GUARANTEE_CONFIG, "exactly_once";
It offers end-to-end exactly once guarantees for a stream processing
application that extends from the data read from Kafka, any state
materialized to Kafka by the Streams app, to the final output
written back to Kafka.
14. The Core Topology Implementation
1. A account command stream is the topology input
builder.stream("kbank.account”,
Consumed.with(Serdes.String(), new JsonSerde()))
consume
15. The Core Topology Implementation
2. The account have to accept the command. Does it have balance ?
3. The input stream is splitted: OK / NOK stream.
Both goes to different topics
?
accountStreams[] = builder.stream(
“kbank.account”,
Consumed.with(Serdes.String(), new JsonSerde()))
.branch(
(account, command) -> hasBalance(account, command.getValue()),
(account, command) -> noBalance(account, command.getValue())
);
Split based on some defined predicates
accountStreams[0] // OK can be applied
accountStreams[1].through(“kbank.account.refused”) // NOK must be refused
16. The Core Topology Implementation
accountStreams[0].through(“kbank.account.accepted”)
.mapValues((accountCommand) -> {
return accountCommand.value;
}).groupByKey();
4. A group by Account is applied to the OK stream
17. The Core Topology Implementation
5. An aggregation summarizing the command value by account
6. The aggregation is materialized and persisted into a
changelog topic
.aggregate(
() -> 0L,
(accountId, balance, oldBalance) -> oldBalance + balance,
Materialized.<String, Long, KeyValueStore<Bytes, byte[]>>
as("kbank.account.store")
.withKeySerde(Serdes.String())
.withValueSerde(Serdes.Long())
);
accountStore = streams.store("kbank.account.store".keyValueStore());
22. Solution Scalability: Kafka
Kafka, by design …
- A cluster is able to handle trillions of events per day
- Absurd throughput rate
- Battle-proven
23. Solution Scalability: Kafka Streams
Highly coupled with Kafka
- Also, must count on distribution to handle high volumes
- A Kafka Stream app must also be “partitioned”
- The Materialized views could be by partition
24. Solution Scalability: Kafka Streams
Command
Account
Stream
Server #3
Command
Account
Stream
Server #1
Command
Account
Stream
Server #2
HTTP
HTTP