A banking system has some special constraints that are very hard to achieve together: High scalability, availability, auditing, and specially, dealing with parallelism respecting the invariances of a typical bank account. In this proposal, Kafka Platform is the backbone of this solution, but It’s not alone. To deal fully with the constraints, a combination with others technologies is necessary and Akka is the main chosen partner. The talk will contextualize the problem, how the Kafka Platform deal with it and the gaps filled by Akka.

1. A Core Banking System:
A Kafka Proposal
By: Diego Irismar da Costa
Brazil, São Paulo Apache Kafka® Meetup
August 22th, 2019

2. About Me
@diegoicosta
Current Manager of Infrastructure, sometimes also Chief Software Architect.
Worked mostly with java financial related backends since the early 2000s.
Always interested on Event Streaming, Event Oriented Systems, Distributed
Architectures, Cloud Computing, Observability, Data Bases
At Wirecard BR since 2010
• Previously Buscapé (2008-2010)
• Previously UOL (2000-2010)

3. Agenda
• Scope
• Kafka General Approach
• Kafka Applied to “A Core Bank System”
• Kafka Streams Topology and Strong Requirements
• Combining AKKA, Kafka and Kafka Streams
• Solution Scalability
• References

4. A Core Banking System ?
Let’s restrict the Scope
Apply Credit
KBank Account
Apply Debit
Balance
Credit/Debit/Read commands
KBank

5. An Old School Approach
The 60/70’s have gone
Get over it !
The Greater RDBMS
of the Universe
User Layer
Business Layer

6. Kafka in General: Basic Approach
Message Broker
Producers
Consumers

7. Kafka Approach: Beyond Basics
Message Broker
Producers
Consumers
Kafka Streams

8. Kafka Streams Possibilities
Stream Process Topologies
- Read Streams
- One Topic
- Join Topic Streams
- Process
- Map
- Sum, Count, Group, Merge
- Produce
- Branches
- Another Stream
- Change Log Stream (Ktable)
- Materialize Views
- Local State Store
?

9. Applying Kafka to a Core Bank System: How ?
?Apply Credit
KBank Account
Apply Debit
Balance
Credit/Debit/Read commands

10. The Core Topology
1. A account command stream is the topology input
2. The account have to accept the command. Does it have balance ?
3. The input stream is splitted: OK / NOK stream.
Both goes to different topics
?
4. A group by Account is applied to the OK stream
5. An aggregation summarizing the command value by account
6. The aggregation is materialized and persisted into a
changelog topic

11. The “Core” Requirements What is Needed to Build this ?
No Parallelism, No Multithread
ACID Transactions
Apply Credit
KBank Account
Apply Debit
Balance
Credit/Debit/Read commands

12. No Parallelism, No Multithread Partitions and Msg Keys
Topic
key:A1 | val:10 key:A2 | val:33 key:A3 | val:77
key:A1 | val:21
Conclusions
If AccountID is the key, each message from
that account will be processed by one
unique thread

13. ACID Transactions exactly-once “magic”
Kafka Streams Perspective
StreamsConfig.PROCESSING_GUARANTEE_CONFIG, "exactly_once";
It offers end-to-end exactly once guarantees for a stream processing
application that extends from the data read from Kafka, any state
materialized to Kafka by the Streams app, to the final output
written back to Kafka.

14. The Core Topology Implementation
1. A account command stream is the topology input
builder.stream("kbank.account”,
Consumed.with(Serdes.String(), new JsonSerde()))
consume

15. The Core Topology Implementation
2. The account have to accept the command. Does it have balance ?
3. The input stream is splitted: OK / NOK stream.
Both goes to different topics
?
accountStreams[] = builder.stream(
“kbank.account”,
Consumed.with(Serdes.String(), new JsonSerde()))
.branch(
(account, command) -> hasBalance(account, command.getValue()),
(account, command) -> noBalance(account, command.getValue())
);
Split based on some defined predicates
accountStreams[0] // OK can be applied
accountStreams[1].through(“kbank.account.refused”) // NOK must be refused

16. The Core Topology Implementation
accountStreams[0].through(“kbank.account.accepted”)
.mapValues((accountCommand) -> {
return accountCommand.value;
}).groupByKey();
4. A group by Account is applied to the OK stream

17. The Core Topology Implementation
5. An aggregation summarizing the command value by account
6. The aggregation is materialized and persisted into a
changelog topic
.aggregate(
() -> 0L,
(accountId, balance, oldBalance) -> oldBalance + balance,
Materialized.<String, Long, KeyValueStore<Bytes, byte[]>>
as("kbank.account.store")
.withKeySerde(Serdes.String())
.withValueSerde(Serdes.Long())
);
accountStore = streams.store("kbank.account.store".keyValueStore());

18. Combining AKKA, Kafka and Kafka Streams
http
COMMAND HANDLER “Sync” COMMAND STREAM PROCESSING “Async”
AKKA
Actors
Kafka
Streams

19. The Kafka Streams role
http
Id:12er3-34b
Id:12er3-34b
JsonResponse
Response
Streams
COMMAND HANDLER “Sync” COMMAND STREAM PROCESSING - Async
Command
Account
Stream

20. The AKKA role
http
Id:12er3-34b
Id:12er3-34b
JsonResponse
Command
Account
Stream
Response
Streams
COMMAND HANDLER “Sync” STREAM PROCESSING - Async
Command
Handler
actorRef
actorRef
Response
Router
Id:12er3-34b actorRef

21. Command
Account
Stream
HTTP
Kafka-Streams
Akka-HTTP
Kafka as …
-Stream Platform
-Event-Store
Changelog
KTable
State store
Materialized
view
Non-Blocking
Exactly-Once
ACID guarantees
Response
Streams
OK / NOK Streams
Actor
per
request

22. Solution Scalability: Kafka
Kafka, by design …
- A cluster is able to handle trillions of events per day
- Absurd throughput rate
- Battle-proven

23. Solution Scalability: Kafka Streams
Highly coupled with Kafka
- Also, must count on distribution to handle high volumes
- A Kafka Stream app must also be “partitioned”
- The Materialized views could be by partition

24. Solution Scalability: Kafka Streams
Command
Account
Stream
Server #3
Command
Account
Stream
Server #1
Command
Account
Stream
Server #2
HTTP
HTTP

25. Final Thoughts
A Bunch of Concepts Together

26. Where these ideas came from ?
Pat Helland
Ben Stopford
Martin Kleppmann
Jim Gray
Mato

27. Some References
https://www.confluent.io/blog/
https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/
https://www.confluent.io/blog/transactions-apache-kafka/
https://www.confluent.io/blog/transactional-systems-with-apache-kafka
https://www.confluent.io/blog/data-dichotomy-rethinking-the-way-we-treat-data-and-services/ (part 1 to 7)
https://www.confluent.io/designing-event-driven-systems
https://docs.confluent.io/current/streams/developer-guide/index.html
https://doc.akka.io/docs/akka/current/actors.htm
http://adrianmarriott.net/logosroot/papers/LifeBeyondTxns.pdf
https://learning.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/

28. The Code
github.com/diegoicosta/kbank

29. The end