Presented at Kafka Summit SF 2017 by Charles Reese and Matthias Margush

1. Building Stateful
Financial Applications
with Kafka Streams
Charles Reese / Matthias Margush

2. Funding Circle is a lending platform for small businesses
InvestorsBorrowers
London
Amsterdam
San Francisco
Berlin

3. What should we do?

4. DELETE FROM ... WHERE ...

5. Kafka as a System of Record

6. Kafka Streams

7. Deposit Received

8. Deposit Received
Repayment Paid

9. Deposit Received
Repayment Paid
Repayment Distributed

10. Deposit Received
Repayment Paid
Repayment Distributed Holdings Updated

11. Money
Movements
Accounting
Loan
Servicing
Exchange
Edge Service
(HTTP)

12. Clojure

13. (defn number-of-moves
[number-of-disks]
(- (reduce * (repeat number-of-disks 2)) 1))
(defn topology
[builder]
(let [stream (j/kstream builder input-topic)]
(-> (j/map-values stream number-of-moves)
(j/to! output-topic))))
(defn -main
[& _]
(let [streams (-> (topology (j/topology-builder))
(j/kafka-streams config))]
(j/start! streams)))

14. (defn number-of-moves
[number-of-disks]
(- (reduce * (repeat number-of-disks 2)) 1))
(defn topology
[builder]
(let [stream (j/kstream builder input-topic)]
(-> (j/map-values stream number-of-moves)
(j/to! output-topic))))
(defn -main
[& _]
(let [streams (-> (topology (j/topology-builder))
(j/kafka-streams config))]
(j/start! streams)))

15. (defn number-of-moves
[number-of-disks]
(- (reduce * (repeat number-of-disks 2)) 1))
(defn topology
[builder]
(let [stream (j/kstream builder input-topic)]
(-> (j/map-values stream number-of-moves)
(j/to! output-topic))))
(defn -main
[& _]
(let [streams (-> (topology (j/topology-builder))
(j/kafka-streams config))]
(j/start! streams)))

16. (defn number-of-moves
[number-of-disks]
(- (reduce * (repeat number-of-disks 2)) 1))
(defn topology
[builder]
(let [stream (j/kstream builder input-topic)]
(-> (j/map-values stream number-of-moves)
(j/to! output-topic))))
(defn -main
[& _]
(let [streams (-> (topology (j/topology-builder))
(j/kafka-streams config))]
(j/start! streams)))

17. Money
Movements
Accounting
Loan
Servicing
Exchange
Edge Service
(HTTP)

18. CRUD
Front End
Order
Service
Place Order

19. CRUD
Front End
Order
Service
Cash
Balance
Place Order
Get Balance

20. CRUD
Front End
Order
Service
Cash
Balance
Loan Part
Place Order
[funds available?] Transfer Ownership
Balance
Get Balance

21. CRUD
Front End
Order
Service
Cash
Balance
Loan Part
Place Order
[funds available?] Transfer Ownership
Balance
Get Balance
Adjust Balance

22. CQRS

23. Command
Query
Responsibility
Segregation

24. 24
Command
Interface
Services
place order
topic
Event Log Business Logic

25. 25
Command
Interface
Services
place order
topic
orders
topic
Event Log Business Logic

26. 26
Command
Interface
Query
Interface
Services
place order
topic
orders
topic
Event Log Business Logic
Database
Materialized Views

27. How can we know a record is new?

28. id: 48
debit: Repayments
credit: Alice
amount: 100
. . .

29. id: 1516
debit: Repayments
credit: Jay Krebs
amount: 500
id: 1516
debit: Repayments
credit: Jay Krebs
amount: 500
id: 48
debit: Repayments
credit: Alice
amount: 100
. . .

30. Exactly Once Processing

31. CA BB . . .

32. CA BB . . .
AA CB CB . . .

33. CA BB . . .
AA CB BD C . . .

34. id: 1516
debit: Repayments
credit: Jay Krebs
amount: 500
id: 1516
debit: Repayments
credit: Jay Krebs
amount: 500
id: 48
debit: Repayments
credit: Alice
amount: 100
. . .

35. id: 8943B979-446...
debit: Repayments
credit: Jay Krebs
amount: 500
id: 8943B979-446...
debit: Repayments
credit: Jay Krebs
amount: 500
id: CFC62DA4-FCC...
debit: Repayments
credit: Alice
amount: 100
. . .

36. Use type 5 UUIDs and keep track of
every* record seen

37. Use type 5 UUIDs and keep track of
every* record seen
* Well, maybe not every...

38. Look up request
by ID
Is it a
dupe?

39. Look up request
by ID
Is it a
dupe?
Update balances
Save request and
starting balances
Produce result
NO

40. Look up request
by ID
Is it a
dupe?
Update balances
Save request and
starting balances
Produce result
Skip side effects
NO
YES

41. How can we migrate existing services?

43. Migrating existing services
GET /balance
HTTP
Client Old and Busted
HTTP
Client
POST /entry
New Hotness
Balances
HTTP
Service

44. Migrating existing services
GET /balance
Kafka
Connect
HTTP
Client Old and Busted
HTTP
Client
POST /entry
New Hotness
Requests
Balances
HTTP
Service

45. Migrating existing services
GET /balance
Kafka
Connect
HTTP
Client Old and Busted
HTTP
Client
POST /entry
New Hotness
Balances'
Requests
Balances
HTTP
Service

46. Migrating existing services
GET /balance
Kafka
Connect
HTTP
Client Old and Busted
HTTP
Client
POST /entry
New Hotness
Balances'
Requests
Balances
HTTP
Service

47. Use a bridge service...
HTTP Client
(writes)
HTTP Client
(writes)
HTTP Client
(writes)
HTTP
Service
DB

48. Use a bridge service...
HTTP Client
(writes)
HTTP Client
(writes)
HTTP Client
(writes)
HTTP
Service
DB
BridgeService

49. Use a bridge service...
HTTP Client
(writes)
HTTP Client
(writes)
HTTP Client
(writes)
HTTP
Service
request
topic
DB
result
topic
BridgeService

50. Use a bridge service...
HTTP Client
(writes)
HTTP Client
(writes)
HTTP Client
(writes)
HTTP
Service
request
topic
DB
result
topic
BridgeService

51. ...and materialized views
HTTP Client
(reads)
DB

52. ...and materialized views
HTTP Client
(reads)
DB
Kafka
Connect
view
topic

53. 53
With great power...