This document discusses how a startup prime broker uses Apache Kafka to manage trading workflows and data. It describes how the prime broker uses Kafka for change data capture to stream trading data from databases to downstream analytics systems. It also discusses how the prime broker models trading entities like orders and trades using event sourcing patterns in Kafka and manages the lifecycles of these entities as events. Finally, it explains how the prime broker uses Kafka to unify different communication protocols within its architecture.

1. COMPANY CONFIDENTIAL. DO NOT REDISTRIBUTE
How a startup Prime Broker uses Kafka

2. About Me
2
Sachin Kumar
skumar@clearstreet.io
- Co-founder, CTO @ Clear Street
- ex-HFT entrepreneur
https://github.com/clear-street

3. Agenda
3
1. What is a Prime Broker?
2. Change data capture
3. Trade management
4. Order management
5. Interoperability

4. What is a Prime Broker, anyway?
4
Manage all aspects of trading, both pre-trade and post-trade
Execution
Venue
buy 100 AAPL
@ $103.48
Prime
Broker
Central
Clearinghouse
order
trade trade

5. Change data capture
5
Biz Logic
API CDC
UI
Kafka
ETL
Microservice
Postgres Redis
ElasticSearch
Snowflake
Primary Store Cache
Secondary Stores
App
Job 0 Job 1
BI
CRUD

6. Change data capture (cont)
6
Transaction Outbox
Avro schema ID
Avro encoded bytes

7. Trade lifecycle [simplified]
7
new
trade
open
trade
settled
trade
canceled
trade
trade_date > settle_date
trade_date <= settle_date
trade canceled
trade_date <= settle_date
trade canceled

8. On trade transition, emit event
8
new
trade
open
trade
{
“event”: "opened”,
“trade”: {
“account”: “foo”,
“price”: “1.23”,
“quantity”: “100”,
“symbol”: “AAPL”
...
},
“position”: “100”,
“balance”: “-123”
...
}

9. Trade processing v1
trade
processor
A
P
I
C
D
C
POST /trade
{
“event”: "opened”,
“trade”: {
“account”: “foo”,
“price”: “1.23”,
“quantity”: “100”,
“symbol”: “AAPL”
...
},
“position”: “100”,
“balance”: “-123”
...
}
St#e%&
t#a)s%+,-on0
postgres

10. Trade processing v2
10

11. What about orders?
11
buy 1000 AAPL
@ 109.23
order
ack
100 shares
executed
150 shares
executed
cancel
order
order
cancelled
t
client
exchange

12. Order event sourcing
12
order 3
sent
order 4
sent
order 3
acked
order 3
executed
snapshot
offset 0
order 3
canceled
Client scans backwards to
first available snapshot
offset N
order 3, order 4 are open
pointer to persistent
storage

13. Interoperability
13
The Financial Information eXchange (FIX)
protocol is an electronic communications
protocol initiated in 1992 for international
real-time exchange of information related
to securities transactions and markets. [1]
[1] https://bit.ly/3rRN3uB

14. Interoperability (cont)
14
session
manager
FIX Client
session
parser
Service Mesh
topic:
sender_id.input
topic:
sender_id.output
blog: https://bit.ly/3s8bXq1

15. We ❤ kafka
15
The kafka patterns we’ve used at Clear Street have unlocked massive value
1. CDC allows separation of online data vs. offline data -- materialized view in
secondary stores allows data analytics that don’t touch critical paths
2. Event sourcing is a fundamental pattern to model core constructs, like orders and
trades.
3. Kafka features like atomic publish allow for novel architectures that otherwise would
be difficult
4. Kafka allows us to normalize disparate streaming protocols within our stack

16. Thank you