1. 1
I
HOW
EXCHANGE
SCALED
TO
160K RPS
WITH
KAFKAManjunath Shivakumar
Teodora Marcu

2. WHO WE ARE
Betfair founded in 2000
2
Merger of Paddy Power and Betfair in 2016
Paddy Power – Leading betting operator in Ireland and UK
Betfair – Largest online betting exchange in the world
Flutter Inc.

3. Betfair 3
BETTING
EXCHANGE
101

4. Betfair 4
PEER-PEER
BETTING
PLATFORM
• Odds set by user
• Cuts out middle man
• Upto 20% Better Odds than
market
• Recreational and
Automated trading
• Covering Sports, Politics,
Special Events
Odds –probability of an outcome
Fractional/Decimal Odds - 7/2 = 4.5
Back – Bet for an outcome
Lay – Bet against an outcome

5. Betfair 5
TRADING CYCLE
• Users interact via web, mobile and
API interfaces
• High performance API to support
order management, and discovery
of current state of exchange
• Low Latency High Frequency
Trader Bots generate 75% of traffic
and revenue
• Peak loads of over 200k RPS
Discover
Markets
Check
Market
State
Calculate
Position
Place /
Update /
Cancel
Orders
Check
Position

6. Betfair 6
LOAD PROFILE
Discover
Markets
Check
Market
State
Calculate
Position
Place /
Update /
Cancel
Orders
Check
Position
CATALOG
API
15k
MARKET
READ API
95k
TRANSACTIONAL
API
3k
BET READ
API
46k
60%
29%
2%
9%
11%

7. Betfair 7
Monolithic Web and API nodes
Unreliable UDP based event stream
Heavy reliance on DB
Legacy code impossible to change
Developed a new lightweight API stack
Cliff edge at 16k rps, but we needed 80k
LEGACY
ARCHITECTURE
DB
P
S S
Migration to new stack had hard deadline

8. Betfair 8
THANOS IS COMING

9. Betfair 9
TARGET
Scalable
Add consumers and scale them
horizontally with minimal impact on the
underlying systems
Complete Stream
Consumers need to catch up quickly
under failure conditions from the
stream

10. Betfair 10
TARGET
Fast
Minimised end to end latency
Reliable
Systems should not go down or if they
do they need to recover immediately

11. Betfair 11
TARGET
Efficient
Publish only updates to stream, not
fully images
Encode data using efficient
serialization/deserialization techniques

12. Betfair
Periodically published full image
of Exchange market state
SNAPSHOTS + DELTAS
12
KEY CONCEPTS – DELTA + SNAPSHOT ( INBAND )
D D D DD
S
D D
S
D D D

13. Betfair 13
DELTA
SNAPSHOTS
EXAMPLE
S S
D D D D D D D D

14. 501
730
502
730
503
730
504
730
505
730
506
730
507
730
508
730
509
730
510
730
511
730
512
730
DELTA STREAM
101
101
102
101
103
101
104
101
105
101
106
101
107
107
108
107
109
107
110
107
111
107
112
107
502 502 502 502 502 511 511 511 511 511 511502
730 730 730 730 730 730 730 730 730 730 730 730
SNAPSHOT STREAM
OrderChanges – MarketAccount Aggregation
EndOfSnapshot
101 Kafka OffsetOrderSnapshot – Market Aggregation
Snapshot Stream Dawn OrderChanges – No Aggregation101
502
730
Delta Stream Dawn
Stream Version
KEY CONCEPTS – DELTA + SNAPSHOT ( OUT OF BAND )

15. Betfair
Using protocol buffers to to
serialize structured data.
4-5 times smaller
Reusable consumer libraries
encoding stream mechanics,
bootstrapping, heart beats,
conditional decoding
Fast serialization /
deserialization, rapid and
reliable delivery, efficient
bootstrapping, fast
failovers
PROTOCOL CONSUMERS SUPER FAST
15
KEY CONCEPTS

16. Betfair
UNDERPINNED
BY
RELIABLE,
RESILIENT,
BLAZING FAST
KAFKA
DB
S S S
P
S
K

17. Betfair
SETUP
● 24 CPU, 32 GB RAM, 6 x 200 GB
● 3 Hosts
● 3 Replicas
● 3 Producers
● 5 Topics
● 100+ Consumers

18. Betfair 18
KAFKA
ASSEMBLED

19. Betfair 19
KEY METRICS
● 200k Requests / sec
● Bytes In / sec
● 300 MBytes Out / sec
● 10k Messages In / sec
● 8 ms 99p response time
● <100 ms end to end lag
● 75 TB over 4.5 years

20. Betfair 20
MIGRATION
STRATEGY

21. Betfair 21
MIGRATION
STRATEGY
● Legacy and new stack in parallel
● Controlled throttle of traffic
● Continuous A/B evaluation
● Minimal customer impact
● Minimal DB impact
● 32 x 5k = 160k RPS
S
P
K
DB
S S
P
SS

22. Betfair 22
KEY BENEFITS
Seamless
Migration
Migrating from old
stack to new stack
was relatively smooth
Rewind Replay
State of system can
be replayed for
troubleshooting and
analysis
Agility
Its very easy to build
new features or add
new data points into
core streams
Efficient Scaling
It is exponentially
easier today to scale
our capacity

23. Betfair 23
THANOS
SUBDUED …

24. Betfair 24
DESTINY STILL ARRIVESA YEAR LATER …

25. Betfair 25
MORE PROBLEMS
● Simplified Order Flow
● 2500 rps
● Input Queue Limits
DB
K
DB
ORDER
PROCESSOR
ORDER
INPUT
HANDLER

26. Betfair 26
REDESIGNED ORDER FLOW
● Rerouted Order Flow
● 25000+ rps
● Limit yet to be hit
● Simpler recovery
ORDER
PROCESSOR
K
K
DB
ORDER
INPUT
HANDLER

27. Betfair 27
ITS REDACTED !
ENDGAME..?

28. Betfair 28
THANOS IS
ALWAYS COMING …

29. Betfair
FRAUD DETECTION
OPERATIONAL
STREAMS
PERSONALISATION HISTORICAL DATA
29
FUTURE
CHALLENGES
Detect service usage
abuse and sanction
dynamically.
Prevent revenue loss.
Operational metrics
streams.
How are customers using
Exchange?
Merge to generate a
personalised data
stream.
We think you might like
this.
Record business streams
into market files.
Play these to train your
bots.

30. Betfair
MULȚUMESC
30
THANK YOU
‫ﺑ‬‫ﮩ‬‫ﺖ‬‫ﺑ‬‫ﮩ‬‫ﺖ‬‫ﺷ‬‫ﮑ‬‫ﺮ‬‫ﯾ‬‫ہ‬
Ευχαριστώ πολύ
Благодаря ти
Merci
Teşekkür ederim
ਤuਹਾਡਾ ਧ'ਨਵਾਦ
ధన#$ా&'ల)
Manjunath.Shivakumar@paddypowerbetfair.com
manjusmail@gmail.com
https://www.linkedin.com/in/manjunathshivakumar

31. Betfair
Appendix

32. Betfair 32
SOLUTION ARCHITECTURE

33. KEY CONCEPTS – Resiliency
BET / MARKET STORE
API API
KAFKA
PRODUCER
DC1 DC2
BET / MARKET STORE
KAFKA
PRODUCER

34. KEY CONCEPTS – Resiliency
BET / MARKET STORE
API API
KAFKA
PRODUCER
DC1 DC2
BET / MARKET STORE
KAFKA
PRODUCER

35. KEY CONCEPTS – Resiliency
BET / MARKET STORE
API API
KAFKA
PRODUCER
DC1 DC2
BET / MARKET STORE
KAFKA
PRODUCER

36. KEY CONCEPTS – Resiliency
BET / MARKET STORE
API API
KAFKA
PRODUCER
DC1 DC2
BET / MARKET STORE
KAFKA
PRODUCER