For one of our projects, we needed to improve the current content delivery system for terminals. In this talk, I will share our experience in building an efficient data replication system for thousands of terminals. We will touch on architecture decisions and tradeoffs, technologies that we used, and a bit of load testing. Spoiler: We didn't use Kafka.

1. BUILDING AN EFFICIENT
REPLICATION SYSTEM
by @antyadev
FOR THOUSANDS OF TERMINALS

2. About me
Software Architect at DraftKings:
- F#
- Distributed Systems
- Databases
OSS:
- Author of NBomber (https://nbomber.com)
Community:
- Organizer of Kyiv F# user group
- Organizer of F# Ukraine conference
Twitter: @antyadev
Github: @antyadev

3. 01. Architecture overview
02. Load testing
Agenda

4. PRELUDE

5. The Global Problem

6. Backend
The Global Problem
GET data to display
Terminal
Bet Shop

7. The Global Problem
GET initial data
GET real-time updates
Terminal
Backend
Bet Shop

8. The Global Problem
Bet Shop
Bet Shop
Bet Shop
Backend
GET
GET
GET

9. The Global Problem
for > 10K terminals
GET
Backend
GET
GET
GET GET
GET
GET
GET

10. Non-functional
requirements

11. 2008

12. Web Scale

13. Web Scale (2008)
1. Scalability
2. High availability
3. Fault Tolerance

14. I extend them a bit

15. Predictable Web Scale
1. Predictability (aka determinism)
2. Cost-effectiveness
3. Scalability (independent resource scale)
4. High availability
5. Fault Tolerance (fast recovery, redundancy)

16. Our hardware limitations
- Slow terminal machines (1-2 CPU, x86, 600MB RAM,
Windows/Linux/Android OS)
- Some customers have a slow internet (5 Mbps)
- Some customers have limited internet bandwidth

17. Data
- At pick we send 2MB per min of compressed data
- At pick we can get about 1-2K updates per sec

18. PART I
Architecture overview

19. Solution 1.0

20. PERIODIC PULL
(web scale 2008)

21. PERIODIC PULL
(web scale 2008)
for > 10K terminals
GET GET
GET
GET
GET
Backend

22. - hard to achieve near real-time
- very inefficient communication (extra traffic,
extra connections)
- no offline
- bad UX experience (slow)
- noticeable desynchronization between
terminals displays, TV
PERIODIC PULL

23. Solution 2.0

24. PUSH
+
LOCAL STORAGE
(predictable web scale)

25. Terminal Local db
GET initial data (snapshot)
SUBSCRIBE on updates (change stream)
PUSH + LOCAL STORAGE
Backend

26. But we have problems
with this solution
- SYNC data
- Network spikes

27. The Technical Problem
SYNC data

28. Bet Shop
Terminal
Terminal
Terminal
Backend
SYNC data
SYNC data

29. Bet Shop
replica
replica
replica
Backend
REPLICATION
SYNC data

30. for > 10K db replicas
REPLICATION
Backend
SYNC data

31. The Technical Problem
Network spikes

32. Network spikes
Usually, all bet shops open at the same time, which means that all terminals start to pull
an initial snapshot simultaneously
GET initial data (snapshot)Backend

33. let’s imagine that
1 terminal snapshot = 100MB
Local db
Network spikes
Usually, all bet shops open at the same time, which means that all terminals start to pull
an initial snapshot simultaneously

34. let’s imagine that
1 terminal snapshot = 100MB
1K terminals = 1GB
Local db
Local db
Local db
Network spikes
Usually, all bet shops open at the same time, which means that all terminals start to pull
an initial snapshot simultaneously

35. let’s imagine that
1 terminal snapshot = 100MB
1K terminals = 1GB
10K terminals = 10GB
for > 10K terminals
Network spikes
Usually, all bet shops open at the same time, which means that all terminals start to pull
an initial snapshot simultaneously

36. Naive Solution
KAFKA

37. Naive Solution
KAFKA
REPLICATION
for > 10K db replicasBackend

38. Why not?
KAFKA

39. - Kafka is designed for internal network communication (not external):
very tricky to expose to a public network
very tricky to apply any rate limit or concurrent clients limit
- Kafka client is not lightweight: it eats 1 thread and quite expensive to run on a slow machine (1-2
CPU, 600MB RAM, Windows/Linux/Android OS)
- We need a cache: it’s not designed as elastic cache layer to handle spikes:
all messages persisted
no fast point lookup to fetch missed updates
Why not?

40. And it’s only blockers!
There are other minor drawbacks:
elastic/independent scaling, messaging
optimizations, topic routing flexibility,
P2P channel support, etc.
Why not?

41. technology abuse
Why not?

42. What then?

43. Let’s back
to the problem

44. for > 10K db replicas
REPLICATION
SYNC data
Network spikes

45. for > 10K db replicas
REPLICATION
LOG
+
PUSH
SYNC data
Network spikes

46. for > 10K db replicas
REPLICATION
Network spikes
SYNC data
Cache
LOG
+
PUSH
IMPORTANT: with the ability to resume
the downloading process from the last
read message

47. Okay, and remember
1. Predictability (aka determinism)
2. Cost-effective
3. Scalability (independent resource scale)
4. High availability
5. Fault Tolerance (fast recovery, redundancy)

48. High-level architecture

49. streamer
> 10K terminals
Cache

50. streamer
> 10K terminals
Cache
ADD

51. streamer
PULL
> 10K terminals
Cache
ADD

52. Message
Broker
streamer
> 10K terminals

53. Message
Broker
streamer
PUBLISH
> 10K terminals

54. Message
Broker
streamer
PUBLISH
> 10K terminals
PUSH

55. Message
Broker
streamer
PULL
PUSH
> 10K terminals
Cache
PUSH
ADD

56. Message
Broker
streamer
> 10K terminals
Cache
An independently scalable component that
represents only persistent connections (sockets)
Computations are separated from sockets
and scales independently
An independently scalable component that
represents only data

57. What about
tech stack?

58. Cache

59. streams

60. streams
(log data structure)

61. producer consumer
groups
conceptually very similar to Kafka but:
- all data in memory (very fast access, range lookup)
- fixed size stream (autoclean by FIFO)
- you can read data from several topics in one RTT

62. How do we use it?

63. change log (1 msg = batch of updates)
- fixed size stream
- autoclean by FIFO
snapshot (splitted by chunks)
- autoclean by TTL

64. change log (1 msg = batch of updates)
- fixed size stream
- autoclean by FIFO
snapshot (splitted by chunks)
- autoclean by TTL
snapshot 1 snapshot 2 snapshot 3
time
change log

65. leader
replicareplica
Redis Streams Cluster

66. Message
Broker

68. EMQ X Broker
- massively scalable, highly extensible distributed MQTT message
broker written in Erlang/OTP
(masterless, low latency, high concurrency)
- open sourced
- TCP/SSL, WebSockets (SSL), MQTT v3.1/5.0
- one instance (16 vCPUs, 64 GB RAM) can handle 1 million connections
(we didn’t check it)

69. MQTT Protocol
- very lightweight PUB/SUB protocol designed for IoT
(binary and header requires 2 bytes)
- supports hierarchical flexible topic structure (+, #)
- supports standard levels of message delivery
guarantee
- good support on all popular platforms/runtimes

70. FULL picture

71. change log
snapshot
EMQ
streamer
PULL
PUSH
ADD
PUBLISH
> 10K terminals

72. Now, a bit of detail...

73. How do we guarantee
consistency?

74. Flow Control

75. Basically, every message has
a monotonically increasing number
(aka Kafka’s offset)
type UpdateBulkMessage = {
StreamerId: string
Updates: UpdateMessage[]
BulkNumber: uint
}

76. What can we do regards
network traffic within one
betshop?

77. Leader Election

78. RAFT
(2014)

79. RAFT
(2014)
Leader election (we implemented only this)
Log replication
Safety

80. RAFT
(2014)
F#: 230 LOC
property & integration tests: ~200 LOC

81. RAFT
(2014)
[<Property>]
let ``handleRequest should handle heartbeat correctly`` (nodeTerm: int, requestTerm: int) =
let requestNodeId = TestInfra.createNodeId(port = 50)
let request = HeartBeat({ NodeId = requestNodeId; Term = requestTerm |> UMX.tag })
let node = Follower <| TestInfra.createFollower(port = 2, currentTerm = UMX.tag nodeTerm)
match LeaderElection.handleRequest(node, request) with
| LeaderAccepted (response, newState) ->
test <@ nodeTerm <= requestTerm @>
test <@ newState.LeaderId.Value = requestNodeId @>
test <@ (int)newState.State.CurrentTerm = requestTerm @>
test <@ response.VoteGranted = true @>
| LeaderRejected response ->
test <@ nodeTerm > requestTerm @>
test <@ (int)response.CurrentTerm = nodeTerm @>
test <@ response.VoteGranted = false @>

82. RAFT
(2014)
[<Fact>]
let ``Leader may be only one among all nodes`` () =
let nodes = TestInfra.startNodes(nodeCount = 3, ipMinPort = 10_010, udpPort = 9001)
try
Task.Delay(3_000).Wait() // wait on leader election
let leader = nodes |> Seq.find(fun x -> x.IsLeader)
let leaderCount = nodes |> Seq.filter(fun x -> x.IsLeader) |> Seq.length
let sameLeaderForAllMembers =
nodes
|> Seq.filter(fun x -> not x.IsLeader)
|> Seq.map(fun x -> x.GetCurrentLeader())
|> Seq.forall(fun x -> leader.GetCurrentLeader() = x)
test <@ leaderCount = 1 @>
test <@ sameLeaderForAllMembers = true @>
finally
nodes |> Seq.iter(fun x -> x.Stop())

83. What about persistence?

84. It’s our internal
in-process
in-memory database
NoSQLite

85. - In-memory KV-store with ability to persist on disk and
configure memory footprint
- ACID (per single node),
limited SQL,
secondary indexes (in-memory)
- Cluster:
- zero configuration (service discovery via UDP broadcast)
- auto failover
- async replication
NoSQLite

86. PART II
Load testing

87. Why!?

88. super simple
question

89. do you know
the throughput (RPS)
and latency
of your server/API?

90. now, the punchline

91. How many instances
you need to have
to handle
1K concurrent users?

92. How you identify
performance degradation?

93. How you prove that
your architecture is
bulletproofed?

94. What about POC?

95. Ok, what do we use for
load testing?

96. NBomber - your drugs for
load testing
https://nbomber.com/
https://nbomber.com/docs/
https://github.com/pragmaticflow

97. Why we build NBomber and what you
can do with it?
01. The main reason behind NBomber is to provide a lightweight framework for
writing load tests which you can use to test literally any system and simulate
any production workload. We wanted to provide only a few abstractions so
that we could describe any type of load and still have a simple, intuitive API.
02. Another goal is to provide building blocks to validate your POC (proof of
concept) projects by applying any complex load distribution.
03. With NBomber you can test any PULL or PUSH system (HTTP, WebSockets,
GraphQl, gRPC, SQL Databse, MongoDb, Redis etc).
04. With NBomber you can convert some of your integration tests to load tests
easily.

98. NBomber as a modern framework
provides:
- Zero dependencies on protocol (HTTP/WebSockets/AMQP/SQL)
- Zero dependencies on semantic model (Pull/Push)
- Very flexible configuration and dead simple API (F#/C#/JSON)
- Cluster support
- Realtime Reporting
- Plugins support
- CI/CD integration
- Data feed support

99. Show me a code!

100. Hello World
var step = Step.Create("step", async context =>
{
// you can define and execute any logic here,
// for example: send http request, SQL query etc
// NBomber will measure how much time it takes to execute your logic
await Task.Delay(TimeSpan.FromSeconds(1));
return Response.Ok();
});
// second, we add our step to the scenario
var scenario = ScenarioBuilder.CreateScenario("hello_world", step);
NBomberRunner
.RegisterScenarios(scenario)
.Run();

101. Hello World
var step = Step.Create("step", async context =>
{
// you can define and execute any logic here,
// for example: send http request, SQL query etc
// NBomber will measure how much time it takes to execute your logic
await Task.Delay(TimeSpan.FromSeconds(1));
return Response.Ok();
});
// second, we add our step to the scenario
var scenario = ScenarioBuilder.CreateScenario("hello_world", step);
NBomberRunner
.RegisterScenarios(scenario)
.Run();

102. var httpClient = new HttpClient();
var step = Step.Create("step", async context =>
{
var response = await httpClient.GetAsync("https://nbomber.com",
context.CancellationToken);
return response.IsSuccessStatusCode
? Response.Ok()
: Response.Fail();
});
var scenario = ScenarioBuilder.CreateScenario("hello_world", step);
NBomberRunner
.RegisterScenarios(scenario)
.Run();
HTTP load test
(raw)

103. var httpClient = new HttpClient();
var step = Step.Create("step", async context =>
{
var response = await httpClient.GetAsync("https://nbomber.com",
context.CancellationToken);
return response.IsSuccessStatusCode
? Response.Ok()
: Response.Fail();
});
var scenario = ScenarioBuilder.CreateScenario("hello_world", step);
NBomberRunner
.RegisterScenarios(scenario)
.Run();
HTTP load test
(raw)

104. var step = HttpStep.Create("fetch_html_page", fun context ->
Http.CreateRequest("GET", "https://nbomber.com")
.WithHeader("GET", "GET")
);
var scenario = ScenarioBuilder.CreateScenario("hello_world", step);
NBomberRunner
.RegisterScenarios(scenario)
.Run();
HTTP load test with HTTP plugin
(production-ready)

105. leader
replicareplica
PULL
PULL
PULL
PULL
coordinator
streamers
ADD
agent
agent
agent
agent

106. Redis v6.0.3
1 master and 2 slaves with async replication, without sharding, without persistence (ONLY
in-memory)
test: 1 writer is publishing to the stream 32KB per 500ms and many readers are reading
random items
20K readers (open system)
RPS 50% 75% 95% StdDev data transfer
20K 303 390 476 121 32 KB
30K readers (open system)
RPS 50% 75% 95% StdDev data transfer
30K 414 520 627 158 32 KB
10K readers (open system)
RPS 50% 75% 95% StdDev data transfer
10K 120 160 216 66 32 KB

107. coordinator
streamers
PUBLISH
EMQ
EMQEMQ
SUBSCRIBE
SUBSCRIBE
agent
agent

108. coordinator
streamers
PUBLISH
EMQ
EMQEMQ
agent
agent

109. agent
agent
EMQ
EMQEMQ
coordinator
RECEIVE

110. EMQX v4.0.7
3 services without persistence (k8 limits: 1CPU, memory: 512MB)
test: 1 writer is publishing to the topic 32KB per 500ms and many subscribers are
consuming messages and verifying order of messages
10K subscribers (closed system)
RPS 50% 75% 95% StdDev data transfer
20K 177 260 400 42 32 KB
5K subscribers (closed system)
RPS 50% 75% 95% StdDev data transfer
10K 121 190 240 22 32 KB

111. What else?

112. NBomber plugins & extensions
Http plugin - a simple DSL for working with HTTP.
Ping plugin - a plugin that executes ping logic to gather networking metrics.
InfluxDb sink - a data sink for InfluxDb.
Prometheus sink - a data sink for Prometheus.
Db Benchmark - a plugin that helps you to simulate popular database workloads
(OLTP, OLAP).
K8 Metrics - a plugin that allows you to fetch and render metrics from K8 and apply
assertions.
K8 Chaos Monkey - a plugin that helps you to inject faults into your target system
and render stats.

113. Documentation,
cluster JSON
Cluster schema
with agents and
groups

114. THANKS
@antyadev
https://nbomber.com/
https://nbomber.com/docs/
https://fsharp-ukraine.org/