"Intro to Stateful Services or How to get 1 million RPS from a sing...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

https://nbomber.com

https://github.com/stereodb

AGENDA
- intro
- why stateless is slow and less reliable
- tools for building stateful services

PART I
intro to sportsbook domain
and
how we come to stateful

Dynamo Kyiv vs Chelsea
2 : 1
Red card
Score
changed
Odds
changed
PUSH
PULL

Dynamo Kyiv vs Chelsea
2 : 1
Red card
Score
changed
Odds
changed
PUSH
PULL
- quite big payloads: 30 KB compressed data (1....

20 KB payload for concurrent read and write
Redis, single node: 4vcpu - 8gb
redis_write: 4K RPS, p75 = 543, p95 = 688, p99...

API API API
DB

API
Cache
DB
but Cache is not queryable

API + DB
Stateful Service

state
state
state

CDC (Debezium)
DB

How to handle a case when
your data is larger than RAM?
10 GB 30 GB

Solution 1: use memory DB that supports data larger than RAM
10 GB
20 GB

UA PL FR
Solution 2: use partition by tenant

Solution 3: use range-based sharding
users
(1-500)
users
(501-1000)
shard A shard B

PART II
why stateless is slow

API + DB
Stateful Service
API
Cache
DB
network latency
network latency

Latency Numbers
Latency
2010 2020
Compress 1KB with Zippy 2μs 2μs
Read 1 MB sequentially from RAM 30μs 3μs
Read 1 MB seque...

API
Cache
DB
CPU: for serialize/deserialize
CPU: serialize/deserialize
API + DB
Stateful Service
CPU for serialize (we don...

API
Cache
DB
CPU: for serialize/deserialize
CPU for ASYNC request handling
CPU: serialize/deserialize
CPU: ASYNC request h...

API
Cache
DB
CPU: for serialize/deserialize
CPU for ASYNC request handling
CPU: managing sockets
CPU: serialize/deserializ...

API
Cache
DB
CPU: for serialize/deserialize
CPU for ASYNC request handling
CPU: managing sockets
CPU: serialize/deserializ...

API
Cache
DB
CPU: for serialize/deserialize
CPU for ASYNC request handling
CPU: managing sockets
Overreads
CPU: serialize/...

Object hit rate / Transactional hit rate

A B
C
API
In order to fulﬁll our transactional ﬂow we need to
fetch records: A, B, C
Record A and B will not impact our la...

$Most existing cache eviction algorithms focus on maximizing object hit rate, or the fraction of single object requests ser...$

async / await

async / await
Imagine that we run Redis on localhost. Even with such setup we
usually use async request handling.

public void SimpleMethod()
{
var k = 0;
for (int i = 0; i < Iterations; i++)
{
k = Add(i, i);
}
}
[MethodImpl(MethodImplOp...

public async Task SimpleMethodAsync()
{
var k = 0;
for (int i = 0; i < Iterations; i++)
{
k = await AddAsync(i, i);
}
}
pr...

public async Task SimpleMethodAsyncYield()
{
var k = 0;
for (int i = 0; i < Iterations; i++)
{
k = await AddAsync(i, i);
}...

public async Task SimpleMethodAsyncYield()
{
var k = 0;
for (int i = 0; i < Iterations; i++)
{
k = await AddAsync(i, i);
}...

PART III
why stateless is less reliable

API
Cache
DB
API + DB
Stateful Service
We have a higher probability of failure

API
Cache
DB
circuit breaker
retry
fallback
timeout
bulkhead isolation
circuit breaker
retry
fallback
timeout
bulkhead iso...

API
Cache
DB
What about cache invalidation
and data consistency?
API + DB
Stateful Service

API
Cache
DB
What about the predictable scale-out?
Will your RPS increase if you add an
additional API or Cache node?
API ...

- Metastable failures occur in open systems with an uncontrolled source of
load where a trigger causes the system to enter...

At least 4 out of 15 major outages in the
last decade at Amazon Web Services
were caused by metastable failures.

PART IV
tools for building stateful services

distributed log with sync replication

In-process memory DB
SQL OLAP

Dynamo Kyiv vs Chelsea
2 : 1
Red card
Score
changed
Odds
changed
PUSH
PULL
- quite big payloads: 30 KB compressed data (1....

THANKS
always benchmark
https://twitter.com/antyadev