"Intro to Stateful Services or How to get 1 million RPS from a single node", Anton Moldovan
Report
•0 likes•92 views
![public void SimpleMethod()
{
var k = 0;
for (int i = 0; i < Iterations; i++)
{
k = Add(i, i);
}
}
[MethodImpl(MethodImplOptions.NoInlining)]
private int Add(int a, int b) => a + b;](https://image.slidesharecdn.com/introtostatefulservicesorhowtoget1millionrpsfromasingle-230923133137-1463573d/75/intro-to-stateful-services-or-how-to-get-1-million-rps-from-a-single-node-anton-moldovan-39-2048.jpg?cb=1695476311)
"Intro to Stateful Services or How to get 1 million RPS from a single node", Anton Moldovan
•0 likes•92 views
Download to read offline
Report
Stateless is the most common approach for developing microservices. There are many reasons for this, but in short: very simple, very reliable, and of course, very scalable. But as we all know - there is no silver bullet. Some time ago, we faced certain limitations of this approach. Spoiler, Stateless turned out to be very slow, less reliable, and also more expensive. In my presentation, I will try to cover the following topics: Why we chose the Stateful approach instead of the Stateless and what our architecture looks like. How we build and manage Stateful services: service scale-out, data consistency, synchronization, and partitioning(sharding). Why Stateless services are less reliable and quite slow. What tools can we use for building Stateful services.
Recommended
More Related Content
Similar to "Intro to Stateful Services or How to get 1 million RPS from a single node", Anton Moldovan
Similar to "Intro to Stateful Services or How to get 1 million RPS from a single node", Anton Moldovan (20)
More from Fwdays
More from Fwdays(20)
Recently uploaded
Recently uploaded(20)
"Intro to Stateful Services or How to get 1 million RPS from a single node", Anton Moldovan
- 4. AGENDA - intro - why stateless is slow and less reliable - tools for building stateful services
- 5. PART I intro to sportsbook domain and how we come to stateful
- 6. Dynamo Kyiv vs Chelsea 2 : 1 Red card Score changed Odds changed PUSH PULL
- 7. Dynamo Kyiv vs Chelsea 2 : 1 Red card Score changed Odds changed PUSH PULL - quite big payloads: 30 KB compressed data (1.5 MB uncompressed) - update rate: 2K RPS (per tenant) - user query rate: 3-4K RPS (per tenant) - live data is very dynamic: no much sense to cache it - data should be queryable: simple KV is not enough - we need secondary indexes
- 8. 20 KB payload for concurrent read and write Redis, single node: 4vcpu - 8gb redis_write: 4K RPS, p75 = 543, p95 = 688, p99 = 842 redis read: 7K RPS, p75 = 970, p95 = 1278, p99 = 1597
- 10. API Cache DB but Cache is not queryable
- 11. API + DB Stateful Service
- 15. How to handle a case when your data is larger than RAM? 10 GB 30 GB
- 16. Solution 1: use memory DB that supports data larger than RAM 10 GB 20 GB
- 17. UA PL FR Solution 2: use partition by tenant
- 18. Solution 3: use range-based sharding users (1-500) users (501-1000) shard A shard B
- 19. PART II why stateless is slow
- 20. API + DB Stateful Service API Cache DB network latency network latency
- 21. Latency Numbers Latency 2010 2020 Compress 1KB with Zippy 2μs 2μs Read 1 MB sequentially from RAM 30μs 3μs Read 1 MB sequentially from SSD 494μs 49μs Read 1 MB sequentially from disk 3ms 825μs Round trip within same datacenter 500μs 500μs Send packet CA -> Netherlands -> CA 150ms 150ms https://colin-scott.github.io/personal_website/research/interactive_latency.html
- 22. API Cache DB CPU: for serialize/deserialize CPU: serialize/deserialize API + DB Stateful Service CPU for serialize (we don’t need to deserialize)
- 23. API Cache DB CPU: for serialize/deserialize CPU for ASYNC request handling CPU: serialize/deserialize CPU: ASYNC request handling API + DB Stateful Service CPU for serialize (we don’t need to deserialize)
- 24. API Cache DB CPU: for serialize/deserialize CPU for ASYNC request handling CPU: managing sockets CPU: serialize/deserialize CPU: ASYNC request handling CPU: managing sockets API + DB Stateful Service CPU for serialize (we don’t need to deserialize) CPU for managing sockets (only clients sockets )
- 25. API Cache DB CPU: for serialize/deserialize CPU for ASYNC request handling CPU: managing sockets CPU: serialize/deserialize CPU: ASYNC request handling CPU: managing sockets API + DB Stateful Service CPU for serialize (we don’t need to deserialize) CPU for managing sockets (only clients sockets ) CPU for handling query (very cheap compared to serialization)
- 26. API Cache DB CPU: for serialize/deserialize CPU for ASYNC request handling CPU: managing sockets Overreads CPU: serialize/deserialize CPU: ASYNC request handling CPU: managing sockets API + DB Stateful Service CPU for serialize (we don’t need to deserialize) CPU for managing sockets (only clients sockets ) CPU for handling query (very cheap compared to serialization)
- 32. Object hit rate / Transactional hit rate
- 33. A B C API In order to fulfill our transactional flow we need to fetch records: A, B, C Record A and B will not impact our latency Overall Latency = Latency of record C
- 36. Most existing cache eviction algorithms focus on maximizing object hit rate, or the fraction of single object requests served from cache. However, this approach fails to capture the inter-object dependencies within transactions.
- 37. async / await
- 38. async / await Imagine that we run Redis on localhost. Even with such setup we usually use async request handling.
- 39. public void SimpleMethod() { var k = 0; for (int i = 0; i < Iterations; i++) { k = Add(i, i); } } [MethodImpl(MethodImplOptions.NoInlining)] private int Add(int a, int b) => a + b;
- 40. public async Task SimpleMethodAsync() { var k = 0; for (int i = 0; i < Iterations; i++) { k = await AddAsync(i, i); } } private Task<int> AddAsync(int a, int b) { return Task.FromResult(a + b); }
- 41. public async Task SimpleMethodAsyncYield() { var k = 0; for (int i = 0; i < Iterations; i++) { k = await AddAsync(i, i); } } private async Task<int> AddAsync(int a, int b) { await Task.Yield(); return a + b; }
- 42. public async Task SimpleMethodAsyncYield() { var k = 0; for (int i = 0; i < Iterations; i++) { k = await AddAsync(i, i); } } private async Task<int> AddAsync(int a, int b) { await Task.Yield(); return await Task.Run(() => a + b); }
- 45. PART III why stateless is less reliable
- 46. API Cache DB API + DB Stateful Service We have a higher probability of failure
- 47. API Cache DB circuit breaker retry fallback timeout bulkhead isolation circuit breaker retry fallback timeout bulkhead isolation API + DB Stateful Service
- 48. API Cache DB What about cache invalidation and data consistency? API + DB Stateful Service
- 49. API Cache DB What about the predictable scale-out? Will your RPS increase if you add an additional API or Cache node? API + DB Stateful Service
- 51. - Metastable failures occur in open systems with an uncontrolled source of load where a trigger causes the system to enter a bad state that persists even when the trigger is removed. - Paradoxically, the root cause of these failures is often features that improve the efficiency or reliability of the system. - The characteristic of a metastable failure is that the sustaining effect keeps the system in the metastable failure state even after the trigger is removed.
- 52. At least 4 out of 15 major outages in the last decade at Amazon Web Services were caused by metastable failures.
- 54. PART IV tools for building stateful services
- 55. distributed log with sync replication
- 56. In-process memory DB SQL OLAP
- 59. Dynamo Kyiv vs Chelsea 2 : 1 Red card Score changed Odds changed PUSH PULL - quite big payloads: 30 KB compressed data (1.5 MB uncompressed) - update rate: 2K RPS (per tenant) - user query rate: 3-4K RPS (per tenant) - live data is very dynamic: no much sense to cache it - data should be queryable: simple KV is not enough - we need secondary indexes At pick to handle big load for 1 tenant we have: 5-10 nodes, 0.5-2 CPU, 6GB RAM