We live in an age of rapid innovation, but our infrastructure shows signs of rust and old age. Under the shiny exterior of our new technology, some old and dusty infrastructure is carrying a weight it was never designed to hold. This is glaringly true in the database world, which remains dominated by databases architected for a different age. These legacy NoSQL databases were designed for a different (nascent) cloud, different hardware, different bottlenecks, and even different programming models. Join us as Avishai Ish-Shalom shares his insights on the massive disconnect between the potential of a brave new distributed world and the reality of “modern” applications relying on fundamentally aged databases. We’ll cover: - The changes in the operating environment of databases - The changes in the business requirements for databases - The architecture gap of legacy databases - New designs for a new world - How the adoption of modern databases impacts engineering teams and the products you’re building

1. Beyond Jurassic NoSQL:
New Designs for a New
World
Avishai Ish-Shalom (@nukemberg)

2. ~ whoami
2
Avishai Ish-Shalom
+ 17 years in the industry
+ On both the ops and dev sides
+ Wix, Aleph VC, ScyllaDB alumni

3. Databases are old
3
Current RDBMS (Oracle, Postgres et-al) ~ 30 years old
Actually, the architecture and design are 40 years old
For reference:
+ AWS launched in 2006, GCP 2013
+ iPhone launched in 2007
+ Facebook launched 2004

4. Agenda
■ Operating environment changes
■ Business requirements changes
■ The architecture gap of legacy databases
■ New designs for a new world
4

5. Part I: Partly
cloudy
The weather changed
5

6. + On premises datacenters
+ Manually managed servers
+ Enterprise grade hardware
+ Fancy SAN with RAID, battery backed cache, etc
+ Dedicated networks (e.g. RAC interconnect)
+ Single active location
The DC, circa 2002
6

7. + Cloud
+ Automatically managed servers
+ Commodity hardware, often virtualized
+ Local storage or distributed Software Defined Storage
+ Common networks, SDN
+ Multi region, multi cloud, global infra
Fast forward to 2022
7

8. Oh, and CPU core:node 1 → 100
Let’s not forget the hardware
8
Change 2002 → 2022 Latency (ratio) Bandwidth (ratio)
RAM:storage 1E6 → 1E3 40 → 2
RAM:Network 2E5 → 1E5 40 → 1
Network:storage 20 → 0.1 2/3 → 4

9. + Virtualized hardware
+ Virtualized network
+ Virtualized storage
+ Short provision time
+ Cost hedging
+ Statistical failure in large numbers
Server uptime: years (2002) → months (2022), days/hours for app servers
Dynamic topologies
9

10. + Cloud compute nodes are ephemeral (yes, that’s a good thing!)
+ The cloud provides replicated storage
+ Node reprovisioning == Really hard reboot
+ Easy to change the number of nodes
+ But can you change it?
+ What about node configuration?
The ephemeral node conundrum
10

11. When replicas replicate
11
“disk” “disk”
Cloud storage servers
Multiple replication layers… Why do we even care?
+ Disjoint failure domains (what AZs are used?)
+ Performance issues
+ Debuggability
If a node failed, do we reboot or fail over?

12. The business environment
changed too
12

13. The world went online
13

14. + Number of of users increased - still growing, but will stabilize
+ But users do more
+ With more devices
+ And the devices also want service
More concurrency, more throughput, more data
Digital everything
14

15. + Users move globally
+ Users interact globally
+ Through a global infrastructure
+ That has local presence
Digital everywhere
15

16. You know what hasn’t
changed? Speed of light!
16

17. Ping time SF → Tokyo: ~100ms (on a good day)
+ Local PoP is no longer optional
+ Data intensive software (e.g. ML/AI) → local DC no longer optional
+ Local data governance (e.g. GDPR/CCPA) → local DC mandatory
Global market, local speeds
17

18. + 1000x more end users
+ 1000x more CPUs
+ 1000x more clients, connections
Concurrency increased
18

19. Part II: How do we
scale?
A crash course in scalability
19

20. The Universal Scalability Law
20
Retrograde
scaling -
coordination,
consensus
Diminishing
returns -
locks, shared
resources

21. Warp space
21
Sharding Replication
datum X
datum X
datum Z
datum Y

22. Disjoint datasets
+ Scale reads/writes
+ Scale dataset
+ Non uniform load
+ Share nothing
+ No joins
+ No transactions
Sharding
22
All replicas are equivalent
+ Scale reads only
+ Requires coordination on write
+ Improve availability
Replication

23. Warp time
23
op 1
op 2
op 3
op 4
…
…
op 413138
op 413139
op 413140
op 1
op 2
op 3
op 4
…
…
op 413138
op 1
op 2
op 3
op 4
…
…
op 413138
op 413139
op 413140
op 1
op 3
op 2
…
…
op 413140
op 4
op 413138
Lag Re-order

24. Async replication
+ Limited parallelism (1 per shard)
+ Deterministic operation results
+ Queuing latency penalty
Lag
24
No coupling between operations
+ No head-of-line blocking
+ Infinite parallelism
+ Resilient, low latency
+ Application must reason operation
results
Re-order

25. The Universal Scalability Law
25
Sync
replication,
Cross shard
transactions
Replication
with lag

26. Part III: Towards
a modern DB
26

27. 27
+ Dynamic topology ← ephemeral cloud nodes
+ Server/AZ failure domains ← no one cares about disks
+ Read, write scalability ← user count will only grow
+ CPU scalability ← CPUs ain’t getting faster
+ Geo replication ← speed of light ain’t changing
+ Support high concurrency ← more CPUs, clients
+ Automatic node configuration, maintenance ← node count will only grow
+ Automatic cluster management
What do we want from a modern DB?

28. + Assumed physical, monolithic servers
+ On failure, reboot and recover
+ Often required special hardware
+ Stateful connections → low concurrency
+ CPU scaling? Locks, locks, locks
+ Replication added as an afterthought
Early 2000s: RDBMS
28

29. + Native sharding, async replication
+ Shard coupled to specific nodes
+ No granularity in shard size
+ Manual cluster management
+ Static topology
+ Small clusters only
+ CPU scaling? Locks, locks, locks
2009: MongoDB
29

30. + Native sharding, async replication, re-ordering
+ Small shards
+ Decoupled shards from servers
+ Dynamic topology
+ Cluster changes still painful though
+ Large clusters
+ Geo replication
+ CPU scaling? Locks, locks, locks
2013: Cassandra 1.2
30

31. + Native sharding, async replication, re-ordering
+ Small shards
+ Decoupled shards from servers
+ Dynamic topology
+ Cluster changes still somewhat painful
+ Large clusters
+ Geo replication
+ CPU scaling: shard-per-core
+ Automatic node configuration
2016: ScyllaDB 1.0
31

32. + Native sharding, semi-sync replication
+ Dynamic shards
+ Decoupled shards from servers
+ Dynamic topology
+ Cluster changes still somewhat painful
+ Large clusters
+ Geo replication
+ CPU scaling: Locks, locks, locks
+ ACID Transactions!
2015: CockroachDB
32

33. + Native sharding, async replication, re-ordering
+ Dynamic shard size (AKA tablets)
+ Decoupled shards from servers
+ Dynamic topology
+ Large clusters
+ Geo replication
+ CPU scaling: shard-per-core
+ Automatic node configuration
2022: ScyllaDB 5.0
33

34. It’s ok!
+ Application developers find re-ordering hard
+ Modern event based architectures enforce order (e.g. Kafka)
+ Network throughput improved 40x since 2002
+ Parallelization through shards
+ Dynamic shard sizing helps
Wait, no re-ordering?
34

35. + Stateless client protocols
+ HTTP/3 would be great!
+ Distributed tracing support
+ Burst capacity
+ Object storage support
+ Geo policies
My personal wishlist
35

36. Thank you
for joining us today.
@scylladb scylladb/
slack.scylladb.com
@scylladb company/scylladb/
scylladb/