See how Redis can work as the primary database for many situations:

1. 10 Reasons why Redis should
be your Primary Database
YIFTACH SHOOLMAN
CO-FOUNDER AND CTO @ REDIS LABS

2. The Database Market
2
NOSQL
35.0%
CAGR
2016-21
RELATIONAL
7.5%
CAGR
2016-21
Source: 451 Research Total Data Market Monitor

3. DB-Engines Ranking
3

4. Docker Hub: The World’s Most Popular Database
# of containers launched as of Feb 2018
630M+
(1.87M/day, 78K/hr, 1.28K/min)
308M+
263M+
24M+
4

5. Redis Enterprise
5
DBaaS
• Available since mid 2013
• 8,100+ enterprise customers
Software
• Available since early 2015
• 300+ enterprise customers
550K+ databases
managed worldwide
• 6 of top Fortune 10 companies
• 3 of top 5 communications companies
Customers
• 3 of top 4 credit card issuers
• 3 of top 5 healthcare companies

6. 6
It’s Fast (Extremely Fast) and Scales Linearly
1

7. Redis Enterprise Cluster
Node 1 Node 2 Node N (odd number)
7
Uneven number of symmetric nodes

8. Redis Enterprise Cluster
Node 1 Node 2 Node N (odd number)
8
Single master database
M

9. Redis Enterprise Cluster
Node 1 Node 2 Node N (odd number)
9
An HA database
SM

10. Redis Enterprise Cluster
Node 1 Node 2 Node N (odd number)
10
A Clustered Database
M1 M2 M3

11. Redis Enterprise Cluster
Node 1 Node 2 Node N (odd number)
11
A Clustered Database
M1 M2 M3S3 S1 S2

12. Redis Enterprise Node
Cluster
Manager
Enterprise Layer
Open Source Layer
REST API
Zero latency proxy
Redis Shards

13. Redis Enterprise: Shared Nothing Symmetric Architecture
Node 1 Node 2 Node N (odd number)
Redis Shards
& Proxies
Data Path
13
Data-Path and Control/Management Path Separation

14. Redis Enterprise: Shared Nothing Symmetric Architecture
Cluster
Management
Path
Node Watchdog
Cluster Watchdog
Node 1 Node 2 Node N (odd number)
Redis Shards
& Proxies
Data Path
14
Data-Path and Control/Management Path Separation

15. 15
Seamless Resharding
M
S
M1
M2
S1
S2

16. 16
Seamless Resharding
Proxy
M
S

17. 17
Seamless Resharding
Proxy
1 Bring trimmed slaves
S1
S2
½ of the
dataset
½ of the
dataset
M
S

18. 18
Seamless Resharding
Proxy
2 Start draining
S1
S2
Until no
outstanding
requests
(~msec)
M
S

19. 19
Seamless Resharding
Proxy
3 MàM1, S->M2 & stop draining
S1
S2
M1
M2

20. 20
Seamless Resharding
Proxy
4 Trim M1, M2
S1
S2
M1
M2

21. 21
Scaling Out / In
Scaling out, Resharding & Rebalancing
Scale out Rebalancing Resharding

22. x2 faster
Application
1 2 3 4
5 6 7
1
2
3
4
5
6
7
Multiplexing &
pipelining
With Proxy
Proxy
How Does The Proxy Work ?

23. Single-proxy – dense policy Multi-proxy – sparse policy
Scale-Out Proxy
Single Database Endpoint Single Database Endpoint

24. Scaling Linearly with OSS Cluster API
Node 1
Proxy 1 Shard 1 Shard 2 Shard n/m
Node 2
Proxy 2 Shard n/m+1 Shard n/2mShard n/m+2
Node M
Proxy M Shard n-m+1 Shard nShard n-m+2
M nodes Redis Enterprise Cluster with N shards database
Client 1
Client 2
Client k
HS ranges = {ranges #1, ranges #2, ranges #n/m}
HS ranges = {ranges #n/m+1, ranges #n/m+2, ranges #n/2m}
HS ranges = {ranges #n-m+1, ranges #n-m+2, ranges #n}

25. 6 nodes, 120 shards cluster 12 nodes, 240 shards cluster 18 nodes, 360 shards cluster
True Linear Scalability

26. 6 nodes, 120 shards cluster 12 nodes, 240 shards cluster 18 nodes, 360 shards cluster
True Linear Scalability

27. 6 nodes, 120 shards cluster 12 nodes, 240 shards cluster 18 nodes, 360 shards cluster
Sub-millisecond latency is maintained across all the tests
True Linear Scalability

28. 28
It’s Highly Available
2

29. 29
Lessons learned
m4.large
a quorum node
with no-data
Redis Enterprise
Node 3
• 5+ years in production
• 550K+ database created
• 50+ data-centers/zones
• 2000+ node failure events
• 100+ complete data-center outages

30. 30
HA Concept #1– Quorum by Nodes, not by Shards
3 replicas Redis
90GB
r4.4xlarge
m4.large
a quorum node
with no-data
Redis Enterprise
90GB
r4.4xlarge
90GB
r4.4xlarge
90GB
r4.4xlarge
90GB
r4.4xlarge
M1 S1 S2
Node 1 Node 2 Node 3 Node 1 Node 2 Node 3
M1 S1

31. 31
HA Concept #1 – Quorum by Nodes, not by Shards
3 replicas Redis
90GB
r4.4xlarge
m4.large
a quorum node
with no-data
Redis Enterprise
90GB
r4.4xlarge
90GB
r4.4xlarge
90GB
r4.4xlarge
90GB
r4.4xlarge
M1 S1 S2
Node 1 Node 2 Node 3 Node 1 Node 2 Node 3
M1 S1

32. 32
HA Concept #1 – Quorum by Nodes, not by Shards
3 replicas Redis
90GB
r4.4xlarge
m4.large
a quorum node
with no-data
Redis Enterprise
90GB
r4.4xlarge
90GB
r4.4xlarge
90GB
r4.4xlarge
90GB
r4.4xlarge
M1 S1 S2
Node 1 Node 2 Node 3 Node 1 Node 2 Node 3
M1 S1
• ~30% infrastructure cost savings
• Less network traffic
• Easy to manage

33. 33
HA Concept #2 – Pure In-Memory Replication
Disk-based Replication
(OSS default)
M S
OSS Diskless Replication
M S
Pure In-Memory Replication
M S
1
2
3 1 2
1

34. 34
HA Concept #2 – Pure In-Memory Replication
Disk-based Replication
(OSS default)
M S
OSS Diskless Replication
M S
Pure In-Memory Replication
M S
1
2
3 1 2
1

35. 35
HA Concept #2 – Pure In-Memory Replication
Disk-based Replication
(OSS default)
M S
OSS Diskless Replication
M S
Pure In-Memory Replication
M S
1
2
3 1 2
1
x2 faster

36. 36
HA Concept #3 – Watchdogs are Part of the Cluster

37. 37
HA Concept #3 – Watchdogs are Part of the Cluster

38. HA Concept #4 – How to deploy a Multi-AZ/Rack Cluster
1. At least 3 AZs/Racks
2. Distance between Azs/Racks
< 10msec
3. Master and Slave of the same
shard must be deployed
on different AZs/Rack
4. For every i,j,k:
#_of_node ( AZi +AZj) > #_of_node AZk

39. Redis Failover Benchmark
%oftimesdatawaslost
Averagetimetorecoverinseconds
Redis
Enterprise
AWS
ElastiCache
Heroku
Redis
Compose (IBM)
Redis
Azure
Redis Cache
<5sec

40. 40
It’s Durable
3

41. Data Loss
SSD - persistent and
ephemeral
data
Failed Instance
SSD - persistent
and ephemeral
data
New Empty Instance
Data-Persistence - The Wrong Way
Uses Network Attached Persistent Storage, not Ephemeral
Discuss SQL Strategy

42. Data Loss
SSD - persistent and
ephemeral
data
Failed Instance
SSD - persistent
and ephemeral
data
New Empty Instance
Data-Persistence - The Wrong Way
No Data Loss
SSD - ephemeral
data
Failed Instance
SSD - ephemeral
data
New Populated Instance
Persistent Storage
AOF,
Snapshot
Data-Persistence - The Right Way
AOF,
Snapshot
Dataload
Uses Network Attached Persistent Storage not Ephemeral
Discuss SQL Strategy

43. Tunable Data Persistence Configuration
Non-Replicated
M

44. Tunable Data Persistence Configuration
Tuned for Speed
Data-Persistence at the slave
M S
Non-Replicated
M

45. Tunable Data Persistence Configuration
Tuned for Speed
Data-Persistence at the slave
M S
Tuned for Reliability
Data-Persistence at the master & slave
M S
Non-Replicated
M
AOF-every-sec, AOF-every-write, Snapshot (RDB)

46. • Redis performance during AOF
rewrite
• Data-persistence when multiple
Redis instances reside on the
same node
Two Main Challenges with Redis Data-Persistence

47. Redis Enterprise Enhanced Storage Engine
Redis Labs Proprietary and Confidential Information

48. 48
It’s Modular
4

49. 49
Couchbas
e
Riak
Cassandra
DSE
OrientDB
Graph
Neo4j
InfiniteGr
aph
Titan
Key Value
Redis
RethinkD
B
DynamoD
B
Oracle
NoSQL
HBase
Wide
Column
Hypertabl
e
Cloud
BigTable
Document
MongoDB
CouchDB
Documen
tDB
Cloudant
Aerospike
Polyglot Persistence

50. 50
Couch
base
Riak
Cassan
dra
DSE
Orient
DB
Graph
Neo4j
Infinite
Graph
Titan
Key Value
Redis
Rethin
kDB
Dynam
oDB
Oracle
NoSQL
HBase
Wide
Column
Hypert
able
Cloud
BigTabl
e
Document
Mongo
DB
Couch
DB
Docum
entDB
Clouda
nt
Aerosp
ike
Polyglot Persistence Multi-Model
The Trend

51. 51
Redis-ML
RediSearch ReJSON
Redis-Timeseries
Redis-Graph
Rebloom
Custom
?
Rate Limiter
Multi-Model – based on Open Core with Modules

52. 52
And It’s still Fast (extremely fast) with Modules
RedisSearch – x5

53. 53
And It’s still Fast (extremely fast) with Modules
ReBloom – x17RedisSearch – x5

54. 54
msec msec
RedisSearch – x5 Redis-ML – x2000
And It’s still Fast (extremely fast) with Modules
ReBloom – x17

55. 55
msec msec
Redis-ML – x2000
And It’s still Fast (extremely fast) with Modules
Redis-Graph
Wait for Redisconf:
Pier 27, San-Francisco
April 26-88
RedisSearch – x5 ReBloom – x17

56. 56
It Uses a Different Approach for Active-Active
5

57. App
Eventual Consistency à 100msec
Active-Active: Existing Approaches are just Slow
App
App

58. App
Active-Active: Existing Approaches are just Slow
App
App
App
App
App
Eventual Consistency à 100msec Strong Consistency à 200msec

59. We Need Something Faster than the Speed of Light
Light > 20msec RTT
Network > 70msec RTT
Redis < 1msec RTT

60. Conflict Resolution is Hard
• Application level solution à too complex to write
• LWW (Last Write Wins) à doesn’t work for many of the Redis use cases, e.g.:
• Counters
• Sets
• Sorted Sets
• Lists
• Modules’ new datatypes

61. CRDT
• Years of academic research
• Based on consensus free protocol
• Strong eventual consistency
• Built to resolve conflicts with complex data types

62. The CRDT Approach
App
App
App
Strong Eventual Consistency à 1 msec

63. Solving Conflicts – Counters
c = 500
Replica A
c = 500
Replica B
c = 500
Replica C
63

64. Solving Conflicts – Counters
c = 500
INCRBY 200
Replica A
c = 500
Replica B
c = 500
Replica C
64

65. Solving Conflicts – Counters
c = 500
INCRBY 200
Replica A
c = 500
DECRBY 300
Replica B
c = 500
Replica C
65

66. Solving Conflicts – Counters
c = 500
INCRBY 200
Replica A
c = 500
DECRBY 300
Replica B
c = 500
INCRBY 1000
Replica C
66

67. Convergence Function (commutative):
500 + ∑c(i) = 500 +200 -300 +1000 = 1400
Solving Conflicts – Counters
c = 500
INCRBY 200
Replica A
c = 500
DECRBY 300
Replica B
c = 500
INCRBY 1000
Replica C
67

68. Solving Conflicts – Sets
S = {A, B, C}
Replica A
S = {A, B, C}
Replica B
S = {A, B, C}
Replica C
68

69. Solving Conflicts – Sets
S = {A, B, C}
SADD D
Replica A
S = {A, B, C}
Replica B
S = {A, B, C}
Replica C
69

70. Solving Conflicts – Sets
S = {A, B, C}
SADD D
Replica A
S = {A, B, C}
SADD A
Replica B
S = {A, B, C}
Replica C
70

71. Solving Conflicts – Sets
S = {A, B, C}
SADD D
Replica A
S = {A, B, C}
SADD A
Replica B
S = {A, B, C}
SREM A
Replica C
71

72. Convergence Function (associative):
• S = S + D + A - A = {A, B, C, D}
• Observed Removed + Add Wins
Solving Conflicts – Sets
S = {A, B, C}
SADD D
Replica A
S = {A, B, C}
SADD A
Replica B
S = {A, B, C}
SREM A
Replica C
72

73. Causal Consistency

74. This is NOT Causal Consistency
Replica A
S = {A}
Replica B Replica C
74
SADD A
SADD A

75. This is NOT Causal Consistency
Replica A
S = {A}
Replica B Replica C
75
SADD A
S = {A, B}
SADD B
SADD A
S = {B, A}

76. This is Casual Consistency
Replica A
S = {A}
Replica B Replica C
76
SADD A
S = {A, B}
SADD B
SADD A
S = {A, B}

77. Active-Active: Comparison
App
App
App
App
App
App
Eventual Consistency
Strong Consistency
App
App
App
Strong Eventual Consistency
+
Causal Consistency
100msec 200msec <1msec

78. 78
It Saves You $$
6

79. Multi-Tenant from Day One
• Single tenant multiple
shards/DBs

80. • Multi-tenant multiple
shards/ DBs
• Customer B
• Customer A
• Customer N
Multi-Tenant from Day One
• Single tenant multiple
shards/DBs
• OR

81. #1
200GB
#2
200GB
#50
200GB
50 x r3.8xlarge instances
#51
200GB
#51
200GB
#100
200GB
1st replica for HA
#101
200GB
#102
200GB
#150
200GB
2nd replica for quorum
Total cost (reserved instances) = $2,132,250/yr
10TB Deployment on AWS with 2 Replicas

82. #1
200GB
#2
200GB
#50
200GB
50 x r3.8xlarge instances
#51
200GB
#51
200GB
#100
200GB
1 replica for HA
#101
15GB
Total cost (reserved instances) = $1,421,500/yr Savings = $710,750/yr
1 quorum server
10TB Deployment on AWS with 1 Replica + a Quorum Server

83. Redis on Flash – Built for a Tiered Memory Architecture
Persistent Storage:
Entire Dataset
AOF, Snapshot
SSD:
Cold Values
DRAM:
Keys & Hot Values
Cluster Node
83

84. RoF - Designed for the New Persistent Memory Technology
84
NVMe vs. SATA

85. RoF - Designed for the New Persistent Memory Technology
85
Optane (3DXP) vs. NVMe

86. $2,200,162/yr
$1,772,606/yr
$766,096/yr
$232,875/yr
$0/yr
$500,000/yr
$1,000,000/yr
$1,500,000/yr
$2,000,000/yr
$2,500,000/yr
Other Redis Provider RCP RAM Cloud-Based NoSQL RCP Flash
<1 msec <10 msec <1 msec
DBaaS Price Comparison
2TB Dataset with HA @ 100k ops/sec (on-demand pricing)
Up to 89% savings!
86
Other Redis Provider
< 1msec
Redis Enterprise VPC
< 1msec
Cloud-Based NoSQL
< 10 msec
RoF
< 1 msec

87. RoF by Numbers
• GA zero touch – 11/2017
• Quite a few customers
• All of them are using it as a primary data-store
• Database size 0.5TB à 10TB+

88. 88
It’s Everywhere
7

89. Multi-Cloud / Hybrid
App
App
App
App

90. All Verticals
Financial Services AdvertisingMedia
Technology Communications EducationGaming
Banks E-commerce
Business Services
Social
Travel
90

91. Device
Devices
• Raspberry Pi support
• A single OSS Redis instance
(3MB footprint)
• Persistent
• Redis Streams Client

92. Edge
Devices
• Raspberry Pi support
• A single OSS Redis instance
(3MB footprint)
• Persistent
• Redis Streams Client
• RPi/x86 nodes
(4 cores/2GB RAM /50GB SSD)
• RedisEnterprise Cluster
• Redis Streams Server & Client
• Persistent & HA
• Redison Flash
• Modules:
‒ Search, JSON, Graph, Time-Series,
ReBloom, ML
Edge

93. Everywhere
Devices
• Raspberry Pi support
• A single OSS Redis instance
(3MB footprint)
• Persistent
• Redis Streams Client
• RPi/x86 nodes
(4 cores/2GB RAM /50GB SSD)
• RedisEnterprise Cluster
• Persistent & HA
• Redison Flash
• Redis Streams Server & Client
• Modules:
‒ Search, JSON, Graph, Time-Series,
ReBloom, ML
Edge
• Large RedisEnterpriseCluster(s)
• Multi-cloud/DBaaS/Self-managed
• Multi-master geo-replication
• Redis Streams Server
• Persistent & HA (multi-az)
• Redison Flash
• Modules:
‒ Search, JSON, Graph, Time-Series,
ReBloom, ML,
Cloud

94. 94
It Simplifies Data Services
8

95. Cloud Data-Services Architecture
AWS Data-Services Flow

96. Cloud Data-Services Architecture
AWS Data-Services Flow
4
1
2
5
3

97. The WRONG Spaghetti Architecture
AWS Data-Services Flow
=

98. With the RIGHT Data-Services Architecture
or
Stream API Streams Data-
Structure
Multi-Function
1 2 3

99. 99
Because most of you have already been using it
as a primary data-store
9

100. Redis Enterprise Survey Data
Use Cases
YES
67%
NO
33%
Primary Database Move to Redis Enterprise

101. 101
You have some responsibility
10

102. One Way Ticket to the Cloud…

103. One Way Ticket to the Cloud…

104. What can you do?
?

105. Make the Tech World Better

106. Thank you!