Redis CRDTs provide a simple and powerful way for developers to deploy global applications with active-active topologies using a multi-master, bi-directional replication technique.

1. Building Active-Active Geo-Distributed Apps with
Redis using CRDTs (conflict free replicated data-types)
DEC 2017
Cihan Biyikoglu
VP. Product Management at Redis Labs

2. • Quick Intro to Redis and Redis Enterprise
• Redis Enterprise Replication Architecture
• Redis CRDTs
–Introductions to Redis CRDTs
• CAP Theorem and Redis CRDTs
• Redis CRDT Use Cases
–Under the hood - Redis CRDTs
–How to develop apps with Redis CRDTs
–Demo! Redis Enterprise 5.0
• Q&A
Agenda

3. Who We Are
Open source. The leading in-memory database platform,
supporting any high performance operational, analytics or
hybrid use case.
The open source home and commercial provider of Redis
Enterprise (Redise) technology, platform, products & services.
3

4. Why Redis?
4
Simplicity ExtensibilityPerformance
Low latency at High Throughput
1
Redis Data Structures
2 3
Redis Modules
Lists
Hashes
Bitmaps
Strings
Bit field
Streams
Hyperloglog
Sorted Sets
Sets
Geospatial Indexes

5. 5
Why Redis Enterprise?
Integrated
ModulesAutomation
& Support
Redise Flash +
More Savings
Performance
at Scale
Built-in
Search
Active-Active
Geo
Distribution
(CRDTs) Robust
Security
HA, Durability,
DR

6. Fast Growing Business
6
DBaaS
• Available since mid 2013
• 7,500+ enterprise customers
Software
• Available since early 2015
• 200+ enterprise customers
550K+ databases
managed worldwide
• 6 of top Fortune 10 companies
• 3 of top 5 communications companies
Customers
• 3 of top 4 credit card issuers
• 3 of top 5 healthcare companies

7. Redis is Uniquely Suited to Modern Apps
A full range of capabilities that simplify and accelerate next generation applications
Graph
Machine
Learning
CachingHigh-Speed
Transactions
Fast Data
Ingest
Time Series Geo-Spatial
Indexing
MessagingAnalytics
Job &
Queue
7
Search

8. Redis Enterprise Architecture
8

9. Redis Enterprise – Cluster Architecture
Cluster
Management
Path
Proxies
Node Watchdog
Cluster Watchdog
Node 1 Node 2 Node N
Redise ShardsData Path
9
Redise Cluster

10. Low Replication Lag & High Replication Throughput
• Local Replication: Built for LAN
• Higher bandwidth
• Lower latency
• High quality links susceptible to fewer failures and retransmits
• Cross-Geo Replication: Built for WAN
• Lower bandwidth
• Higher latency
• “Noisier” network quality susceptible to more failures and retransmits
Replication Architecture with Redis Enterprise

11. • Unidirectional Replication
• Replica Of – Data Movement between Source to Destination DB
• Content Distribution to Various Geographies for low latency, local reads
• Continuous Data Transfer to and from Other Redis or Redise Databases
• Bi-Directional Replication
• Redis CRDTs (Conflict-free Replicated Data Tpes) - Active-Active Writes & Reads
• Advanced Protection against Regional Failures
• Geo-Distributed Apps for Concurrent Active-Active Writes & Reads
Cross-Geo Replication
Geo-Distributed Clusters

12. Introducing Redis CRDTs
(Conflict-free Replicated Data Types)

13. • Consensus Driven Protocols – 2 Phase Commit
– Very chatty over LAN and WAN with very low throughput
– Strictly “Consistent” thus not “Available” by nature
– Products: Relational Databases
• Quorum Based Writes and Reads
– Very chatty over LAN and WAN with very low throughput
– Strictly “Consistent” thus not “Available” by nature
– Products: Cassandra
Active-Active Geo-Distributed Applications Are Hard!
13
Consistent
Available Partition
Tolerance
ConsistentbutNOTAvailable

14. • Consensus Driven Protocols – 2 Phase Commit
– Very chatty over LAN and WAN with very low throughput
– Strictly “Consistent” thus not “Available” by nature
– Products: Relational Databases
• Quorum Based Writes and Reads
– Very chatty over LAN and WAN with very low throughput
– Strictly “Consistent” thus not “Available” by nature
– Products: Cassandra
• LWW - Last Writer Wins Conflict Resolution
– Inaccurate and Insufficient for many apps (see next slide…)
– Products: Couchbase
• MVCC - Multi Version Concurrency Control
– High overhead and slow throughput due to large metadata
– Complex to program against
– Products: CouchDB
Active-Active Geo-Distributed Applications Are Hard!
14
Consistent
Available Partition
Tolerance
ConsistentbutNOTAvailableAvailablebutNOTConsistent

15. • Consistency: Strong eventual consistency
– Consistent: Any two nodes that receive same but unordered updates will be in the same states
• Availability: Based on consensus-free protocol
– Available: Local cluster availability is sufficient
– No expensive multi-step consensus communication
• Smart conflict resolution with complex data types
– Each Redis “type + method” expresses developer intent in resolving conflicts
• Based on proven Mathematical Models
– Eric Brewer “CAP Twelve Years Later: How the "Rules" Have Changed”
– Marc Shapiro “Strong Eventual Consistency and Conflict-free Replicated Data Types”
Introducing CRDTs (Conflict-Free Replicated Data Types)
15
Consistent
Available Partition
Tolerance

16. • Geo-Replicated Active-Active Deployments
• Smart & Transparent Conflict Resolution
• Based on CRDT Technology (conflict free
replicated data types)
Active - Active Geo Distributed Apps with Redis CRDTs
16
Redis App
Redis App
Redis App
Redis CRDTs
Active-Active Geo-Distributed Redis Apps with
Smart Auto-Conflict Resolution and Simplified Development

17. • Geo-Replicated Active-Active Deployments
• Global Database distributed across clusters in
multiple regions
• Fast bi-directional replication across data-centers
• Low latency local reads & writes: sub-millisecond
latencies
Redis CRDTs
Redis App
Redis App
Redis App

18. • Smart & Transparent Conflict Resolution
• CRDBs power concurrent writes across
geographies
• CRDBs handle failover conflicts safely!
Redis CRDTs
time US Data Center EU Data Center
t1 SADD cart1 “costume”
t2 US Data Center Fails – Sync Fails
t3 SADD cart1 “mask”
t4 US Data Center Recovers – Resume Sync
t5 SMEMBERS cart1
“costume”
“mask”
SMEMBERS cart1
“costume”
“mask”
Redis App
Redis App

19. • Smart & Transparent Conflict Resolution
• CRDBs power concurrent writes across
geographies
• CRDBs handle failover conflicts safely!
Redis CRDTs
time US Data Center EU Data Center
t1 SADD cart1 “costume”
t2 US Data Center Fails – Sync Fails
t3 SADD cart1 “mask”
t4 US Data Center Recovers – Resume Sync
t5 SMEMBERS cart1
“costume”
“mask”
SMEMBERS cart1
“costume”
“mask”
Redis App
Redis App

20. • Smart & Transparent Conflict Resolution
• CRDBs power concurrent writes across
geographies
• CRDBs handle failover conflicts safely!
Redis CRDTs
time US Data Center EU Data Center
t1 SADD cart1 “costume”
t2 US Data Center Fails – Sync Fails
t3 SADD cart1 “mask”
t4 US Data Center Recovers – Resume Sync
t5 SMEMBERS cart1
“costume”
“mask”
SMEMBERS cart1
“costume”
“mask”
Redis App
Redis App

21. • Smart & Transparent Conflict Resolution
• CRDBs power concurrent writes across
geographies
• CRDBs handle failover conflicts safely!
Redis CRDTs
time US Data Center EU Data Center
t1 SADD cart1 “costume”
t2 US Data Center Fails – Sync Fails
t3 SADD cart1 “mask”
t4 US Data Center Recovers – Resume Sync
t5 SMEMBERS cart1
“costume”
“mask”
SMEMBERS cart1
“costume”
“mask”
Redis App
Redis App

22. • Smart & Transparent Conflict Resolution
• CRDBs power concurrent writes across
geographies
• CRDBs handle failover conflicts safely!
Redis CRDTs
time US Data Center EU Data Center
t1 SADD cart1 “costume”
t2 US Data Center Fails – Sync Fails
t3 SADD cart1 “mask”
t4 US Data Center Recovers – Resume Sync
t5 SMEMBERS cart1
“costume”
“mask”
SMEMBERS cart1
“costume”
“mask”
Redis App
Redis App

23. • Based on CRDT Technology
• Simple to develop with Redis Commands
• Smarter-conflict resolution based on
“developers intent”
Redis CRDTs
time US Data Center EU Data Center
t1 INCR TxCounter1
t2 INCR TxCounter1
t3 INCR TxCounter1
t4 Sync
t5 GET TxCounter1
3
GET TxCounter1
3
Redis App
Redis App
Redis App

24. • Based on CRDT Technology
• Simple to develop with Redis Commands
• Smarter-conflict resolution based on
“developers intent”
Redis CRDTs
time US Data Center EU Data Center
t1 INCR TxCounter1
t2 INCR TxCounter1
t3 INCR TxCounter1
t4 Sync
t5 GET TxCounter1
3
GET TxCounter1
3
Redis App
Redis App
Redis App

25. • Based on CRDT Technology
• Simple to develop with Redis Commands
• Smarter-conflict resolution based on
“developers intent”
Redis CRDTs
time US Data Center EU Data Center
t1 INCR TxCounter1
t2 INCR TxCounter1
t3 INCR TxCounter1
t4 Sync
t5 GET TxCounter1
3
GET TxCounter1
3
Redis App
Redis App
Redis App

26. • Based on CRDT Technology
• Simple to develop with Redis Commands
• Smarter-conflict resolution based on
“developers intent”
Redis CRDTs
time US Data Center EU Data Center
t1 INCR TxCounter1
t2 INCR TxCounter1
t3 INCR TxCounter1
t4 Sync
t5 GET TxCounter1
3
GET TxCounter1
3
Redis App
Redis App
Redis App

27. • Based on CRDT Technology
• Simple to develop with Redis Commands
• Smarter-conflict resolution based on
“developers intent”
Redis CRDTs
time US Data Center EU Data Center
t1 INCR TxCounter1
t2 INCR TxCounter1
t3 INCR TxCounter1
t4 Sync
t5 GET TxCounter1
3
GET TxCounter1
3
Redis App
Redis App
Redis App

28. How Does Redis CRDTs Work?

29. • CREATING a CRDB
–Set global CRDB options
–Initialize member CRDB on each participating
cluster
• In case of Error, Rollback
–Establish bi-directional replication among all
members
Redis CRDTs == CRDBs (Conflict free replicated databases)
Redis App
Redis App
Redis App

30. Bi-directional Replication
• Syncer uses slave-replicas to replicate operations in a streaming fashion
• Resume-able replication under failures but may resync if a participating
cluster goes stale (long duration of absence by a member)
CRDB Architecture
Compressed Stream
Syncer
Syncer
Participating Cluster 1
Participating Cluster 2

31. How to Develop Applications with CRDBs?

32. • DEVELOPING APPS with a CRDB
–All apps connect to the local member CRDB in the
participating cluster
–Reads/Writes occur on local member CRDB
• Outcome of concurrent writes are predictable and
based on a set of rules
–Bi-directional replication automatically syncs
changes
• No application code required to resolve conflicts
CRDBs (Conflict free replicated databases)
Redis App
Redis App
Redis App

33. • Simply use Redis with Redis Client SDKs!
• Conflict Resolution is smart and automatic
–Counters - using INCR or INCRBY, INCRBYFLOAT and so on
–Sets – SADD/SREM and so on
–Hash – HSET/HDEL and so on
–Strings - using SET/APPEND and so on
• Conflict Resolution is specific to data type + commands used
–Total increments since last sync for Counters
–Add wins with observed remove for Sets
–Last writer wins for Strings
–Counter or String semantics for Hash
–And so on.
Developing Applications with CRDBs
33
Redis App
Redis App
Redis App

34. Demo!REDIS CRDTS IN ACTION!
34

35. Recap

36. Redis Enterprise with Redis CRDTs (Conflict Free Replicated Data-types)
Build Fast Active-Active Geo-Distributed Redis Apps
with Smart Auto-Conflict Resolution
• Geo-Replicated Active-Active Deployments
–Fast bi-directional across data-centers
–Low latency local reads & writes : sub-millisecond latencies
• Smart & Transparent Conflict Resolution
–Build safe regional failover and geo-distributed apps
–Each Redis Data Type have smart conflict resolution based on “developer intent”
• Based on CRDT Technology (conflict free replicated data types)
–Simplifies development
Active - Active Geo Distribution
36

37. Get Started with Redis CRDTs Today!
Redislabs.com
–Download Redis Enterprise 5.0
–Visit Docker Hub: redislabs/redis:latest
37

38. Building Active-Active Geo-Distributed Apps with
Redis using CRDTs (conflict free replicated data-types)
DEC 2017
Cihan Biyikoglu
VP. Products Management at Redis Labs