SlideShare a Scribd company logo
1 of 25
Download to read offline
REPLICATION
IN THE WILD
Ensar Basri Kahveci
Hello! Ensar Basri Kahveci
Distributed Systems Engineer @ Hazelcast
website: basrikahveci.com
linkedin: basrikahveci
twitter & github: metanet
- Leading open source Java IMDG
- Distributed Java collections, JCache, HD store, …
- Distributed computations and messaging
- Embedded or client - server deployment
- Integration modules & cloud friendly
- Highly available, scalable, elastic
REPLICATION
- Putting a data set into
multiple nodes.
- Each replica has a full copy.
- A few reasons for replication:
- Performance
- Availability
REPLICATION + PARTITIONING
- Mostly used with
partitioning.
- Two partitions: P1, P2
- Two replicas for each
partition.
NOTHING FOR FREE!
- Very easy to do when the data is immutable.
- Two main difficulties:
- Handling updates,
- Handling failures.
The dangers of replIcatIon and a solutIon
- Gray et al. [1] classify replication models by 2
parameters:
- Where to make updates: primary copy or update
anywhere
- When to make updates: eagerly or lazily
WHERE: PRIMARY COPY
- There is a single replica
managing the updates.
- No conflicts and no
conflict-handling logic.
- Implies sticky availability.
- When primary fails, a new
primary is elected.
WHERE: UPDATE ANYWHERE
- Each replica can initiate a
transaction to make an update.
- Complex concurrency control.
- Deadlocks or conflicts are
possible.
- In practice, there is also
multi-leader.
WHEN: EAGER REPLICATION
- Synchronously updates all
replicas as part of one atomic
transaction.
- Strong consistency.
- Level of availability can
degrade on node failures.
- Consensus algorithms
WHEN: LAZY REPLICATION
- Updates each replica with a
separate transaction.
- Updates can execute quite fast.
- High availability.
- Data copies can diverge.
WHERE?
WHEN?
PRIMARY COPY UPDATE ANYWHERE
EAGER
2PC [24]
Multi Paxos [5]
etcd and Consul (RAFT) [6]
Zookeeper (Zab) [7]
Kafka
2PC [24]
Paxos [5]
Hazelcast Cluster State Change [12]
MySQL 5.7 Group Replication [23]
LAZY
Hazelcast
MongoDB
ElasticSearch
Redis
Kafka
Dynamo [4]
Cassandra
Riak
Hazelcast Active-Active WAN
Replication [22]
PRIMARY COPY + EAGER REPLICATION
- When the primary fails, secondaries are
guaranteed to be up to date.
- Majority approach in consensus algorithms.
- Expensive. Mostly used for storing metadata.
- In Kafka, in-sync-replica set [11] is maintained.
UPDATE ANYWHERE + EAGER REPLICATION
- Each replica can initiate a new transaction.
- Concurrent transactions can compete with
each other.
- Possibility of races and deadlocks.
- Hazelcast Cluster State Change [12]
PRIMARy COPY + LAZY REPLICATION
- Hazelcast, Redis, ElasticSearch, Kafka ...
- The primary copy can execute updates fast.
- Secondaries can fall behind the primary. It is
called replication lag.
- It can lead to data loss during leader failover, but
still no conflicts.
- Secondaries can be used for reads.
Hazelcast: PRIMARy COPY + LAZY REPLICATION
PRIMARY
COPY
strong consistency
on a stable cluster
sticky availability
LAZY
REPLICATION
high throughput replication log
UPDATE ANYWHERE + LAZY REPLICATION
- Dynamo-style [4] highly available databases.
- Tunable quorums.
- Concurrent updates are first-class citizens.
- Possibility of conflicts
- Avoiding, discarding, detecting & resolving conflicts
- Eventual convergence
- Write repair, read repair and anti-entropy
Tunable QUORUMS
- W + R > N
- W = 3, R = 1, N = 3
- W = 2, R = 1, N = 3
- If W or R are not met
- Sloppy quorums and
hinted handoff
CONCURRENT UPDATES
- Avoiding conflicts: CRDTs [2]
- Strong eventual consistency
- Riak Data Types [3]
- Discarding conflicts: Last write wins
- Physical timestamps in Cassandra [8], [9]
- Detecting conflicts: Vector clocks [10]
- Dynamo-style databases [4]
EVENTUAL CONVERGENCE
- Write repair
- Read repair
- Active Anti-entropy
- Merkle trees
WHERE?
WHEN?
PRIMARY COPY UPDATE ANYWHERE
EAGER
strong consistency
simple concurrency
slow
inflexible
strong consistency
complex concurrency
expensive
deadlocks
LAZY
fast
eventual consistency
simple concurrency
Inconsistency
flexible
high availability
eventual consistency
conflicts
Recap
- We apply replication to make distributed
systems performant, available and fault
tolerant.
- Various replication protocols are built based
on when and where to make updates.
- No silver bullet. It is a world of trade-offs.
- We are hiring!
- Senior Java Developer
http://stackoverflow.com/jobs/129435/senior-java-developer-hazelcast
- Solution Architect
http://stackoverflow.com/jobs/131938/solutions-architect-hazelcast
REFerences
[1] Gray, Jim, et al. "The dangers of replication and a solution." ACM SIGMOD Record 25.2 (1996): 173-182.
[2] Shapiro, Marc, et al. "Conflict-free replicated data types." Symposium on Self-Stabilizing Systems. Springer, Berlin, Heidelberg, 2011.
[3] http://docs.basho.com/riak/kv/2.2.0/learn/concepts/crdts/
[4] DeCandia, Giuseppe, et al. "Dynamo: amazon's highly available key-value store." ACM SIGOPS operating systems review 41.6 (2007): 205-220.
[5] Lamport, Leslie. "Paxos made simple." ACM Sigact News 32.4 (2001): 18-25.
[6] Ongaro, Diego, and John K. Ousterhout. "In Search of an Understandable Consensus Algorithm." USENIX Annual Technical Conference. 2014.
[7] Hunt, Patrick, et al. "ZooKeeper: Wait-free Coordination for Internet-scale Systems." USENIX annual technical conference. Vol. 8. 2010.
[8] http://www.datastax.com/dev/blog/why-cassandra-doesnt-need-vector-clocks
[9] https://aphyr.com/posts/299-the-trouble-with-timestamps
[10] Raynal, Michel, and Mukesh Singhal. "Logical time: Capturing causality in distributed systems." Computer 29.2 (1996): 49-56.
[11] http://kafka.apache.org/documentation.html#replication
[12] http://docs.hazelcast.org/docs/latest/manual/html-single/index.html#managing-cluster-and-member-states
[13] E. Brewer, "Towards Robust Distributed Systems," Proc. 19th Ann. ACM Symp. Principles of Distributed Computing (PODC 00), ACM, 2000, pp. 7-10
[14] https://codahale.com/you-cant-sacrifice-partition-tolerance/
[15] http://blog.nahurst.com/visual-guide-to-nosql-systems
[16] http://www.allthingsdistributed.com/2008/12/eventually_consistent.html
[17] https://www.somethingsimilar.com/2013/01/14/notes-on-distributed-systems-for-young-bloods/
[18] https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed
[19] Gilbert, Seth, and Nancy Lynch. "Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services." Acm Sigact News 33.2 (2002): 51-59.
[20] https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html
[21] https://henryr.github.io/cap-faq/
[22] http://docs.hazelcast.org/docs/3.7/manual/html-single/index.html#wan-replication
[23] https://dev.mysql.com/doc/refman/5.7/en/group-replication.html
[24] Notes on data base operating systems, JN Gray - Operating Systems, 1978
THANKS!Any questions?

More Related Content

What's hot

An Overview of Distributed Debugging
An Overview of Distributed DebuggingAn Overview of Distributed Debugging
An Overview of Distributed DebuggingAnant Narayanan
 
BASE: An Acid Alternative
BASE: An Acid AlternativeBASE: An Acid Alternative
BASE: An Acid AlternativeHiroshi Ono
 
Nondeterminism is unavoidable, but data races are pure evil
Nondeterminism is unavoidable, but data races are pure evilNondeterminism is unavoidable, but data races are pure evil
Nondeterminism is unavoidable, but data races are pure evilracesworkshop
 
Introduction 1
Introduction 1Introduction 1
Introduction 1Yasir Khan
 
DIY: A distributed database cluster, or: MySQL Cluster
DIY: A distributed database cluster, or: MySQL ClusterDIY: A distributed database cluster, or: MySQL Cluster
DIY: A distributed database cluster, or: MySQL ClusterUlf Wendel
 
Queue centric pattern
Queue centric patternQueue centric pattern
Queue centric patternSagar Rao
 
CAP theorem and distributed systems
CAP theorem and distributed systemsCAP theorem and distributed systems
CAP theorem and distributed systemsKlika Tech, Inc
 
Vote NO for MySQL
Vote NO for MySQLVote NO for MySQL
Vote NO for MySQLUlf Wendel
 
Directory based cache coherence
Directory based cache coherenceDirectory based cache coherence
Directory based cache coherenceHoang Nguyen
 
MySQL native driver for PHP (mysqlnd) - Introduction and overview, Edition 2011
MySQL native driver for PHP (mysqlnd) - Introduction and overview, Edition 2011MySQL native driver for PHP (mysqlnd) - Introduction and overview, Edition 2011
MySQL native driver for PHP (mysqlnd) - Introduction and overview, Edition 2011Ulf Wendel
 
Multiprocessing -Interprocessing communication and process sunchronization,se...
Multiprocessing -Interprocessing communication and process sunchronization,se...Multiprocessing -Interprocessing communication and process sunchronization,se...
Multiprocessing -Interprocessing communication and process sunchronization,se...Neena R Krishna
 
MySQL 5.6 Global Transaction Identifier - Use case: Failover
MySQL 5.6 Global Transaction Identifier - Use case: FailoverMySQL 5.6 Global Transaction Identifier - Use case: Failover
MySQL 5.6 Global Transaction Identifier - Use case: FailoverUlf Wendel
 
The mysqlnd replication and load balancing plugin
The mysqlnd replication and load balancing pluginThe mysqlnd replication and load balancing plugin
The mysqlnd replication and load balancing pluginUlf Wendel
 
Mule esb flow processing strategies
Mule esb flow processing strategiesMule esb flow processing strategies
Mule esb flow processing strategieshimajareddys
 
Database replication
Database replicationDatabase replication
Database replicationArslan111
 

What's hot (20)

An Overview of Distributed Debugging
An Overview of Distributed DebuggingAn Overview of Distributed Debugging
An Overview of Distributed Debugging
 
BASE: An Acid Alternative
BASE: An Acid AlternativeBASE: An Acid Alternative
BASE: An Acid Alternative
 
Nondeterminism is unavoidable, but data races are pure evil
Nondeterminism is unavoidable, but data races are pure evilNondeterminism is unavoidable, but data races are pure evil
Nondeterminism is unavoidable, but data races are pure evil
 
Introduction 1
Introduction 1Introduction 1
Introduction 1
 
DIY: A distributed database cluster, or: MySQL Cluster
DIY: A distributed database cluster, or: MySQL ClusterDIY: A distributed database cluster, or: MySQL Cluster
DIY: A distributed database cluster, or: MySQL Cluster
 
Distruted applications
Distruted applicationsDistruted applications
Distruted applications
 
Queue centric pattern
Queue centric patternQueue centric pattern
Queue centric pattern
 
CAP theorem and distributed systems
CAP theorem and distributed systemsCAP theorem and distributed systems
CAP theorem and distributed systems
 
Vote NO for MySQL
Vote NO for MySQLVote NO for MySQL
Vote NO for MySQL
 
Directory based cache coherence
Directory based cache coherenceDirectory based cache coherence
Directory based cache coherence
 
Micro-Services RabbitMQ vs Kafka
Micro-Services RabbitMQ vs KafkaMicro-Services RabbitMQ vs Kafka
Micro-Services RabbitMQ vs Kafka
 
MySQL native driver for PHP (mysqlnd) - Introduction and overview, Edition 2011
MySQL native driver for PHP (mysqlnd) - Introduction and overview, Edition 2011MySQL native driver for PHP (mysqlnd) - Introduction and overview, Edition 2011
MySQL native driver for PHP (mysqlnd) - Introduction and overview, Edition 2011
 
data replication
data replicationdata replication
data replication
 
Multiprocessing -Interprocessing communication and process sunchronization,se...
Multiprocessing -Interprocessing communication and process sunchronization,se...Multiprocessing -Interprocessing communication and process sunchronization,se...
Multiprocessing -Interprocessing communication and process sunchronization,se...
 
MySQL 5.6 Global Transaction Identifier - Use case: Failover
MySQL 5.6 Global Transaction Identifier - Use case: FailoverMySQL 5.6 Global Transaction Identifier - Use case: Failover
MySQL 5.6 Global Transaction Identifier - Use case: Failover
 
Cache coherence
Cache coherenceCache coherence
Cache coherence
 
Replication in Distributed Systems
Replication in Distributed SystemsReplication in Distributed Systems
Replication in Distributed Systems
 
The mysqlnd replication and load balancing plugin
The mysqlnd replication and load balancing pluginThe mysqlnd replication and load balancing plugin
The mysqlnd replication and load balancing plugin
 
Mule esb flow processing strategies
Mule esb flow processing strategiesMule esb flow processing strategies
Mule esb flow processing strategies
 
Database replication
Database replicationDatabase replication
Database replication
 

Similar to Replication in the Wild - Warsaw Cloud Native Meetup - May 2017

Replication in the wild ankara cloud meetup - feb 2017
Replication in the wild   ankara cloud meetup - feb 2017Replication in the wild   ankara cloud meetup - feb 2017
Replication in the wild ankara cloud meetup - feb 2017Onur Dayıbaşı
 
Replication in the wild ankara cloud meetup - feb 2017
Replication in the wild   ankara cloud meetup - feb 2017Replication in the wild   ankara cloud meetup - feb 2017
Replication in the wild ankara cloud meetup - feb 2017AnkaraCloud
 
MySQL 5.7 Fabric: Introduction to High Availability and Sharding
MySQL 5.7 Fabric: Introduction to High Availability and Sharding MySQL 5.7 Fabric: Introduction to High Availability and Sharding
MySQL 5.7 Fabric: Introduction to High Availability and Sharding Ulf Wendel
 
Scaling MySQL -- Swanseacon.co.uk
Scaling MySQL -- Swanseacon.co.uk Scaling MySQL -- Swanseacon.co.uk
Scaling MySQL -- Swanseacon.co.uk Dave Stokes
 
MySQL 5.7 clustering: The developer perspective
MySQL 5.7 clustering: The developer perspectiveMySQL 5.7 clustering: The developer perspective
MySQL 5.7 clustering: The developer perspectiveUlf Wendel
 
Bhupeshbansal bigdata
Bhupeshbansal bigdata Bhupeshbansal bigdata
Bhupeshbansal bigdata Bhupesh Bansal
 
MySQL Group Replication
MySQL Group ReplicationMySQL Group Replication
MySQL Group ReplicationUlf Wendel
 
Clustered Architecture Patterns Delivering Scalability And Availability
Clustered Architecture Patterns Delivering Scalability And AvailabilityClustered Architecture Patterns Delivering Scalability And Availability
Clustered Architecture Patterns Delivering Scalability And AvailabilityConSanFrancisco123
 
Data Engineering for Data Scientists
Data Engineering for Data Scientists Data Engineering for Data Scientists
Data Engineering for Data Scientists jlacefie
 
Planning to Fail #phpuk13
Planning to Fail #phpuk13Planning to Fail #phpuk13
Planning to Fail #phpuk13Dave Gardner
 
MySQL InnoDB Cluster - New Features in 8.0 Releases - Best Practices
MySQL InnoDB Cluster - New Features in 8.0 Releases - Best PracticesMySQL InnoDB Cluster - New Features in 8.0 Releases - Best Practices
MySQL InnoDB Cluster - New Features in 8.0 Releases - Best PracticesKenny Gryp
 
Big Data Streams Architectures. Why? What? How?
Big Data Streams Architectures. Why? What? How?Big Data Streams Architectures. Why? What? How?
Big Data Streams Architectures. Why? What? How?Anton Nazaruk
 
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCScalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCCal Henderson
 
MySQL InnoDB Cluster HA Overview & Demo
MySQL InnoDB Cluster HA Overview & DemoMySQL InnoDB Cluster HA Overview & Demo
MySQL InnoDB Cluster HA Overview & DemoKeith Hollman
 
PoC: Using a Group Communication System to improve MySQL Replication HA
PoC: Using a Group Communication System to improve MySQL Replication HAPoC: Using a Group Communication System to improve MySQL Replication HA
PoC: Using a Group Communication System to improve MySQL Replication HAUlf Wendel
 
Talon systems - Distributed multi master replication strategy
Talon systems - Distributed multi master replication strategyTalon systems - Distributed multi master replication strategy
Talon systems - Distributed multi master replication strategySaptarshi Chatterjee
 
Mysql User Camp : 20th June - Mysql New Features
Mysql User Camp : 20th June - Mysql New FeaturesMysql User Camp : 20th June - Mysql New Features
Mysql User Camp : 20th June - Mysql New FeaturesTarique Saleem
 
Mysql User Camp : 20-June-14 : Mysql New features and NoSQL Support
 Mysql User Camp : 20-June-14 : Mysql New features and NoSQL Support Mysql User Camp : 20-June-14 : Mysql New features and NoSQL Support
Mysql User Camp : 20-June-14 : Mysql New features and NoSQL SupportMysql User Camp
 
MySQL High Availability Solutions - Avoid loss of service by reducing the r...
MySQL High Availability Solutions  -  Avoid loss of service by reducing the r...MySQL High Availability Solutions  -  Avoid loss of service by reducing the r...
MySQL High Availability Solutions - Avoid loss of service by reducing the r...Olivier DASINI
 

Similar to Replication in the Wild - Warsaw Cloud Native Meetup - May 2017 (20)

Replication in the wild ankara cloud meetup - feb 2017
Replication in the wild   ankara cloud meetup - feb 2017Replication in the wild   ankara cloud meetup - feb 2017
Replication in the wild ankara cloud meetup - feb 2017
 
Replication in the wild ankara cloud meetup - feb 2017
Replication in the wild   ankara cloud meetup - feb 2017Replication in the wild   ankara cloud meetup - feb 2017
Replication in the wild ankara cloud meetup - feb 2017
 
MySQL 5.7 Fabric: Introduction to High Availability and Sharding
MySQL 5.7 Fabric: Introduction to High Availability and Sharding MySQL 5.7 Fabric: Introduction to High Availability and Sharding
MySQL 5.7 Fabric: Introduction to High Availability and Sharding
 
Scaling MySQL -- Swanseacon.co.uk
Scaling MySQL -- Swanseacon.co.uk Scaling MySQL -- Swanseacon.co.uk
Scaling MySQL -- Swanseacon.co.uk
 
MySQL 5.7 clustering: The developer perspective
MySQL 5.7 clustering: The developer perspectiveMySQL 5.7 clustering: The developer perspective
MySQL 5.7 clustering: The developer perspective
 
Bhupeshbansal bigdata
Bhupeshbansal bigdata Bhupeshbansal bigdata
Bhupeshbansal bigdata
 
MySQL Group Replication
MySQL Group ReplicationMySQL Group Replication
MySQL Group Replication
 
Clustered Architecture Patterns Delivering Scalability And Availability
Clustered Architecture Patterns Delivering Scalability And AvailabilityClustered Architecture Patterns Delivering Scalability And Availability
Clustered Architecture Patterns Delivering Scalability And Availability
 
Data Engineering for Data Scientists
Data Engineering for Data Scientists Data Engineering for Data Scientists
Data Engineering for Data Scientists
 
Planning to Fail #phpuk13
Planning to Fail #phpuk13Planning to Fail #phpuk13
Planning to Fail #phpuk13
 
MySQL InnoDB Cluster - New Features in 8.0 Releases - Best Practices
MySQL InnoDB Cluster - New Features in 8.0 Releases - Best PracticesMySQL InnoDB Cluster - New Features in 8.0 Releases - Best Practices
MySQL InnoDB Cluster - New Features in 8.0 Releases - Best Practices
 
Big Data Streams Architectures. Why? What? How?
Big Data Streams Architectures. Why? What? How?Big Data Streams Architectures. Why? What? How?
Big Data Streams Architectures. Why? What? How?
 
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCScalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
 
MySQL InnoDB Cluster HA Overview & Demo
MySQL InnoDB Cluster HA Overview & DemoMySQL InnoDB Cluster HA Overview & Demo
MySQL InnoDB Cluster HA Overview & Demo
 
PoC: Using a Group Communication System to improve MySQL Replication HA
PoC: Using a Group Communication System to improve MySQL Replication HAPoC: Using a Group Communication System to improve MySQL Replication HA
PoC: Using a Group Communication System to improve MySQL Replication HA
 
Galera webinar migration to galera cluster from my sql async replication
Galera webinar migration to galera cluster from my sql async replicationGalera webinar migration to galera cluster from my sql async replication
Galera webinar migration to galera cluster from my sql async replication
 
Talon systems - Distributed multi master replication strategy
Talon systems - Distributed multi master replication strategyTalon systems - Distributed multi master replication strategy
Talon systems - Distributed multi master replication strategy
 
Mysql User Camp : 20th June - Mysql New Features
Mysql User Camp : 20th June - Mysql New FeaturesMysql User Camp : 20th June - Mysql New Features
Mysql User Camp : 20th June - Mysql New Features
 
Mysql User Camp : 20-June-14 : Mysql New features and NoSQL Support
 Mysql User Camp : 20-June-14 : Mysql New features and NoSQL Support Mysql User Camp : 20-June-14 : Mysql New features and NoSQL Support
Mysql User Camp : 20-June-14 : Mysql New features and NoSQL Support
 
MySQL High Availability Solutions - Avoid loss of service by reducing the r...
MySQL High Availability Solutions  -  Avoid loss of service by reducing the r...MySQL High Availability Solutions  -  Avoid loss of service by reducing the r...
MySQL High Availability Solutions - Avoid loss of service by reducing the r...
 

More from Ensar Basri Kahveci

java.util.concurrent for Distributed Coordination - Berlin Expert Days 2019
java.util.concurrent for Distributed Coordination - Berlin Expert Days 2019java.util.concurrent for Distributed Coordination - Berlin Expert Days 2019
java.util.concurrent for Distributed Coordination - Berlin Expert Days 2019Ensar Basri Kahveci
 
java.util.concurrent for Distributed Coordination, Riga DevDays 2019
java.util.concurrent for Distributed Coordination, Riga DevDays 2019java.util.concurrent for Distributed Coordination, Riga DevDays 2019
java.util.concurrent for Distributed Coordination, Riga DevDays 2019Ensar Basri Kahveci
 
java.util.concurrent for Distributed Coordination, GeeCON Krakow 2019
java.util.concurrent for Distributed Coordination, GeeCON Krakow 2019java.util.concurrent for Distributed Coordination, GeeCON Krakow 2019
java.util.concurrent for Distributed Coordination, GeeCON Krakow 2019Ensar Basri Kahveci
 
java.util.concurrent for Distributed Coordination, JEEConf 2019
java.util.concurrent for Distributed Coordination, JEEConf 2019java.util.concurrent for Distributed Coordination, JEEConf 2019
java.util.concurrent for Distributed Coordination, JEEConf 2019Ensar Basri Kahveci
 
Replication Distilled: Hazelcast Deep Dive @ In-Memory Computing Summit San F...
Replication Distilled: Hazelcast Deep Dive @ In-Memory Computing Summit San F...Replication Distilled: Hazelcast Deep Dive @ In-Memory Computing Summit San F...
Replication Distilled: Hazelcast Deep Dive @ In-Memory Computing Summit San F...Ensar Basri Kahveci
 
Replication Distilled: Hazelcast Deep Dive - Berlin Expert Days 2018
Replication Distilled: Hazelcast Deep Dive - Berlin Expert Days 2018Replication Distilled: Hazelcast Deep Dive - Berlin Expert Days 2018
Replication Distilled: Hazelcast Deep Dive - Berlin Expert Days 2018Ensar Basri Kahveci
 
From AP to CP and Back: The Curious Case of Hazelcast (jdk.io 2018)
From AP to CP and Back: The Curious Case of Hazelcast (jdk.io 2018)From AP to CP and Back: The Curious Case of Hazelcast (jdk.io 2018)
From AP to CP and Back: The Curious Case of Hazelcast (jdk.io 2018)Ensar Basri Kahveci
 
Distributed Systems Theory for Mere Mortals - Software Craftsmanship Turkey
Distributed Systems Theory for Mere Mortals - Software Craftsmanship TurkeyDistributed Systems Theory for Mere Mortals - Software Craftsmanship Turkey
Distributed Systems Theory for Mere Mortals - Software Craftsmanship TurkeyEnsar Basri Kahveci
 
Distributed Systems Theory for Mere Mortals - Topconf Dusseldorf October 2017
Distributed Systems Theory for Mere Mortals - Topconf Dusseldorf October 2017Distributed Systems Theory for Mere Mortals - Topconf Dusseldorf October 2017
Distributed Systems Theory for Mere Mortals - Topconf Dusseldorf October 2017Ensar Basri Kahveci
 
Ankara Jug - Practical Functional Programming with Scala
Ankara Jug - Practical Functional Programming with ScalaAnkara Jug - Practical Functional Programming with Scala
Ankara Jug - Practical Functional Programming with ScalaEnsar Basri Kahveci
 

More from Ensar Basri Kahveci (10)

java.util.concurrent for Distributed Coordination - Berlin Expert Days 2019
java.util.concurrent for Distributed Coordination - Berlin Expert Days 2019java.util.concurrent for Distributed Coordination - Berlin Expert Days 2019
java.util.concurrent for Distributed Coordination - Berlin Expert Days 2019
 
java.util.concurrent for Distributed Coordination, Riga DevDays 2019
java.util.concurrent for Distributed Coordination, Riga DevDays 2019java.util.concurrent for Distributed Coordination, Riga DevDays 2019
java.util.concurrent for Distributed Coordination, Riga DevDays 2019
 
java.util.concurrent for Distributed Coordination, GeeCON Krakow 2019
java.util.concurrent for Distributed Coordination, GeeCON Krakow 2019java.util.concurrent for Distributed Coordination, GeeCON Krakow 2019
java.util.concurrent for Distributed Coordination, GeeCON Krakow 2019
 
java.util.concurrent for Distributed Coordination, JEEConf 2019
java.util.concurrent for Distributed Coordination, JEEConf 2019java.util.concurrent for Distributed Coordination, JEEConf 2019
java.util.concurrent for Distributed Coordination, JEEConf 2019
 
Replication Distilled: Hazelcast Deep Dive @ In-Memory Computing Summit San F...
Replication Distilled: Hazelcast Deep Dive @ In-Memory Computing Summit San F...Replication Distilled: Hazelcast Deep Dive @ In-Memory Computing Summit San F...
Replication Distilled: Hazelcast Deep Dive @ In-Memory Computing Summit San F...
 
Replication Distilled: Hazelcast Deep Dive - Berlin Expert Days 2018
Replication Distilled: Hazelcast Deep Dive - Berlin Expert Days 2018Replication Distilled: Hazelcast Deep Dive - Berlin Expert Days 2018
Replication Distilled: Hazelcast Deep Dive - Berlin Expert Days 2018
 
From AP to CP and Back: The Curious Case of Hazelcast (jdk.io 2018)
From AP to CP and Back: The Curious Case of Hazelcast (jdk.io 2018)From AP to CP and Back: The Curious Case of Hazelcast (jdk.io 2018)
From AP to CP and Back: The Curious Case of Hazelcast (jdk.io 2018)
 
Distributed Systems Theory for Mere Mortals - Software Craftsmanship Turkey
Distributed Systems Theory for Mere Mortals - Software Craftsmanship TurkeyDistributed Systems Theory for Mere Mortals - Software Craftsmanship Turkey
Distributed Systems Theory for Mere Mortals - Software Craftsmanship Turkey
 
Distributed Systems Theory for Mere Mortals - Topconf Dusseldorf October 2017
Distributed Systems Theory for Mere Mortals - Topconf Dusseldorf October 2017Distributed Systems Theory for Mere Mortals - Topconf Dusseldorf October 2017
Distributed Systems Theory for Mere Mortals - Topconf Dusseldorf October 2017
 
Ankara Jug - Practical Functional Programming with Scala
Ankara Jug - Practical Functional Programming with ScalaAnkara Jug - Practical Functional Programming with Scala
Ankara Jug - Practical Functional Programming with Scala
 

Recently uploaded

System Simulation and Modelling with types and Event Scheduling
System Simulation and Modelling with types and Event SchedulingSystem Simulation and Modelling with types and Event Scheduling
System Simulation and Modelling with types and Event SchedulingBootNeck1
 
Mine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptxMine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptxRomil Mishra
 
Artificial Intelligence in Power System overview
Artificial Intelligence in Power System overviewArtificial Intelligence in Power System overview
Artificial Intelligence in Power System overviewsandhya757531
 
DEVICE DRIVERS AND INTERRUPTS SERVICE MECHANISM.pdf
DEVICE DRIVERS AND INTERRUPTS  SERVICE MECHANISM.pdfDEVICE DRIVERS AND INTERRUPTS  SERVICE MECHANISM.pdf
DEVICE DRIVERS AND INTERRUPTS SERVICE MECHANISM.pdfAkritiPradhan2
 
Input Output Management in Operating System
Input Output Management in Operating SystemInput Output Management in Operating System
Input Output Management in Operating SystemRashmi Bhat
 
Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Romil Mishra
 
signals in triangulation .. ...Surveying
signals in triangulation .. ...Surveyingsignals in triangulation .. ...Surveying
signals in triangulation .. ...Surveyingsapna80328
 
US Department of Education FAFSA Week of Action
US Department of Education FAFSA Week of ActionUS Department of Education FAFSA Week of Action
US Department of Education FAFSA Week of ActionMebane Rash
 
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTIONTHE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTIONjhunlian
 
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catchers
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor CatchersTechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catchers
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catcherssdickerson1
 
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENT
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENTFUNCTIONAL AND NON FUNCTIONAL REQUIREMENT
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENTSneha Padhiar
 
SOFTWARE ESTIMATION COCOMO AND FP CALCULATION
SOFTWARE ESTIMATION COCOMO AND FP CALCULATIONSOFTWARE ESTIMATION COCOMO AND FP CALCULATION
SOFTWARE ESTIMATION COCOMO AND FP CALCULATIONSneha Padhiar
 
Engineering Drawing section of solid
Engineering Drawing     section of solidEngineering Drawing     section of solid
Engineering Drawing section of solidnamansinghjarodiya
 
TEST CASE GENERATION GENERATION BLOCK BOX APPROACH
TEST CASE GENERATION GENERATION BLOCK BOX APPROACHTEST CASE GENERATION GENERATION BLOCK BOX APPROACH
TEST CASE GENERATION GENERATION BLOCK BOX APPROACHSneha Padhiar
 
Katarzyna Lipka-Sidor - BIM School Course
Katarzyna Lipka-Sidor - BIM School CourseKatarzyna Lipka-Sidor - BIM School Course
Katarzyna Lipka-Sidor - BIM School Coursebim.edu.pl
 
ROBOETHICS-CCS345 ETHICS AND ARTIFICIAL INTELLIGENCE.ppt
ROBOETHICS-CCS345 ETHICS AND ARTIFICIAL INTELLIGENCE.pptROBOETHICS-CCS345 ETHICS AND ARTIFICIAL INTELLIGENCE.ppt
ROBOETHICS-CCS345 ETHICS AND ARTIFICIAL INTELLIGENCE.pptJohnWilliam111370
 
Paper Tube : Shigeru Ban projects and Case Study of Cardboard Cathedral .pdf
Paper Tube : Shigeru Ban projects and Case Study of Cardboard Cathedral .pdfPaper Tube : Shigeru Ban projects and Case Study of Cardboard Cathedral .pdf
Paper Tube : Shigeru Ban projects and Case Study of Cardboard Cathedral .pdfNainaShrivastava14
 
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSHigh Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSsandhya757531
 
11. Properties of Liquid Fuels in Energy Engineering.pdf
11. Properties of Liquid Fuels in Energy Engineering.pdf11. Properties of Liquid Fuels in Energy Engineering.pdf
11. Properties of Liquid Fuels in Energy Engineering.pdfHafizMudaserAhmad
 
Levelling - Rise and fall - Height of instrument method
Levelling - Rise and fall - Height of instrument methodLevelling - Rise and fall - Height of instrument method
Levelling - Rise and fall - Height of instrument methodManicka Mamallan Andavar
 

Recently uploaded (20)

System Simulation and Modelling with types and Event Scheduling
System Simulation and Modelling with types and Event SchedulingSystem Simulation and Modelling with types and Event Scheduling
System Simulation and Modelling with types and Event Scheduling
 
Mine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptxMine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptx
 
Artificial Intelligence in Power System overview
Artificial Intelligence in Power System overviewArtificial Intelligence in Power System overview
Artificial Intelligence in Power System overview
 
DEVICE DRIVERS AND INTERRUPTS SERVICE MECHANISM.pdf
DEVICE DRIVERS AND INTERRUPTS  SERVICE MECHANISM.pdfDEVICE DRIVERS AND INTERRUPTS  SERVICE MECHANISM.pdf
DEVICE DRIVERS AND INTERRUPTS SERVICE MECHANISM.pdf
 
Input Output Management in Operating System
Input Output Management in Operating SystemInput Output Management in Operating System
Input Output Management in Operating System
 
Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________
 
signals in triangulation .. ...Surveying
signals in triangulation .. ...Surveyingsignals in triangulation .. ...Surveying
signals in triangulation .. ...Surveying
 
US Department of Education FAFSA Week of Action
US Department of Education FAFSA Week of ActionUS Department of Education FAFSA Week of Action
US Department of Education FAFSA Week of Action
 
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTIONTHE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
 
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catchers
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor CatchersTechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catchers
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catchers
 
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENT
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENTFUNCTIONAL AND NON FUNCTIONAL REQUIREMENT
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENT
 
SOFTWARE ESTIMATION COCOMO AND FP CALCULATION
SOFTWARE ESTIMATION COCOMO AND FP CALCULATIONSOFTWARE ESTIMATION COCOMO AND FP CALCULATION
SOFTWARE ESTIMATION COCOMO AND FP CALCULATION
 
Engineering Drawing section of solid
Engineering Drawing     section of solidEngineering Drawing     section of solid
Engineering Drawing section of solid
 
TEST CASE GENERATION GENERATION BLOCK BOX APPROACH
TEST CASE GENERATION GENERATION BLOCK BOX APPROACHTEST CASE GENERATION GENERATION BLOCK BOX APPROACH
TEST CASE GENERATION GENERATION BLOCK BOX APPROACH
 
Katarzyna Lipka-Sidor - BIM School Course
Katarzyna Lipka-Sidor - BIM School CourseKatarzyna Lipka-Sidor - BIM School Course
Katarzyna Lipka-Sidor - BIM School Course
 
ROBOETHICS-CCS345 ETHICS AND ARTIFICIAL INTELLIGENCE.ppt
ROBOETHICS-CCS345 ETHICS AND ARTIFICIAL INTELLIGENCE.pptROBOETHICS-CCS345 ETHICS AND ARTIFICIAL INTELLIGENCE.ppt
ROBOETHICS-CCS345 ETHICS AND ARTIFICIAL INTELLIGENCE.ppt
 
Paper Tube : Shigeru Ban projects and Case Study of Cardboard Cathedral .pdf
Paper Tube : Shigeru Ban projects and Case Study of Cardboard Cathedral .pdfPaper Tube : Shigeru Ban projects and Case Study of Cardboard Cathedral .pdf
Paper Tube : Shigeru Ban projects and Case Study of Cardboard Cathedral .pdf
 
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSHigh Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
 
11. Properties of Liquid Fuels in Energy Engineering.pdf
11. Properties of Liquid Fuels in Energy Engineering.pdf11. Properties of Liquid Fuels in Energy Engineering.pdf
11. Properties of Liquid Fuels in Energy Engineering.pdf
 
Levelling - Rise and fall - Height of instrument method
Levelling - Rise and fall - Height of instrument methodLevelling - Rise and fall - Height of instrument method
Levelling - Rise and fall - Height of instrument method
 

Replication in the Wild - Warsaw Cloud Native Meetup - May 2017

  • 2. Hello! Ensar Basri Kahveci Distributed Systems Engineer @ Hazelcast website: basrikahveci.com linkedin: basrikahveci twitter & github: metanet
  • 3. - Leading open source Java IMDG - Distributed Java collections, JCache, HD store, … - Distributed computations and messaging - Embedded or client - server deployment - Integration modules & cloud friendly - Highly available, scalable, elastic
  • 4. REPLICATION - Putting a data set into multiple nodes. - Each replica has a full copy. - A few reasons for replication: - Performance - Availability
  • 5. REPLICATION + PARTITIONING - Mostly used with partitioning. - Two partitions: P1, P2 - Two replicas for each partition.
  • 6. NOTHING FOR FREE! - Very easy to do when the data is immutable. - Two main difficulties: - Handling updates, - Handling failures.
  • 7. The dangers of replIcatIon and a solutIon - Gray et al. [1] classify replication models by 2 parameters: - Where to make updates: primary copy or update anywhere - When to make updates: eagerly or lazily
  • 8. WHERE: PRIMARY COPY - There is a single replica managing the updates. - No conflicts and no conflict-handling logic. - Implies sticky availability. - When primary fails, a new primary is elected.
  • 9. WHERE: UPDATE ANYWHERE - Each replica can initiate a transaction to make an update. - Complex concurrency control. - Deadlocks or conflicts are possible. - In practice, there is also multi-leader.
  • 10. WHEN: EAGER REPLICATION - Synchronously updates all replicas as part of one atomic transaction. - Strong consistency. - Level of availability can degrade on node failures. - Consensus algorithms
  • 11. WHEN: LAZY REPLICATION - Updates each replica with a separate transaction. - Updates can execute quite fast. - High availability. - Data copies can diverge.
  • 12. WHERE? WHEN? PRIMARY COPY UPDATE ANYWHERE EAGER 2PC [24] Multi Paxos [5] etcd and Consul (RAFT) [6] Zookeeper (Zab) [7] Kafka 2PC [24] Paxos [5] Hazelcast Cluster State Change [12] MySQL 5.7 Group Replication [23] LAZY Hazelcast MongoDB ElasticSearch Redis Kafka Dynamo [4] Cassandra Riak Hazelcast Active-Active WAN Replication [22]
  • 13. PRIMARY COPY + EAGER REPLICATION - When the primary fails, secondaries are guaranteed to be up to date. - Majority approach in consensus algorithms. - Expensive. Mostly used for storing metadata. - In Kafka, in-sync-replica set [11] is maintained.
  • 14. UPDATE ANYWHERE + EAGER REPLICATION - Each replica can initiate a new transaction. - Concurrent transactions can compete with each other. - Possibility of races and deadlocks. - Hazelcast Cluster State Change [12]
  • 15. PRIMARy COPY + LAZY REPLICATION - Hazelcast, Redis, ElasticSearch, Kafka ... - The primary copy can execute updates fast. - Secondaries can fall behind the primary. It is called replication lag. - It can lead to data loss during leader failover, but still no conflicts. - Secondaries can be used for reads.
  • 16. Hazelcast: PRIMARy COPY + LAZY REPLICATION PRIMARY COPY strong consistency on a stable cluster sticky availability LAZY REPLICATION high throughput replication log
  • 17. UPDATE ANYWHERE + LAZY REPLICATION - Dynamo-style [4] highly available databases. - Tunable quorums. - Concurrent updates are first-class citizens. - Possibility of conflicts - Avoiding, discarding, detecting & resolving conflicts - Eventual convergence - Write repair, read repair and anti-entropy
  • 18. Tunable QUORUMS - W + R > N - W = 3, R = 1, N = 3 - W = 2, R = 1, N = 3 - If W or R are not met - Sloppy quorums and hinted handoff
  • 19. CONCURRENT UPDATES - Avoiding conflicts: CRDTs [2] - Strong eventual consistency - Riak Data Types [3] - Discarding conflicts: Last write wins - Physical timestamps in Cassandra [8], [9] - Detecting conflicts: Vector clocks [10] - Dynamo-style databases [4]
  • 20. EVENTUAL CONVERGENCE - Write repair - Read repair - Active Anti-entropy - Merkle trees
  • 21. WHERE? WHEN? PRIMARY COPY UPDATE ANYWHERE EAGER strong consistency simple concurrency slow inflexible strong consistency complex concurrency expensive deadlocks LAZY fast eventual consistency simple concurrency Inconsistency flexible high availability eventual consistency conflicts
  • 22. Recap - We apply replication to make distributed systems performant, available and fault tolerant. - Various replication protocols are built based on when and where to make updates. - No silver bullet. It is a world of trade-offs.
  • 23. - We are hiring! - Senior Java Developer http://stackoverflow.com/jobs/129435/senior-java-developer-hazelcast - Solution Architect http://stackoverflow.com/jobs/131938/solutions-architect-hazelcast
  • 24. REFerences [1] Gray, Jim, et al. "The dangers of replication and a solution." ACM SIGMOD Record 25.2 (1996): 173-182. [2] Shapiro, Marc, et al. "Conflict-free replicated data types." Symposium on Self-Stabilizing Systems. Springer, Berlin, Heidelberg, 2011. [3] http://docs.basho.com/riak/kv/2.2.0/learn/concepts/crdts/ [4] DeCandia, Giuseppe, et al. "Dynamo: amazon's highly available key-value store." ACM SIGOPS operating systems review 41.6 (2007): 205-220. [5] Lamport, Leslie. "Paxos made simple." ACM Sigact News 32.4 (2001): 18-25. [6] Ongaro, Diego, and John K. Ousterhout. "In Search of an Understandable Consensus Algorithm." USENIX Annual Technical Conference. 2014. [7] Hunt, Patrick, et al. "ZooKeeper: Wait-free Coordination for Internet-scale Systems." USENIX annual technical conference. Vol. 8. 2010. [8] http://www.datastax.com/dev/blog/why-cassandra-doesnt-need-vector-clocks [9] https://aphyr.com/posts/299-the-trouble-with-timestamps [10] Raynal, Michel, and Mukesh Singhal. "Logical time: Capturing causality in distributed systems." Computer 29.2 (1996): 49-56. [11] http://kafka.apache.org/documentation.html#replication [12] http://docs.hazelcast.org/docs/latest/manual/html-single/index.html#managing-cluster-and-member-states [13] E. Brewer, "Towards Robust Distributed Systems," Proc. 19th Ann. ACM Symp. Principles of Distributed Computing (PODC 00), ACM, 2000, pp. 7-10 [14] https://codahale.com/you-cant-sacrifice-partition-tolerance/ [15] http://blog.nahurst.com/visual-guide-to-nosql-systems [16] http://www.allthingsdistributed.com/2008/12/eventually_consistent.html [17] https://www.somethingsimilar.com/2013/01/14/notes-on-distributed-systems-for-young-bloods/ [18] https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed [19] Gilbert, Seth, and Nancy Lynch. "Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services." Acm Sigact News 33.2 (2002): 51-59. [20] https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html [21] https://henryr.github.io/cap-faq/ [22] http://docs.hazelcast.org/docs/3.7/manual/html-single/index.html#wan-replication [23] https://dev.mysql.com/doc/refman/5.7/en/group-replication.html [24] Notes on data base operating systems, JN Gray - Operating Systems, 1978