Replication Distilled: Hazelcast Deep Dive @ In-Memory Computing Summit San Francisco 2018

•

1 like•191 views

This document summarizes Hazelcast's replication protocol. It uses primary copy replication where operations are sent to primary replicas for strong consistency when reachable. It partitions data to scale primaries, updates replicas asynchronously, and uses anti-entropy to sync backups. During network partitions, consistency can be lost and divergence must be resolved upon reconnection using merge policies. The protocol is classified as PA/EC under the PACELC principle, favoring availability over consistency, though failure detectors and CP data structures allow it to behave more consistently in some cases.

Engineering

Replication Distilled:
Hazelcast Deep Dive
Ensar Basri Kahveci
Hazelcast

Hazelcast
▪ The leading open source Java IMDG
▪ Distributed Java collections, concurrency primitives, ...
▪ Distributed computations, messaging, ...

In-Memory Data Grids
▪ Distributed caching
▪ Keeping data in local JVM for fast access & processing
▪ Elasticity, availability, high throughput, and low latency
▪ Multiple copies of data to tolerate failures

Replication
▪ Putting a data set into multiple nodes
▪ Fault tolerance
▪ Latency
▪ Throughput

Challenges
▪ Where to perform reads & writes?
▪ How to keep replicas sync?
▪ How to handle concurrent reads & writes?
▪ How to handle failures?

CAP Principle
▪ Pick two of C, A, and P
▪ CP versus AP

PACELC Principle
▪ If there is a network partition (P), we have to choose
between availability and consistency (AC).
▪ Else (E), during normal operation, we can choose
between latency and consistency (LC).

Let’s build
the core replication protocol
of Hazelcast

Primary Copy
▪ Operations are sent to primary replicas.
▪ Strong consistency when the primary is reachable.

Partitioning (Sharding)
▪ Partitioning helps to scale primaries.
▪ A primary replica is elected for each partition.

Updating Replicas
partition id = hash(serialize(key)) % partition count

Async Replication
▪ Each replica is updated separately.
▪ High throughput and availability

Anti-Entropy
▪ Backup replicas can fall behind the primary.
▪ Non-sync backups are fixed with an active anti-entropy
mechanism.

Replicas are not sync
▪ The client reads a key from the current primary replica.

Network Partitioning
▪ The client reads the same key.

Split-Brain
▪ Strong consistency is lost.

Resolving the Divergence
▪ Merge policies: higher hits, latest update / access, …
▪ Merging may cause lost updates.

Let’s classify this protocol
with PACELC

Hazelcast is PA/EC
▪ Consistency is usually traded to availability and latency
together.
▪ Hazelcast works in memory and mostly used in a single
computing cluster.
▪ Consistency - latency trade-off is minimal.
▪ PA/EC works fine for distributed caching.

Scaling Reads
▪ Reads can be served locally from near caches and
backup replicas.

Failure Detectors
▪ Local failure detectors rely on timeouts.
▪ Operations are blocked after the cluster size falls below
a threshold.

Failure Detectors
▪ It takes some time to detect an unresponsive node.
▪ Minimizes divergence and maintains the baseline
consistency.

Isolated Failure Detectors
▪ Configure failure detectors independently for data
structures
▪ Phi-Accrual Failure Detector

CP Data Structures
▪ IDGenerator
▪ Distributed impls of java.util.concurrent.*
▪ PA/EC is not the perfect fit for CP data structures.

Flake IDs
▪ Local unique id generation
▪ Nodes get a unique node id during join.
▪ K-ordered IDs

CRDTs
▪ CRDTs: Conflict-free Replicated Data Types
▪ Replicas are updated concurrently without coordination.
▪ Strong eventual consistency
▪ Counters, sets, maps, graphs, ...

Sync Replication
▪ Concurrency primitives imply the true CP behavior.
▪ Paxos, Raft, ZAB, VR
▪ Re-implementing Hazelcast concurrency primitives with
Raft

Recap
▪ http://bit.ly/hazelcast-replication-consistency
▪ http://bit.ly/hazelcast-network-partitions
▪ http://dbmsmusings.blogspot.com/2017/10/hazelcast-an
d-mythical-paec-system.html

Thanks!
You can find me at
▪ @metanet
▪ ebkahveci@gmail.com

Distributed data stores cope with the challenges of managing replicated data. Building a distributed data store is hard, and clarifying its high level properties is even harder. Hazelcast makes use of several replication techniques to offer choices for the trade-offs between consistency, availability, and latency properties. In this talk, we take a closer look at these techniques from the perspective of the CAP and PACELC principles. We first break apart the main replication technique and challenge the rationale behind it. Then, we discuss what has gone wrong with it, and how we gradually extended and replaced it with other techniques to fix the issues.

Designing for concurrency and performance

Knoldus Inc.

Distributed Caching Using the JCACHE API and ehcache, Including a Case Study ...elliando dias

Web session replication with Hazelcast

Emrah Kocaman

Session Handling Using Memcache

Anand Ghaywankar

StarWind Virtual SAN Overview

StarWind Software

StarWind Virtual SAN is entirely software-based, hypervisor-centric virtual machine storage. It creates a fully fault-tolerant and high-performing storage pool that is built for the virtualization workload “from scratch”. StarWind Virtual SAN basically “mirrors” inexpensive internal storage between hosts. Virtual SAN completely eliminates any need for an expensive SAN or NAS or other physical shared storage. It seamlessly integrates into the hypervisor for unbeatable performance and exceptional simplicity of use.

Caching technology comparison

Rohit Kelapure

NoSQL databases like Cassandra and Couchbase are quickly becoming key components of the modern IT infrastructure. But this modernization creates new challenges – especially for storage. Storage in the broad sense. In-memory databases perform well when there is enough memory available. However, when data sets get too large and they need to access storage, application performance degrades dramatically. Moreover, even if enough memory is available, persistent client requests can bring the servers to their knees. Join Storage Switzerland and Plexistor where you will learn: 1. What is Cassandra and Couchbase? 2. Why organizations are adopting them? 3. What are the storage challenges they create? 4. How organizations attempt to workaround these challenges. 5. How to design a solution to these challenges instead of a workaround.

Vm13 vnx mixed workloadspittmantony

Varrow datacenter storage today and tomorrow

pittmantony

Cassandra On EC2Matthew Dennis

Caching principles-solutionspmanvi

Building high traffic http front-ends. theo schlossnagle. зал 1rit2011

Building low latency java applications with ehcache

Chris Westin

Ensuring performance for real time packet processing in open stack white paper

hptoga

Optimization_of_Virtual_Machines_for_High_Performance

StorPool Storage

BigData as a Platform: Cassandra and Current Trends

Matthew Dennis

SM16 - Can i move my stuff to openstack

pittmantony

Scalabe MySQL Infrastructure

Balazs Pocze

Caching In Java- Best Practises and Pitfalls

HARIHARAN ANANTHARAMAN

My S Q L Replication Getting The Most From SlavesPerconaPerformance

Distributed Caching Essential Lessons (Ts 1402)Yury Kaliaha

Scalability, Availability & Stability Patterns

Jonas Bonér

From cache to in-memory data grid. Introduction to Hazelcast.

Taras Matyashovsky

Scylla Summit 2018: Rebuilding the Ceph Distributed Storage Solution with Sea...

ScyllaDB

RedHat built a distributed object storage solution named Ceph which first debuted ten years ago. Now we are seeing rapid developments in the industry and we want to take advantage of them. In this talk, we will briefly introduce Ceph, revisit the problems we are seeing when profiling its I/O performance with flash device, and explain why we want to embrace the future by switching to Seastar. We’ll share our experiences with the audience of how and when we are porting our software to this framework.

What's hot

StarWind_V_SAN_product_presentation

Cheer Chain Enterprise Co., Ltd.

Hybrid Storage Pools (Now with the benefit of hindsight!)

ahl0003

Varrow madness 2013 virtualizing sql presentationpittmantony

Webinar: StorPool and WHIR - better storage, better business

StorPool Storage

Webinar: Overcoming the Storage Challenges Cassandra and Couchbase Create

Storage Switzerland

Vm13 vnx mixed workloadspittmantony

Varrow datacenter storage today and tomorrow

pittmantony

Cassandra On EC2Matthew Dennis

Caching principles-solutionspmanvi

Building high traffic http front-ends. theo schlossnagle. зал 1rit2011

Building low latency java applications with ehcache

Chris Westin

Ensuring performance for real time packet processing in open stack white paper

hptoga

Optimization_of_Virtual_Machines_for_High_Performance

StorPool Storage

BigData as a Platform: Cassandra and Current Trends

Matthew Dennis

SM16 - Can i move my stuff to openstack

pittmantony

Scalabe MySQL Infrastructure

Balazs Pocze

Caching In Java- Best Practises and Pitfalls

HARIHARAN ANANTHARAMAN

My S Q L Replication Getting The Most From SlavesPerconaPerformance

What's hot (18)

StarWind_V_SAN_product_presentation

Hybrid Storage Pools (Now with the benefit of hindsight!)

Varrow madness 2013 virtualizing sql presentation

Webinar: StorPool and WHIR - better storage, better business

Webinar: Overcoming the Storage Challenges Cassandra and Couchbase Create

Vm13 vnx mixed workloads

Varrow datacenter storage today and tomorrow

Cassandra On EC2

Caching principles-solutions

Building high traffic http front-ends. theo schlossnagle. зал 1

Building low latency java applications with ehcache

Ensuring performance for real time packet processing in open stack white paper

Optimization_of_Virtual_Machines_for_High_Performance

BigData as a Platform: Cassandra and Current Trends

SM16 - Can i move my stuff to openstack

Scalabe MySQL Infrastructure

Caching In Java- Best Practises and Pitfalls

My S Q L Replication Getting The Most From Slaves

Similar to Replication Distilled: Hazelcast Deep Dive @ In-Memory Computing Summit San Francisco 2018

Distributed Caching Essential Lessons (Ts 1402)Yury Kaliaha

Scalability, Availability & Stability Patterns

Jonas Bonér

From cache to in-memory data grid. Introduction to Hazelcast.

Taras Matyashovsky

Scylla Summit 2018: Rebuilding the Ceph Distributed Storage Solution with Sea...

ScyllaDB

Jug Lugano - Scale over the limitsDavide Carnevali

Challenges and Opportunities of Big Data GenomicsYasin Memari

In-memory Data Management Trends & Techniques

Hazelcast

"JBoss clustering solutions Mission Critical Enterprise" by Mircea Markus @ e...

eLiberatica

This is a presentation held at eLiberatica 2009. http://www.eliberatica.ro/2009/ One of the biggest events of its kind in Eastern Europe, eLiberatica brings community leaders from around the world to discuss about the hottest topics in FLOSS movement, demonstrating the advantages of adopting, using and developing Open Source and Free Software solutions. The eLiberatica organizational committee together with our speakers and guests, have graciously allowed media representatives and all attendees to photograph, videotape and otherwise record their sessions, on the condition that the photos, videos and recordings are licensed under the Creative Commons Share-Alike 3.0 License.

Tips and Tricks for SAP Sybase IQDon Brizendine

CPU Caches

shinolajla

Caches are used in many layers of applications that we develop today, holding data inside or outside of your runtime environment, or even distributed across multiple platforms in data fabrics. However, considerable performance gains can often be realized by configuring the deployment platform/environment and coding your application to take advantage of the properties of CPU caches. In this talk, we will explore what CPU caches are, how they work and how to measure your JVM-based application data usage to utilize them for maximum efficiency. We will discuss the future of CPU caches in a many-core world, as well as advancements that will soon arrive such as HP's Memristor.

How swift is your Swift - SD.pptx

OpenStack Foundation

HBase Operations and Best Practices

Venu Anuganti

Cassandra for Sysadmins

Nathan Milford

MySQL 5.7 clustering: The developer perspective

Ulf Wendel

(Compiled from revised slides of previous presentations - skip if you know the old presentations) A summary on clustering MySQL 5.7 with focus on the PHP clients view and the PHP driver. Which kinds on MySQL clusters are there, what are their goal, how does wich one scale, what extra work does which clustering technique put at the client and finally, how the PHP driver (PECL/mysqlnd_ms) helps you.

Application Profiling for Memory and Performance

pradeepfn

Lightweight Grids With Terracotta

PT.JUG

Performance and predictability

RichardWarburton

These days fast code needs to operate in harmony with its environment. At the deepest level this means working well with hardware: RAM, disks and SSDs. A unifying theme is treating memory access patterns in a uniform and predictable that is sympathetic to the underlying hardware. For example writing to and reading from RAM and Hard Disks can be significantly sped up by operating sequentially on the device, rather than randomly accessing the data. In this talk we’ll cover why access patterns are important, what kind of speed gain you can get and how you can write simple high level code which works well with these kind of patterns.

Why new hardware may not make Oracle databases faster

SolarWinds

How can you know if hardware is the right answer to your Oracle database performance issues? How can you know for sure which hardware components will have the biggest impact? As a DBA or database developer, you should know that you can gain significant performance improvements without the time, money and risk associated with providing the latest server or flash storage array. Learn why new hardware may not make your Oracle database faster and what you can do instead.

Spring One 2 GX 2014 - CACHING WITH SPRING: ADVANCED TOPICS AND BEST PRACTICES

Michael Plöd

Caching is relevant for a wide range of business applications and there is a huge variety of products in the market ranging from easy to adopt local heap based caches to powerful distributed data grids. This talk addresses advanced usage of Spring’s caching abstraction such as integrating a cache provider that is not integrated by the default Spring Package. In addition to that I will also give an overview of the JCache Specification and it’s adoption in the Spring ecosystem. Finally the presentation will also address various best practices for integrating various caching solutions into enterprise grade applications that don’t have the luxury of having „eventual consistency“ as a non-functional requirement.

Scale your Alfresco Solutions

Alfresco Software

Similar to Replication Distilled: Hazelcast Deep Dive @ In-Memory Computing Summit San Francisco 2018 (20)

Distributed Caching Essential Lessons (Ts 1402)

Scalability, Availability & Stability Patterns

From cache to in-memory data grid. Introduction to Hazelcast.

Scylla Summit 2018: Rebuilding the Ceph Distributed Storage Solution with Sea...

Jug Lugano - Scale over the limits

Challenges and Opportunities of Big Data Genomics

In-memory Data Management Trends & Techniques

"JBoss clustering solutions Mission Critical Enterprise" by Mircea Markus @ e...

Tips and Tricks for SAP Sybase IQ

CPU Caches

How swift is your Swift - SD.pptx

HBase Operations and Best Practices

Cassandra for Sysadmins

MySQL 5.7 clustering: The developer perspective

Application Profiling for Memory and Performance

Lightweight Grids With Terracotta

Performance and predictability

Why new hardware may not make Oracle databases faster

Spring One 2 GX 2014 - CACHING WITH SPRING: ADVANCED TOPICS AND BEST PRACTICES

Scale your Alfresco Solutions

More from Ensar Basri Kahveci

java.util.concurrent for Distributed Coordination - Berlin Expert Days 2019

Ensar Basri Kahveci

The evolution of distributed coordination tools shows that high-level APIs ease implementation of coordination tasks, such as leader election, locking, synchronized actions. For instance, the Chubby paper highlights familiarity of lock-based interfaces. Similarly, Apache Curator hides complexity of ZooKeeper recipes behind Java APIs, while etcd and Consul implement concurrency primitives on their own. A different path in this journey would be extending the long-lasting java.util.concurrent APIs, such as Lock, Semaphore, etc. Simplicity of these APIs makes them very useful in distributed coordination use cases. Join this talk to explore Hazelcast’s brand new implementation of java.util.concurrent APIs on top of the Raft consensus algorithm. I will walk through code samples to demonstrate how Java locks, semaphores, etc. can be used in distributed environments that involve partial failures. I will also share our experience of how we coped with several challenges we faced while developing and testing our Raft implementation.

java.util.concurrent for Distributed Coordination, Riga DevDays 2019

Ensar Basri Kahveci

java.util.concurrent for Distributed Coordination, GeeCON Krakow 2019

Ensar Basri Kahveci

java.util.concurrent for Distributed Coordination, JEEConf 2019

Ensar Basri Kahveci

Distributed Systems Theory for Mere Mortals - Software Craftsmanship Turkey

Ensar Basri Kahveci

Distributed Systems Theory for Mere Mortals - Topconf Dusseldorf October 2017

Ensar Basri Kahveci

Distributed Systems Theory for Mere Mortals - GeeCON Krakow May 2017

Ensar Basri Kahveci

Replication in the Wild - Warsaw Cloud Native Meetup - May 2017

Ensar Basri Kahveci

Distributed Systems Theory for Mere Mortals - Java Day Istanbul May 2017

Ensar Basri Kahveci

Client-centric Consistency Models

Ensar Basri Kahveci

Replication in the Wild

Ensar Basri Kahveci

Distributed Systems Theory for Mere Mortals

Ensar Basri Kahveci

Ankara Jug - Practical Functional Programming with Scala

Ensar Basri Kahveci

More from Ensar Basri Kahveci (13)