Distributed Locking in Kubernetes

Distributed Locking in Kubernetes
Rafał Leszko
@RafalLeszko
rafalleszko.com
Hazelcast
About me
● Cloud-Native Team Lead at Hazelcast
● Worked at Google and CERN
● Author of the book "Continuous Delivery
with Docker and Jenkins"
● Trainer and conference speaker
● Live in Kraków, Poland
About Hazelcast
● Distributed Company
● Open Source Software
● 140+ Employees
● Products:
○ Hazelcast IMDG
○ Hazelcast Jet
○ Hazelcast Cloud
@Hazelcast
● Introduction
● Distributed Locking
○ Single-Instance
○ Multi-Instance
○ RedLock
○ Consensus-Based
○ Kubernetes Native
○ Fenced
● Summary
Agenda
Introduction
Reasons for Distributed Locking
● Efficiency
○ prevent executing the same work more than once, e.g.,
performing some expensive calculation
○ failing lock results in some additional costs or some
inconvenience
● Correctness
○ prevent data corruption, data loss, inconsistency
○ failing lock results in some serious problems with the
system
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
write()
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
write()
unlock()
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
write()
unlock()
lock()
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
write()
unlock()
lock()
lock held by app 2
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
write()
unlock()
lock()
lock held by app 2
write()
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock held by app 1 lock held by app 2
lock() unlock()
write()
write()
lock() unlock()
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
lease
expired
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock held by app 1 lock held by app 2
lock()
write()
lock() unlock()
lease
expired
Single-Instance
Application 1
Single-Instance Distributed Locking
Application 2
lock()
lock()
Hazelcast
Application 1
Single-Instance Distributed Locking
Application 2
lock()
lock()
Redis
Application 1
Single-Instance Distributed Locking
Application 2
lock()
lock()
Hazelcast
Code: Single-Instance Distributed Locking
var hazelcast = HazelcastClient.
newHazelcastClient();
var lock = hazelcast.getCPSubsystem().getLock(
"my-lock");
lock.lock();
writeToSharedResource();
lock.unlock();
Code: Single-Instance Distributed Locking
var config = new ClientConfig();
config.getNetworkConfig().addAddress("hazelcast");
var hazelcast = HazelcastClient.newHazelcastClient(config);
var lock = hazelcast.getCPSubsystem().getLock("my-lock");
lock.lock();
writeToSharedResource();
lock.unlock();
Server
Application
kubectl run hazelcast --image hazelcast/hazelcast --port 5701 --expose
Single-Instance Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock held by app 1 lock held by app 2
lock() unlock()
write()
write()
lock() unlock()
Single-Instance Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock held by app 1 lock held by app 2
lock() unlock()
write()
write()
lock() unlock()
Application 1
Single-Instance Distributed Locking
Application 2
lock()
lock()
Hazelcast
Summary: Single-Instance Distributed Locking
Efficiency
Correctness
Complexity
Performance
● Introduction ✔
● Distributed Locking
○ Single-Instance ✔
○ Multi-Instance
○ RedLock
○ Consensus-Based
○ Kubernetes Native
○ Fenced
● Summary
Agenda
Multi-Instance
Application 1
Multi-Instance Distributed Locking
Application 2
lock()
lock()
Hazelcast Cluster
Application 1
Multi-Instance Distributed Locking
Application 2
lock()
lock()
Hazelcast Cluster
Code: Multi-Instance Distributed Locking
var config = new ClientConfig();
config.getNetworkConfig().addAddress("hazelcast");
var hazelcast = HazelcastClient.newHazelcastClient(config);
var lock = hazelcast.getCPSubsystem().getLock("my-lock");
lock.lock();
writeToSharedResource();
lock.unlock();
Server
Application
helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/
helm install hazelcast hazelcast/hazelcast
Application 1
Multi-Instance Distributed Locking
Application 2
lock()
lock()
Hazelcast Cluster
Application 1
Multi-Instance Distributed Locking
Application 2
lock()
lock()
Hazelcast Cluster(s)
Summary: Multi-Instance Distributed Locking
Efficiency
Correctness
Complexity
Performance
● Introduction ✔
● Distributed Locking
○ Single-Instance ✔
○ Multi-Instance ✔
○ RedLock
○ Consensus-Based
○ Kubernetes Native
○ Fenced
● Summary
Agenda
RedLock
Application 1
RedLock
Application 2
lock()
Redis
Redis
Redis
Application 1
RedLock
Application 2
lock()
Redis
Redis
Redis
Application 1
RedLock
Application 2
lock()
Redis
Redis
Redis
Application 1
RedLock
Application 2
lock()
Redis
Redis
Redis
Code: RedLock
var lock = new RedissonRedLock(
client("redis-1").getLock("lock1"),
client("redis-2").getLock("lock2"),
client("redis-3").getLock("lock3")
);
lock.lock();
writeToSharedResource();
lock.unlock();
Server
Application
kubectl run redis-1 --image redis:6.2.1 --port 6379 --expose
kubectl run redis-2 --image redis:6.2.1 --port 6379 --expose
kubectl run redis-3 --image redis:6.2.1 --port 6379 --expose
Application 1
RedLock
Application 2
lock()
Redis
Redis
Redis
RedLock: Issues
● Server clocks need to be in sync
● Executing lock() and write() is not atomic
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
process blocked
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
process blocked
lock held by app 2
lock()
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
process blocked
lock held by app 2
lock()
write()
Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock held by app 1 lock held by app 2
write()
write()
lock()
process blocked
lock()
Summary: RedLock
Efficiency
Correctness
Complexity
Performance
● Introduction ✔
● Distributed Locking
○ Single-Instance ✔
○ Multi-Instance ✔
○ RedLock ✔
○ Consensus-Based
○ Kubernetes Native
○ Fenced
● Summary
Agenda
Consensus-Based
Application 1
Consensus-Based Distributed Locking
Application 2
lock()
lock()
Hazelcast Cluster
(CP Subsystem Enabled)
Application 1
Consensus-Based Distributed Locking
Application 2
lock()
lock()
Zookeeper Cluster
Application 1
Consensus-Based Distributed Locking
Application 2
lock()
lock()
Hazelcast Cluster
(CP Subsystem Enabled)
Code: Consensus-Based Distributed Locking
var config = new ClientConfig();
config.getNetworkConfig().addAddress("hazelcast");
var hazelcast = HazelcastClient.newHazelcastClient(config);
var lock = hazelcast.getCPSubsystem().getLock("my-lock");
lock.lock();
writeToSharedResource();
lock.unlock();
Server
Application
helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/
helm install hazelcast hazelcast/hazelcast 
--set hazelcast.yaml.hazelcast.cp-subsystem.cp-member-count=3
Summary: Consensus-Based Distributed Locking
Efficiency
Correctness
Complexity
Performance
Application 1
Consensus-Based Distributed Locking
Application 2
lock()
lock()
Hazelcast Cluster
(CP Subsystem Enabled)
Application 1
Consensus-Based Distributed Locking
Application 2
lock()
lock()
Zookeeper Cluster
Kubernetes Native
Application 1
Kubernetes Native Distributed Locking
Application 2
lock()
lock()
Kubernetes Cluster
Kubernetes Native Distributed Locking
Distributed Locking in Kubernetes
Libraries for Kubernetes Native Distributed Locking
● Lockgate: cross-platform locking library with distributed locks
using Kubernetes
● Kube-lock: simple library that implements a distributed lock
using annotations on a Kubernetes resource
Summary: Kubernetes Native Distributed Locking
Efficiency
Correctness
Complexity
Performance
● Introduction ✔
● Distributed Locking
○ Single-Instance ✔
○ Multi-Instance ✔
○ RedLock ✔
○ Consensus-Based ✔
○ Kubernetes Native ✔
○ Fenced
● Summary
Agenda
Fenced
Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock held by app 1 lock held by app 2
lock()
write()
write()
lock()
process blocked
Application
Fenced Distributed Locking
token =
lockAndGetToken()
Hazelcast Cluster
(CP Subsystem Enabled)
Shared
Resource
token
1
2
Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
token: 11
Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
token: 11
process blocked
Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
token: 11
process blocked lock()
Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
token: 11
process blocked lock()
lock held by app 2
token: 12
Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
token: 11
process blocked lock()
lock held by app 2
token: 12
write()
token: 12
Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
token: 11
process blocked lock()
lock held by app 2
token: 12
write()
token: 12
Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
token: 11
process blocked lock()
lock held by app 2
token: 12
write()
token: 12
write()
token: 11
Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock held by app 1 lock held by app 2
lock()
write()
token: 11
write()
token: 12
lock()
process blocked
token: 11 token: 12
rejected
Code: Fenced Distributed Locking
var config = new ClientConfig();
config.getNetworkConfig().addAddress("hazelcast");
var hazelcast = HazelcastClient.newHazelcastClient(config);
var lock = hazelcast.getCPSubsystem().getLock("my-lock");
long token = lock.lockAndGetFence();
writetoSharedResource(token);
lock.unlock();
Server
Application
helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/
helm install hazelcast hazelcast/hazelcast 
--set hazelcast.yaml.hazelcast.cp-subsystem.cp-member-count=3
Summary: Fenced Distributed Locking
Efficiency
Correctness
Complexity
Performance
● Introduction ✔
● Distributed Locking
○ Single-Instance ✔
○ Multi-Instance ✔
○ RedLock ✔
○ Consensus-Based ✔
○ Kubernetes Native ✔
○ Fenced ✔
● Summary
Agenda
Summary
Efficiency
Performance Complexity
Correctness
Single-Instance
Multi-Instance
RedLock
Consensus-Based
Kubernetes Native
Fenced
Resources
● Code for this presentation:
https://github.com/leszko/distributed-locking
● How to do distributed locking:
https://martin.kleppmann.com/2016/02/08/how-to-do-distrib
uted-locking.htm
● Distributed Locks are Dead; Long Live Distributed Locks!
https://hazelcast.com/blog/long-live-distributed-locks/
Thank You!
Rafał Leszko
@RafalLeszko
rafalleszko.com
1 of 90

More Related Content

What's hot(20)

HashiCorp's Vault - The ExamplesHashiCorp's Vault - The Examples
HashiCorp's Vault - The Examples
Michał Czeraszkiewicz3.2K views
NGINX: Basics and Best PracticesNGINX: Basics and Best Practices
NGINX: Basics and Best Practices
NGINX, Inc.6.7K views
NginxNginx
Nginx
Geeta Vinnakota1.3K views
Kafka 101Kafka 101
Kafka 101
Clement Demonchy2.4K views
Altinity Quickstart for ClickHouseAltinity Quickstart for ClickHouse
Altinity Quickstart for ClickHouse
Altinity Ltd1.5K views
Vault Open Source vs Enterprise v2Vault Open Source vs Enterprise v2
Vault Open Source vs Enterprise v2
Stenio Ferreira2.1K views
HandsOn ProxySQL Tutorial - PLSC18HandsOn ProxySQL Tutorial - PLSC18
HandsOn ProxySQL Tutorial - PLSC18
Derek Downey752 views
ELK StackELK Stack
ELK Stack
Phuc Nguyen9.1K views
Introduction to Storm Introduction to Storm
Introduction to Storm
Chandler Huang20.1K views
Introduction to RedisIntroduction to Redis
Introduction to Redis
Arnab Mitra11K views
NginxNginx
Nginx
Dhrubaji Mandal ♛1.9K views

Similar to Distributed Locking in Kubernetes(20)

Grizzly 20080925 V2Grizzly 20080925 V2
Grizzly 20080925 V2
Eduardo Pelegri-Llopart859 views
What the Struts?What the Struts?
What the Struts?
Joe Kutner409 views
Nanocloud   cloud scale jvmNanocloud   cloud scale jvm
Nanocloud cloud scale jvm
aragozin1K views
sanlock overview and its consensus algorithmssanlock overview and its consensus algorithms
sanlock overview and its consensus algorithms
Roger Zhou 周志强333 views
Elasticsearch und die Java-WeltElasticsearch und die Java-Welt
Elasticsearch und die Java-Welt
Florian Hopf2.1K views
SWT Tech Sharing: Node.js + RedisSWT Tech Sharing: Node.js + Redis
SWT Tech Sharing: Node.js + Redis
Infinity Levels Studio1.5K views
ATT&CKING Containers in The CloudATT&CKING Containers in The Cloud
ATT&CKING Containers in The Cloud
MITRE ATT&CK647 views

Distributed Locking in Kubernetes