1. Distributed Locking in Kubernetes
Rafał Leszko
@RafalLeszko
rafalleszko.com
Hazelcast

2. About me
● Cloud-Native Team Lead at Hazelcast
● Worked at Google and CERN
● Author of the book "Continuous Delivery
with Docker and Jenkins"
● Trainer and conference speaker
● Live in Kraków, Poland

3. About Hazelcast
● Distributed Company
● Open Source Software
● 140+ Employees
● Products:
○ Hazelcast IMDG
○ Hazelcast Jet
○ Hazelcast Cloud
@Hazelcast

4. ● Introduction
● Distributed Locking
○ Single-Instance
○ Multi-Instance
○ RedLock
○ Consensus-Based
○ Kubernetes Native
○ Fenced
● Summary
Agenda

5. Introduction

6. Reasons for Distributed Locking
● Efficiency
○ prevent executing the same work more than once, e.g.,
performing some expensive calculation
○ failing lock results in some additional costs or some
inconvenience
● Correctness
○ prevent data corruption, data loss, inconsistency
○ failing lock results in some serious problems with the
system

7. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager

8. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()

9. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1

10. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
write()

11. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
write()
unlock()

12. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
write()
unlock()
lock()

13. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
write()
unlock()
lock()
lock held by app 2

14. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
write()
unlock()
lock()
lock held by app 2
write()

15. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock held by app 1 lock held by app 2
lock() unlock()
write()
write()
lock() unlock()

16. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager

17. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1

18. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1

19. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
lease
expired

20. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock held by app 1 lock held by app 2
lock()
write()
lock() unlock()
lease
expired

21. Single-Instance

22. Application 1
Single-Instance Distributed Locking
Application 2
lock()
lock()
Hazelcast

23. Application 1
Single-Instance Distributed Locking
Application 2
lock()
lock()
Redis

24. Application 1
Single-Instance Distributed Locking
Application 2
lock()
lock()
Hazelcast

25. Code: Single-Instance Distributed Locking
var hazelcast = HazelcastClient.
newHazelcastClient();
var lock = hazelcast.getCPSubsystem().getLock(
"my-lock");
lock.lock();
writeToSharedResource();
lock.unlock();

26. Code: Single-Instance Distributed Locking
var config = new ClientConfig();
config.getNetworkConfig().addAddress("hazelcast");
var hazelcast = HazelcastClient.newHazelcastClient(config);
var lock = hazelcast.getCPSubsystem().getLock("my-lock");
lock.lock();
writeToSharedResource();
lock.unlock();
Server
Application
kubectl run hazelcast --image hazelcast/hazelcast --port 5701 --expose

27. Single-Instance Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock held by app 1 lock held by app 2
lock() unlock()
write()
write()
lock() unlock()

28. Single-Instance Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock held by app 1 lock held by app 2
lock() unlock()
write()
write()
lock() unlock()

29. Application 1
Single-Instance Distributed Locking
Application 2
lock()
lock()
Hazelcast

30. Summary: Single-Instance Distributed Locking
Efficiency
Correctness
Complexity
Performance

31. ● Introduction ✔
● Distributed Locking
○ Single-Instance ✔
○ Multi-Instance
○ RedLock
○ Consensus-Based
○ Kubernetes Native
○ Fenced
● Summary
Agenda

32. Multi-Instance

33. Application 1
Multi-Instance Distributed Locking
Application 2
lock()
lock()
Hazelcast Cluster

34. Application 1
Multi-Instance Distributed Locking
Application 2
lock()
lock()
Hazelcast Cluster

35. Code: Multi-Instance Distributed Locking
var config = new ClientConfig();
config.getNetworkConfig().addAddress("hazelcast");
var hazelcast = HazelcastClient.newHazelcastClient(config);
var lock = hazelcast.getCPSubsystem().getLock("my-lock");
lock.lock();
writeToSharedResource();
lock.unlock();
Server
Application
helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/
helm install hazelcast hazelcast/hazelcast

36. Application 1
Multi-Instance Distributed Locking
Application 2
lock()
lock()
Hazelcast Cluster

37. Application 1
Multi-Instance Distributed Locking
Application 2
lock()
lock()
Hazelcast Cluster(s)

38. Summary: Multi-Instance Distributed Locking
Efficiency
Correctness
Complexity
Performance

39. ● Introduction ✔
● Distributed Locking
○ Single-Instance ✔
○ Multi-Instance ✔
○ RedLock
○ Consensus-Based
○ Kubernetes Native
○ Fenced
● Summary
Agenda

40. RedLock

41. Application 1
RedLock
Application 2
lock()
Redis
Redis
Redis

42. Application 1
RedLock
Application 2
lock()
Redis
Redis
Redis

43. Application 1
RedLock
Application 2
lock()
Redis
Redis
Redis

44. Application 1
RedLock
Application 2
lock()
Redis
Redis
Redis

45. Code: RedLock
var lock = new RedissonRedLock(
client("redis-1").getLock("lock1"),
client("redis-2").getLock("lock2"),
client("redis-3").getLock("lock3")
);
lock.lock();
writeToSharedResource();
lock.unlock();
Server
Application
kubectl run redis-1 --image redis:6.2.1 --port 6379 --expose
kubectl run redis-2 --image redis:6.2.1 --port 6379 --expose
kubectl run redis-3 --image redis:6.2.1 --port 6379 --expose

46. Application 1
RedLock
Application 2
lock()
Redis
Redis
Redis

47. RedLock: Issues
● Server clocks need to be in sync
● Executing lock() and write() is not atomic

48. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()

49. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1

50. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
process blocked

51. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
process blocked
lock held by app 2
lock()

52. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
process blocked
lock held by app 2
lock()
write()

53. Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock held by app 1 lock held by app 2
write()
write()
lock()
process blocked
lock()

54. Summary: RedLock
Efficiency
Correctness
Complexity
Performance

55. ● Introduction ✔
● Distributed Locking
○ Single-Instance ✔
○ Multi-Instance ✔
○ RedLock ✔
○ Consensus-Based
○ Kubernetes Native
○ Fenced
● Summary
Agenda

56. Consensus-Based

57. Application 1
Consensus-Based Distributed Locking
Application 2
lock()
lock()
Hazelcast Cluster
(CP Subsystem Enabled)

58. Application 1
Consensus-Based Distributed Locking
Application 2
lock()
lock()
Zookeeper Cluster

59. Application 1
Consensus-Based Distributed Locking
Application 2
lock()
lock()
Hazelcast Cluster
(CP Subsystem Enabled)

60. Code: Consensus-Based Distributed Locking
var config = new ClientConfig();
config.getNetworkConfig().addAddress("hazelcast");
var hazelcast = HazelcastClient.newHazelcastClient(config);
var lock = hazelcast.getCPSubsystem().getLock("my-lock");
lock.lock();
writeToSharedResource();
lock.unlock();
Server
Application
helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/
helm install hazelcast hazelcast/hazelcast
--set hazelcast.yaml.hazelcast.cp-subsystem.cp-member-count=3

61. Summary: Consensus-Based Distributed Locking
Efficiency
Correctness
Complexity
Performance

62. Application 1
Consensus-Based Distributed Locking
Application 2
lock()
lock()
Hazelcast Cluster
(CP Subsystem Enabled)

63. Application 1
Consensus-Based Distributed Locking
Application 2
lock()
lock()
Zookeeper Cluster

64. Kubernetes Native

65. Application 1
Kubernetes Native Distributed Locking
Application 2
lock()
lock()
Kubernetes Cluster

66. Kubernetes Native Distributed Locking

68. Libraries for Kubernetes Native Distributed Locking
● Lockgate: cross-platform locking library with distributed locks
using Kubernetes
● Kube-lock: simple library that implements a distributed lock
using annotations on a Kubernetes resource

69. Summary: Kubernetes Native Distributed Locking
Efficiency
Correctness
Complexity
Performance

70. ● Introduction ✔
● Distributed Locking
○ Single-Instance ✔
○ Multi-Instance ✔
○ RedLock ✔
○ Consensus-Based ✔
○ Kubernetes Native ✔
○ Fenced
● Summary
Agenda

71. Fenced

72. Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock held by app 1 lock held by app 2
lock()
write()
write()
lock()
process blocked

73. Application
Fenced Distributed Locking
token =
lockAndGetToken()
Hazelcast Cluster
(CP Subsystem Enabled)
Shared
Resource
token
1
2

74. Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager

75. Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()

76. Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
token: 11

77. Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
token: 11
process blocked

78. Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
token: 11
process blocked lock()

79. Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
token: 11
process blocked lock()
lock held by app 2
token: 12

80. Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
token: 11
process blocked lock()
lock held by app 2
token: 12
write()
token: 12

81. Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
token: 11
process blocked lock()
lock held by app 2
token: 12
write()
token: 12

82. Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock()
lock held by app 1
token: 11
process blocked lock()
lock held by app 2
token: 12
write()
token: 12
write()
token: 11

83. Fenced Distributed Locking
Shared
Resource
Application 2
Application 1
Lock Manager
lock held by app 1 lock held by app 2
lock()
write()
token: 11
write()
token: 12
lock()
process blocked
token: 11 token: 12
rejected

84. Code: Fenced Distributed Locking
var config = new ClientConfig();
config.getNetworkConfig().addAddress("hazelcast");
var hazelcast = HazelcastClient.newHazelcastClient(config);
var lock = hazelcast.getCPSubsystem().getLock("my-lock");
long token = lock.lockAndGetFence();
writetoSharedResource(token);
lock.unlock();
Server
Application
helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/
helm install hazelcast hazelcast/hazelcast
--set hazelcast.yaml.hazelcast.cp-subsystem.cp-member-count=3

85. Summary: Fenced Distributed Locking
Efficiency
Correctness
Complexity
Performance

86. ● Introduction ✔
● Distributed Locking
○ Single-Instance ✔
○ Multi-Instance ✔
○ RedLock ✔
○ Consensus-Based ✔
○ Kubernetes Native ✔
○ Fenced ✔
● Summary
Agenda

87. Summary

88. Efficiency
Performance Complexity
Correctness
Single-Instance
Multi-Instance
RedLock
Consensus-Based
Kubernetes Native
Fenced

89. Resources
● Code for this presentation:
https://github.com/leszko/distributed-locking
● How to do distributed locking:
https://martin.kleppmann.com/2016/02/08/how-to-do-distrib
uted-locking.htm
● Distributed Locks are Dead; Long Live Distributed Locks!
https://hazelcast.com/blog/long-live-distributed-locks/

90. Thank You!
Rafał Leszko
@RafalLeszko
rafalleszko.com