Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Distributed Locking in Kubernetes
Similar to Distributed Locking in Kubernetes (20)
More from Rafał Leszko
More from Rafał Leszko (15)
Recently uploaded
Recently uploaded (20)
Distributed Locking in Kubernetes
- 1. Distributed Locking in Kubernetes Rafał Leszko @RafalLeszko rafalleszko.com Hazelcast
- 2. About me ● Cloud-Native Team Lead at Hazelcast ● Worked at Google and CERN ● Author of the book "Continuous Delivery with Docker and Jenkins" ● Trainer and conference speaker ● Live in Kraków, Poland
- 3. About Hazelcast ● Distributed Company ● Open Source Software ● 140+ Employees ● Products: ○ Hazelcast IMDG ○ Hazelcast Jet ○ Hazelcast Cloud @Hazelcast
- 4. ● Introduction ● Distributed Locking ○ Single-Instance ○ Multi-Instance ○ RedLock ○ Consensus-Based ○ Kubernetes Native ○ Fenced ● Summary Agenda
- 5. Introduction
- 6. Reasons for Distributed Locking ● Efficiency ○ prevent executing the same work more than once, e.g., performing some expensive calculation ○ failing lock results in some additional costs or some inconvenience ● Correctness ○ prevent data corruption, data loss, inconsistency ○ failing lock results in some serious problems with the system
- 7. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager
- 8. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock()
- 9. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1
- 10. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 write()
- 11. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 write() unlock()
- 12. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 write() unlock() lock()
- 13. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 write() unlock() lock() lock held by app 2
- 14. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 write() unlock() lock() lock held by app 2 write()
- 15. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() unlock() write() write() lock() unlock()
- 16. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager
- 17. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1
- 18. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1
- 19. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 lease expired
- 20. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() write() lock() unlock() lease expired
- 21. Single-Instance
- 22. Application 1 Single-Instance Distributed Locking Application 2 lock() lock() Hazelcast
- 23. Application 1 Single-Instance Distributed Locking Application 2 lock() lock() Redis
- 24. Application 1 Single-Instance Distributed Locking Application 2 lock() lock() Hazelcast
- 25. Code: Single-Instance Distributed Locking var hazelcast = HazelcastClient. newHazelcastClient(); var lock = hazelcast.getCPSubsystem().getLock( "my-lock"); lock.lock(); writeToSharedResource(); lock.unlock();
- 26. Code: Single-Instance Distributed Locking var config = new ClientConfig(); config.getNetworkConfig().addAddress("hazelcast"); var hazelcast = HazelcastClient.newHazelcastClient(config); var lock = hazelcast.getCPSubsystem().getLock("my-lock"); lock.lock(); writeToSharedResource(); lock.unlock(); Server Application kubectl run hazelcast --image hazelcast/hazelcast --port 5701 --expose
- 27. Single-Instance Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() unlock() write() write() lock() unlock()
- 28. Single-Instance Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() unlock() write() write() lock() unlock()
- 29. Application 1 Single-Instance Distributed Locking Application 2 lock() lock() Hazelcast
- 30. Summary: Single-Instance Distributed Locking Efficiency Correctness Complexity Performance
- 31. ● Introduction ✔ ● Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ○ RedLock ○ Consensus-Based ○ Kubernetes Native ○ Fenced ● Summary Agenda
- 32. Multi-Instance
- 33. Application 1 Multi-Instance Distributed Locking Application 2 lock() lock() Hazelcast Cluster
- 34. Application 1 Multi-Instance Distributed Locking Application 2 lock() lock() Hazelcast Cluster
- 35. Code: Multi-Instance Distributed Locking var config = new ClientConfig(); config.getNetworkConfig().addAddress("hazelcast"); var hazelcast = HazelcastClient.newHazelcastClient(config); var lock = hazelcast.getCPSubsystem().getLock("my-lock"); lock.lock(); writeToSharedResource(); lock.unlock(); Server Application helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/ helm install hazelcast hazelcast/hazelcast
- 36. Application 1 Multi-Instance Distributed Locking Application 2 lock() lock() Hazelcast Cluster
- 37. Application 1 Multi-Instance Distributed Locking Application 2 lock() lock() Hazelcast Cluster(s)
- 38. Summary: Multi-Instance Distributed Locking Efficiency Correctness Complexity Performance
- 39. ● Introduction ✔ ● Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ✔ ○ RedLock ○ Consensus-Based ○ Kubernetes Native ○ Fenced ● Summary Agenda
- 40. RedLock
- 45. Code: RedLock var lock = new RedissonRedLock( client("redis-1").getLock("lock1"), client("redis-2").getLock("lock2"), client("redis-3").getLock("lock3") ); lock.lock(); writeToSharedResource(); lock.unlock(); Server Application kubectl run redis-1 --image redis:6.2.1 --port 6379 --expose kubectl run redis-2 --image redis:6.2.1 --port 6379 --expose kubectl run redis-3 --image redis:6.2.1 --port 6379 --expose
- 47. RedLock: Issues ● Server clocks need to be in sync ● Executing lock() and write() is not atomic
- 48. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock()
- 49. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1
- 50. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 process blocked
- 51. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 process blocked lock held by app 2 lock()
- 52. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 process blocked lock held by app 2 lock() write()
- 53. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 write() write() lock() process blocked lock()
- 55. ● Introduction ✔ ● Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ✔ ○ RedLock ✔ ○ Consensus-Based ○ Kubernetes Native ○ Fenced ● Summary Agenda
- 56. Consensus-Based
- 57. Application 1 Consensus-Based Distributed Locking Application 2 lock() lock() Hazelcast Cluster (CP Subsystem Enabled)
- 58. Application 1 Consensus-Based Distributed Locking Application 2 lock() lock() Zookeeper Cluster
- 59. Application 1 Consensus-Based Distributed Locking Application 2 lock() lock() Hazelcast Cluster (CP Subsystem Enabled)
- 60. Code: Consensus-Based Distributed Locking var config = new ClientConfig(); config.getNetworkConfig().addAddress("hazelcast"); var hazelcast = HazelcastClient.newHazelcastClient(config); var lock = hazelcast.getCPSubsystem().getLock("my-lock"); lock.lock(); writeToSharedResource(); lock.unlock(); Server Application helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/ helm install hazelcast hazelcast/hazelcast --set hazelcast.yaml.hazelcast.cp-subsystem.cp-member-count=3
- 61. Summary: Consensus-Based Distributed Locking Efficiency Correctness Complexity Performance
- 62. Application 1 Consensus-Based Distributed Locking Application 2 lock() lock() Hazelcast Cluster (CP Subsystem Enabled)
- 63. Application 1 Consensus-Based Distributed Locking Application 2 lock() lock() Zookeeper Cluster
- 65. Application 1 Kubernetes Native Distributed Locking Application 2 lock() lock() Kubernetes Cluster
- 66. Kubernetes Native Distributed Locking
- 68. Libraries for Kubernetes Native Distributed Locking ● Lockgate: cross-platform locking library with distributed locks using Kubernetes ● Kube-lock: simple library that implements a distributed lock using annotations on a Kubernetes resource
- 69. Summary: Kubernetes Native Distributed Locking Efficiency Correctness Complexity Performance
- 70. ● Introduction ✔ ● Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ✔ ○ RedLock ✔ ○ Consensus-Based ✔ ○ Kubernetes Native ✔ ○ Fenced ● Summary Agenda
- 71. Fenced
- 72. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() write() write() lock() process blocked
- 73. Application Fenced Distributed Locking token = lockAndGetToken() Hazelcast Cluster (CP Subsystem Enabled) Shared Resource token 1 2
- 74. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager
- 75. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock()
- 76. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11
- 77. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked
- 78. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock()
- 79. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock() lock held by app 2 token: 12
- 80. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock() lock held by app 2 token: 12 write() token: 12
- 81. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock() lock held by app 2 token: 12 write() token: 12
- 82. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock() lock held by app 2 token: 12 write() token: 12 write() token: 11
- 83. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() write() token: 11 write() token: 12 lock() process blocked token: 11 token: 12 rejected
- 84. Code: Fenced Distributed Locking var config = new ClientConfig(); config.getNetworkConfig().addAddress("hazelcast"); var hazelcast = HazelcastClient.newHazelcastClient(config); var lock = hazelcast.getCPSubsystem().getLock("my-lock"); long token = lock.lockAndGetFence(); writetoSharedResource(token); lock.unlock(); Server Application helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/ helm install hazelcast hazelcast/hazelcast --set hazelcast.yaml.hazelcast.cp-subsystem.cp-member-count=3
- 85. Summary: Fenced Distributed Locking Efficiency Correctness Complexity Performance
- 86. ● Introduction ✔ ● Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ✔ ○ RedLock ✔ ○ Consensus-Based ✔ ○ Kubernetes Native ✔ ○ Fenced ✔ ● Summary Agenda
- 87. Summary
- 89. Resources ● Code for this presentation: https://github.com/leszko/distributed-locking ● How to do distributed locking: https://martin.kleppmann.com/2016/02/08/how-to-do-distrib uted-locking.htm ● Distributed Locks are Dead; Long Live Distributed Locks! https://hazelcast.com/blog/long-live-distributed-locks/