Successfully reported this slideshow.

More Related Content

More from Rafał Leszko

Related Books

Free with a 14 day trial from Scribd

See all

Related Audiobooks

Free with a 14 day trial from Scribd

See all

Distributed Locking in Kubernetes

  1. 1. Distributed Locking in Kubernetes Rafał Leszko @RafalLeszko rafalleszko.com Hazelcast
  2. 2. About me ● Cloud-Native Team Lead at Hazelcast ● Worked at Google and CERN ● Author of the book "Continuous Delivery with Docker and Jenkins" ● Trainer and conference speaker ● Live in Kraków, Poland
  3. 3. About Hazelcast ● Distributed Company ● Open Source Software ● 140+ Employees ● Products: ○ Hazelcast IMDG ○ Hazelcast Jet ○ Hazelcast Cloud @Hazelcast
  4. 4. ● Introduction ● Distributed Locking ○ Single-Instance ○ Multi-Instance ○ RedLock ○ Consensus-Based ○ Kubernetes Native ○ Fenced ● Summary Agenda
  5. 5. Introduction
  6. 6. Reasons for Distributed Locking ● Efficiency ○ prevent executing the same work more than once, e.g., performing some expensive calculation ○ failing lock results in some additional costs or some inconvenience ● Correctness ○ prevent data corruption, data loss, inconsistency ○ failing lock results in some serious problems with the system
  7. 7. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager
  8. 8. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock()
  9. 9. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1
  10. 10. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 write()
  11. 11. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 write() unlock()
  12. 12. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 write() unlock() lock()
  13. 13. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 write() unlock() lock() lock held by app 2
  14. 14. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 write() unlock() lock() lock held by app 2 write()
  15. 15. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() unlock() write() write() lock() unlock()
  16. 16. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager
  17. 17. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1
  18. 18. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1
  19. 19. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 lease expired
  20. 20. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() write() lock() unlock() lease expired
  21. 21. Single-Instance
  22. 22. Application 1 Single-Instance Distributed Locking Application 2 lock() lock() Hazelcast
  23. 23. Application 1 Single-Instance Distributed Locking Application 2 lock() lock() Redis
  24. 24. Application 1 Single-Instance Distributed Locking Application 2 lock() lock() Hazelcast
  25. 25. Code: Single-Instance Distributed Locking var hazelcast = HazelcastClient. newHazelcastClient(); var lock = hazelcast.getCPSubsystem().getLock( "my-lock"); lock.lock(); writeToSharedResource(); lock.unlock();
  26. 26. Code: Single-Instance Distributed Locking var config = new ClientConfig(); config.getNetworkConfig().addAddress("hazelcast"); var hazelcast = HazelcastClient.newHazelcastClient(config); var lock = hazelcast.getCPSubsystem().getLock("my-lock"); lock.lock(); writeToSharedResource(); lock.unlock(); Server Application kubectl run hazelcast --image hazelcast/hazelcast --port 5701 --expose
  27. 27. Single-Instance Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() unlock() write() write() lock() unlock()
  28. 28. Single-Instance Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() unlock() write() write() lock() unlock()
  29. 29. Application 1 Single-Instance Distributed Locking Application 2 lock() lock() Hazelcast
  30. 30. Summary: Single-Instance Distributed Locking Efficiency Correctness Complexity Performance
  31. 31. ● Introduction ✔ ● Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ○ RedLock ○ Consensus-Based ○ Kubernetes Native ○ Fenced ● Summary Agenda
  32. 32. Multi-Instance
  33. 33. Application 1 Multi-Instance Distributed Locking Application 2 lock() lock() Hazelcast Cluster
  34. 34. Application 1 Multi-Instance Distributed Locking Application 2 lock() lock() Hazelcast Cluster
  35. 35. Code: Multi-Instance Distributed Locking var config = new ClientConfig(); config.getNetworkConfig().addAddress("hazelcast"); var hazelcast = HazelcastClient.newHazelcastClient(config); var lock = hazelcast.getCPSubsystem().getLock("my-lock"); lock.lock(); writeToSharedResource(); lock.unlock(); Server Application helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/ helm install hazelcast hazelcast/hazelcast
  36. 36. Application 1 Multi-Instance Distributed Locking Application 2 lock() lock() Hazelcast Cluster
  37. 37. Application 1 Multi-Instance Distributed Locking Application 2 lock() lock() Hazelcast Cluster(s)
  38. 38. Summary: Multi-Instance Distributed Locking Efficiency Correctness Complexity Performance
  39. 39. ● Introduction ✔ ● Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ✔ ○ RedLock ○ Consensus-Based ○ Kubernetes Native ○ Fenced ● Summary Agenda
  40. 40. RedLock
  41. 41. Application 1 RedLock Application 2 lock() Redis Redis Redis
  42. 42. Application 1 RedLock Application 2 lock() Redis Redis Redis
  43. 43. Application 1 RedLock Application 2 lock() Redis Redis Redis
  44. 44. Application 1 RedLock Application 2 lock() Redis Redis Redis
  45. 45. Code: RedLock var lock = new RedissonRedLock( client("redis-1").getLock("lock1"), client("redis-2").getLock("lock2"), client("redis-3").getLock("lock3") ); lock.lock(); writeToSharedResource(); lock.unlock(); Server Application kubectl run redis-1 --image redis:6.2.1 --port 6379 --expose kubectl run redis-2 --image redis:6.2.1 --port 6379 --expose kubectl run redis-3 --image redis:6.2.1 --port 6379 --expose
  46. 46. Application 1 RedLock Application 2 lock() Redis Redis Redis
  47. 47. RedLock: Issues ● Server clocks need to be in sync ● Executing lock() and write() is not atomic
  48. 48. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock()
  49. 49. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1
  50. 50. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 process blocked
  51. 51. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 process blocked lock held by app 2 lock()
  52. 52. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 process blocked lock held by app 2 lock() write()
  53. 53. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 write() write() lock() process blocked lock()
  54. 54. Summary: RedLock Efficiency Correctness Complexity Performance
  55. 55. ● Introduction ✔ ● Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ✔ ○ RedLock ✔ ○ Consensus-Based ○ Kubernetes Native ○ Fenced ● Summary Agenda
  56. 56. Consensus-Based
  57. 57. Application 1 Consensus-Based Distributed Locking Application 2 lock() lock() Hazelcast Cluster (CP Subsystem Enabled)
  58. 58. Application 1 Consensus-Based Distributed Locking Application 2 lock() lock() Zookeeper Cluster
  59. 59. Application 1 Consensus-Based Distributed Locking Application 2 lock() lock() Hazelcast Cluster (CP Subsystem Enabled)
  60. 60. Code: Consensus-Based Distributed Locking var config = new ClientConfig(); config.getNetworkConfig().addAddress("hazelcast"); var hazelcast = HazelcastClient.newHazelcastClient(config); var lock = hazelcast.getCPSubsystem().getLock("my-lock"); lock.lock(); writeToSharedResource(); lock.unlock(); Server Application helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/ helm install hazelcast hazelcast/hazelcast --set hazelcast.yaml.hazelcast.cp-subsystem.cp-member-count=3
  61. 61. Summary: Consensus-Based Distributed Locking Efficiency Correctness Complexity Performance
  62. 62. Application 1 Consensus-Based Distributed Locking Application 2 lock() lock() Hazelcast Cluster (CP Subsystem Enabled)
  63. 63. Application 1 Consensus-Based Distributed Locking Application 2 lock() lock() Zookeeper Cluster
  64. 64. Kubernetes Native
  65. 65. Application 1 Kubernetes Native Distributed Locking Application 2 lock() lock() Kubernetes Cluster
  66. 66. Kubernetes Native Distributed Locking
  67. 67. Libraries for Kubernetes Native Distributed Locking ● Lockgate: cross-platform locking library with distributed locks using Kubernetes ● Kube-lock: simple library that implements a distributed lock using annotations on a Kubernetes resource
  68. 68. Summary: Kubernetes Native Distributed Locking Efficiency Correctness Complexity Performance
  69. 69. ● Introduction ✔ ● Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ✔ ○ RedLock ✔ ○ Consensus-Based ✔ ○ Kubernetes Native ✔ ○ Fenced ● Summary Agenda
  70. 70. Fenced
  71. 71. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() write() write() lock() process blocked
  72. 72. Application Fenced Distributed Locking token = lockAndGetToken() Hazelcast Cluster (CP Subsystem Enabled) Shared Resource token 1 2
  73. 73. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager
  74. 74. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock()
  75. 75. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11
  76. 76. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked
  77. 77. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock()
  78. 78. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock() lock held by app 2 token: 12
  79. 79. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock() lock held by app 2 token: 12 write() token: 12
  80. 80. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock() lock held by app 2 token: 12 write() token: 12
  81. 81. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock() lock held by app 2 token: 12 write() token: 12 write() token: 11
  82. 82. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() write() token: 11 write() token: 12 lock() process blocked token: 11 token: 12 rejected
  83. 83. Code: Fenced Distributed Locking var config = new ClientConfig(); config.getNetworkConfig().addAddress("hazelcast"); var hazelcast = HazelcastClient.newHazelcastClient(config); var lock = hazelcast.getCPSubsystem().getLock("my-lock"); long token = lock.lockAndGetFence(); writetoSharedResource(token); lock.unlock(); Server Application helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/ helm install hazelcast hazelcast/hazelcast --set hazelcast.yaml.hazelcast.cp-subsystem.cp-member-count=3
  84. 84. Summary: Fenced Distributed Locking Efficiency Correctness Complexity Performance
  85. 85. ● Introduction ✔ ● Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ✔ ○ RedLock ✔ ○ Consensus-Based ✔ ○ Kubernetes Native ✔ ○ Fenced ✔ ● Summary Agenda
  86. 86. Summary
  87. 87. Efficiency Performance Complexity Correctness Single-Instance Multi-Instance RedLock Consensus-Based Kubernetes Native Fenced
  88. 88. Resources ● Code for this presentation: https://github.com/leszko/distributed-locking ● How to do distributed locking: https://martin.kleppmann.com/2016/02/08/how-to-do-distrib uted-locking.htm ● Distributed Locks are Dead; Long Live Distributed Locks! https://hazelcast.com/blog/long-live-distributed-locks/
  89. 89. Thank You! Rafał Leszko @RafalLeszko rafalleszko.com

×