Download as PDF, PPTX
![企业云原生架构解决方案提供商/用技术推动商业进步
MegaEase
企业云原生架构提供商
11
Redlock
Distributed locks with Redis
SET resource_name my_random_value NX PX 30000
Not Exist Expired of 30s
Unique across all clients
Algorithm
1. Client get the current time in milliseconds
2. Client tries to acquire the lock in all the N instances sequentially, using the same key name and random
value in all the instances.
3. Successfully acquire the lock - majority Redis nodes accepts && elapsed time < lock validity time.
4. Actual lock validity time = initial validity time - elapsed time
5. If acquire the lock failed, try to unlock all instances
https://redis.io/topics/distlock
if redis.call(“get”,KEYS[1]) == ARGV[1] then
redis.call("del",KEYS[1])
end](https://image.slidesharecdn.com/distributed-210621130714/75/Distributed-Lock-Manager-11-2048.jpg)
Uploaded byHao Chen
Distributed Lock Manager
The document discusses the challenges of maintaining consistency in distributed systems, particularly focusing on locking mechanisms such as distributed lock managers, Redlock, Google Chubby, and Zookeeper. It emphasizes the importance of transaction synchronization and the limitations of sharding data. Key takeaways include the necessity for high availability, strong consistency, and deadlock detection in distributed lock services.
![[오픈소스컨설팅] 쿠버네티스와 쿠버네티스 on 오픈스택 비교 및 구축 방법](https://cdn.slidesharecdn.com/ss_thumbnails/osck8svsk8sonopenstackkhoj-210310051504-thumbnail.jpg?width=640&height=640&fit=bounds)
Distributed Lock Manager
- 1.
- 2. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 Outline What’s topics • What’sthe problem? • Possible Solutions • Distributed Lock Manager • Redlock • Google Chubby • ZooKeeper • Takeaways
- 3. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 Consistency What’s the realproblem is? Order Ship the Order Cancel the Order Seller Buyer How to drive the State Machine? What’s the end state?
- 4. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 4 Order (Canceled) Ship the Order Cancelthe Order Seller Buyer Consistency Why should we need the lock? Within Locked Transaction Ship the Order Order (Canceled) Waiting For Lock Within Locked Transaction Error Invalid State Order (Shipped) Ship the Order Cancel the Order Seller Buyer Within Locked Transaction Order (Return) Waiting For Lock Within Locked Transaction Cancel the Order Return Process
- 5. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 5 Locking Locking Solutions Shard ApplicationCentralized Distributed Lock Manager Database Lock
- 6. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 6 DB Lock MySQL InnoDBLocking Solutions SELECT ... LOCK IN SHARE MODE SELECT ... FOR UPDATE Exclusive Lock Shared Lock Versioning Optimistic Lock SELECT id, data, version FROM table WHERE id = $ID UPDATE table SET data = $data version = version + 1 WHERE id = $ID AND version=$VER if {UPDATE} Failed, go to step 1 else Done! 1 3 4 2 { Business Code }
- 7. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 7 Shard HotSpots indices Operation for Order A Operation forOrder A Shard Data Nodes Operation Gateway Operation Gateway forward f o r w a r d • Using the Shard data Solution • Forward all of the operations to a single node • Consistency can be easy to be guaranteed • Consistent Hash • Gossip the indices Operation for Order A Operation for Order A Operation for Order A forw ard forward fo rw a rd Uber Ringpop https://eng.uber.com/ringpop-open-source-nodejs-library/
- 8. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 8 The Problems ofShard Why the shard is not a good design? Hotspots Multiple Entities Transaction Node Failed Load is not balanced Transfer money from account A to account B Data Replication cause more consistent issues R+W > N CAP Theorem
- 9. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 9 Lock Manager Centralized LockManager? • Distributed Lock Manager (DLM) • https://en.wikipedia.org/wiki/Distributed_lock_manager • Provides a lock-leasing protocol • Transaction Service is stateless • Lock Manager must be designed for HA • No single point failure • The lock state can be persistent. • Dead Lock Detection • Lock expired • Keepalive heart beats DLM Storage Persistence Service lock unlock
- 10. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 10 DLM Lock Modes Sixlock modes Mode Requesting Process Other Processes Null (NL) No access Read or write access Concurrent Read (CR) Read access only Read or write access Concurrent Write (CW) Read or write access Read or write access Protected Read (PR) Read access only Read access only Protected Write (PW) Read or write access Read access only Exclusive (EX) Read or write access No access
- 11. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 11 Redlock Distributed locks withRedis SET resource_name my_random_value NX PX 30000 Not Exist Expired of 30s Unique across all clients Algorithm 1. Client get the current time in milliseconds 2. Client tries to acquire the lock in all the N instances sequentially, using the same key name and random value in all the instances. 3. Successfully acquire the lock - majority Redis nodes accepts && elapsed time < lock validity time. 4. Actual lock validity time = initial validity time - elapsed time 5. If acquire the lock failed, try to unlock all instances https://redis.io/topics/distlock if redis.call(“get”,KEYS[1]) == ARGV[1] then redis.call("del",KEYS[1]) end
- 12. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 12 How to dodistributed locking https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html Challenge Redlock Using time to solve consensus? 5 Redis nodes (A, B, C, D and E) -- 2 Clients (1, 2) A B C D E 1 2 OK OK OK OK OK OK 1. Node C time jumps à Lock expired 2. Client 1 hangs while Node C reply, and become live after Lock expired 3. Client 1 get very long network delay from Node C “the algorithm makes dangerous assumptions about timing and system clocks” Martin Kleppmann
- 13. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 13 Lock Expired How todeal with dead lock ?
- 14. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 14 Lock Expired Making thelock safe with fencing?
- 15. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 15 Is Redlock Safe? A technical debates http://antirez.com/news/101 TL;DR • Fencing is great, with this, no need distributed lock! • Clock jump is a big problem, Redlock cannot work correctly under this problem. • For network delay & client hangs, they all will be fine.
- 16. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 16 Google Chubby Google Chubbylock service • Two Components – Master & Client SDK • Cluster – typically 5 nodes • Replicas - use Paxos protocol to elect a master and replicate the logs • Failover – Master fails, other replicas run the election • Session & Keeplive • Client request or end a session • Lease Interval and Lease Timeout • Performance is average Source: https://www.slideshare.net/romain_jacotin/the-google-chubby-lock-service-for-looselycoupled-distributed-systems (45s) (12s)
- 17. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 17 Zookeeper Lock How touse Zookeeper as lock service 1 2 Create `_locknode/lock-` with Sequence & Ephemeral flag https://zookeeper.apache.org/doc/r3.1.2/recipes.html#sc_recipes_Locks Get children to see to check if I am the lowest sequence number • Yes. Get the Lock • No. Set the watch locknode _c_8a27d9a2-d177-11eb-b8bc-0242ac130003-lock-0000000001 _c_8a27d9a2-d177-11eb-b8bc-0242ac130003-lock-0000000002 Client A Client B
- 18. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 18 Takeaways Summary • Concurrent transactionneed be synchronized • DB Lock is fine, but the Optimistic Lock is great. • Sharding the data cannot solve the all of problem • Distributed Lock Service need the following features: • High Availability • Data Replicas - strong consistent protocol – Pasox, Raft, zab • Master Failover – Leader election • Deadlock Detection • Keepalive & Lease Timeout
- 19.
- 20.