Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Leadership Election with Spring Cloud Cluster

3,317 views

Published on

SpringOne Platform 2016
Speaker: Dave Syer; Senior Consulting Engineer, Pivotal

Leader election allows application to work together with other applications to coordinate a cluster leadership via a third party system. A leader can then be used to provide global state or global ordering, generally without sacrificing availability. In this presentation we show how Spring Cloud Cluster provides a simple abstraction for leader election and how it is implemented using zookeeper, hazelcast and etcd.

Published in: Technology
  • Be the first to comment

Leadership Election with Spring Cloud Cluster

  1. 1. Locks and Leaders with Spring Integration Dave Syer, 2016  Twitter: @david_syer (see also @artem_bilan, @gprussell)  Email: dsyer@pivotal.io
  2. 2. Agenda What is a distributed lock? What is a leader election? What is it good for? How do I do it? Some tools for thinking about failure Warning: Here be Dragons!
  3. 3. A Simple Distributed Operation
  4. 4. Two Clients
  5. 5. Shuffle!
  6. 6. Shuffle!
  7. 7. Shuffle!
  8. 8. Shuffle!
  9. 9. Shuffle!
  10. 10. Shuffle!
  11. 11. Scary, Eh?
  12. 12. Locks Example code using java.util.concurrent.locks.Lock: boolean acquired = false; try { acquired = lock.tryLock(10, TimeUnit.SECONDS); if (acquired) { // Do something unique! } } catch (InterruptedException e) { Thread.currentThread().interrupt(); throw new RuntimeException("Interrupted"); } finally { if (acquired) { lock.unlock(); } }
  13. 13. Spring Integration: LockRegistry public interface LockRegistry { Lock obtain(Object lockKey); }
  14. 14. Locks with Spring Integration Example code using LockRegistry: boolean acquired = false; try { acquired = lock.tryLock(10, TimeUnit.SECONDS); if (acquired) { // Do something unique! } } catch (InterruptedException e) { Thread.currentThread().interrupt(); throw new RuntimeException("Interrupted"); } finally { if (acquired) { lock.unlock(); } } (same code)
  15. 15. Locks and Leases A distributed lock nearly always has a shelf life (it expires). Technically, that makes it a "lease". Without expiry system can't make progress when a lock holder dies.
  16. 16. Dragons (At least) two problems are lurking: 1. Acquiring a lock requires consensus. 2. Leases expire and holder can never be sure when that happens. Read this: http://martin.kleppmann.com/2016/02/08/how­to­do­distributed­locking.html TL;DR If using a lock for "correctness" not "efficiency" be very careful. Important: you can tune the system to adjust the probability, or how long it lasts, but fundamentally you cannot prevent the system from ever allowing more than one holder of a lock.
  17. 17. Leader Elections Simple idea: if you hold a lock you are the leader. What can you do with it? Highly available globally unique things, often with messages sequences message aggregation scheduling, e.g. cron service
  18. 18. Spring Integration: Leader Initiator Implementations of leader election need to be able to start an election and fire events on granted and revoked. Zookeeper Hazelcast Etcd(*) Generic (lock­based) For a user it looks like this (create a new bean which is a SmartLifecycle): @Bean public LeaderInitiator leaderInitiator(CuratorFramework client, Candidate candidate) { return new LeaderInitiator(client, candidate); } (*) No support for etcd v3. Probably dead.
  19. 19. Spring Integration: Callbacks Callbacks on leadership events: public interface Candidate { void onGranted(Context ctx) throws InterruptedException; void onRevoked(Context ctx); ... } See also: @EventListener(OnGrantedEvent.class) public void start() { } @EventListener(OnRevokedEvent.class) public void stop() { }
  20. 20. Closing When to use locks and leaders: HA active/passive failover How to avoid it: latency from restarting a failed app might be fine Otherwise, be idempotent. For correctness, de­duplication store has to be ACID. Relational databases can be really useful, as is Zookeeper. Careful with the physics.
  21. 21. Links http://presos.dsyer.com/decks/locks­and­leaders.html Sample code: https://github.com/SpringOnePlatform2016/dsyer­locks­and­leaders Spring Integration: https://github.com/spring­projects/spring­integration Spring Cloud Cluster: https://github.com/spring­cloud/spring­cloud­cluster

×