Download to read offline
Uploaded byEnsar Basri Kahveci
java.util.concurrent for Distributed Coordination - Devoxx Ukraine 2019
The evolution of distributed coordination tools shows that high-level APIs ease implementation of coordination tasks, such as leader election, locking, synchronized actions. For instance, the Chubby paper highlights familiarity of lock-based interfaces. Similarly, Apache Curator hides complexity of ZooKeeper recipes behind Java APIs, while etcd and Consul implement concurrency primitives on their own. A different path in this journey would be extending the long-lasting java.util.concurrent APIs, such as Lock, Semaphore, etc. Simplicity of these APIs makes them very useful in distributed coordination use cases. Join this talk to explore Hazelcast’s brand new implementation of java.util.concurrent APIs on top of the Raft consensus algorithm. I will walk through code samples to demonstrate how Java locks, semaphores, etc. can be used in distributed environments that involve partial failures. I will also share our experience of how we coped with several challenges we faced while developing and testing our Raft implementation.
java.util.concurrent for Distributed Coordination - Devoxx Ukraine 2019
- 1. java.util.concurrent for distributed coordination DevoxxUkraine 2019 Ensar Basri Kahveci Distributed Systems Engineer @ Hazelcast @ metanet
- 2. @metanet Hazelcast The leading opensource Java IMDG Distributed Java collections, concurrency primitives, messaging Caching, application scaling, distributed coordination 🎉 Hazelcast Cloud https://hazelcast.cloud #devoxxua2019
- 3. @metanet Agenda What is distributedcoordination? How distributed coordination APIs evolved over time? Why to use java.util.concurrent.* Demo on Hazelcast IMDG 3.12 #devoxxua2019
- 4.
- 5. @metanet Distributed Coordination Leader election Synchronization Servicediscovery Configuration management for microservices #devoxxua2019
- 6. @metanet Challenges Concurrency problems Partial failuresin distributed systems Consistency issues #devoxxua2019
- 7.
- 8. @metanet Consensus algorithms underthe hood CP with respect to CAP Deployed as a central service APIs for coordination tasks Distributed Coordination Systems #devoxxua2019
- 9. @metanet Distributed Coordination Systems GoogleChubby (Paxos) Apache ZooKeeper (ZAB) etcd (Raft) #devoxxua2019
- 10. @metanet APIs for Coordination:Layer #1 Some form of KV store API Suitable for configuration management and service discovery Foundational layer for building higher-level APIs /services /payment /product /photo /services /services/payment /services/product /services/product/photo Chubby & ZooKeeper etcd #devoxxua2019
- 11. @metanet APIs for Coordination:Layer #2 Chubby: Locking APIs on files ZooKeeper: Recipes Algorithms to build high-level primitives on top its FS “Friends don't let friends write ZK recipes.” Apache Curator Tech Notes #6 etcd: Leader election and distributed lock primitives on top of its KV store #devoxxua2019
- 12. @metanet Need for High-levelAPIs KV-store / low-level file-system API is easy to misuse, not suitable for all coordination tasks. “Consensus in the Cloud: Paxos Systems Demystified” High-level APIs minimise guesswork and development effort. java.util.concurrent.* in JDK #devoxxua2019
- 13. @metanet Distributed Coordination Systems GoogleChubby (Paxos) Apache ZooKeeper (ZAB) etcd (Raft) Hazelcast IMDG 3.12 CP Subsystem (Raft) #devoxxua2019
- 14. @metanet Hazelcast IMDG 3.12CP Subsystem IAtomicReference Configuration and metadata management for basic use-cases IAtomicLong, ICountDownLatch, ISemaphore Distributed synchronization FencedLock Distributed locking and leader election #devoxxua2019
- 15. @metanet Hazelcast IMDG 3.12CP Subsystem Well-defined failure semantics Strong consistency CP with respect to CAP DIY-style tested with Jepsen #devoxxua2019
- 16. @metanet Strong consistency Handles crashesand network failures. Operational as long as the majority is up. Consensus Node Count Majority Degree of Fault Tolerance 2 2 0 3 2 1 4 3 1 5 3 2 6 4 2 7 4 3 #devoxxua2019
- 17. @metanet Understandability as theprimary goal Runtime concerns (snapshotting, dynamic membership) Performance optimizations (fast reads, batching) https://raft.github.io Why Raft? #devoxxua2019
- 18.
- 19. @metanet https://github.com/metanet/juc-talk DEMO #1: LeaderElection #devoxxua2019
- 20. @metanet CP Sessions A sessionstarts on the first lock / semaphore request. Session heartbeats are periodically committed in the background. If no heartbeat for some time (session TTL), the session is closed. Auto-release mechanism for FencedLock and ISemaphore #devoxxua2019
- 21. @metanet CP sessions offera trade-off between safety and liveness. #devoxxua2019
- 22. @metanet “How to dodistributed locking” “Distributed locks are dead; long live distributed locks!” DEMO #2: Fencing-off Stale Lock Holders #devoxxua2019
- 23. @metanet Replacing crashed CPmembers to preserve high availability Demo #3: Handling Server Failures #devoxxua2019
- 24. @metanet Recap Avoid writing yourown implementations for coordination. High-level APIs minimise guesswork and development effort. java.util.concurrent.* FTW! Operational simplicity matters. Dynamic clustering Horizontal scalability #devoxxua2019
- 25. @metanet Current Status In Hazelcast4.0: CP Subsystem disk persistence Metrics In roadmap: KV Store Event Listeners More client languages #devoxxua2019
- 26. @metanet Resources https://github.com/metanet/juc-talk (demos) Hazelcast IMDGDocs CP Subsystem Code Samples https://hazelcast.com/blog/author/ensarbasri Hazelcast IMDG 3.12 #devoxxua2019
- 27. Дякую! Devoxx Ukraine 2019 #DevoxxUA EnsarBasri Kahveci Distributed Systems Engineer @ Hazelcast @ metanet