Talk at SF Big Analytics https://www.meetup.com/sf-big-analytics/events/285731741/ Distributed systems are made up of many components such as authentication, a persistence layer, stateless services, load balancers, and stateful coordination services. These coordination services are central to the operation of the system, performing tasks such as maintaining system configuration state, ensuring service availability, name resolution, and storing other system metadata. Given their central role in the system it is essential that these systems remain available, fault tolerant and consistent. By providing a highly available file system-like abstraction as well as powerful recipes such as leader election, Apache Zookeeper is often used to implement these services. Although powerful, the Zookeeper interface may not be flexible enough or provide sufficient performance for all applications and many systems are replacing Zookeeper based solutions with Raft which provides a more generic interface to high availability and fault tolerance through the use of State Machine replication. This talk will go over a generic example of stateful coordination service moving from Zookeeper to Raft. Speaker: Tyler Crain ( Alluxio) Tyler Crain is a software engineer at Alluxio, working on distributed systems within the Alluxio core team. Before this, Tyler held Post-Doc positions at the University of Sydney and Sorbonne Universities where he performed research on topics including distributed key-value stores, distributed consensus and blockchain. Tyler received his PhD from the University of Rennes where he worked on Transactional Memory. He also holds a Masters degree in Computer Science from University of California Santa Barbara. talk at SF Big Analytics: Related Blog: https://www.alluxio.io/blog/from-zookeeper-to-raft-how-alluxio-stores-file-system-state-with-high-availability-and-fault-tolerance/

1. Zookeeper vs Raft: Stateful
distributed coordination with
HA and Fault Tolerance
Tyler Crain
ALLUXIO 1

2. Outline
1. Stateful distributed configuration services
2. What is Alluxio
3. High availability and fault tolerance in Alluxio
a. Implementation 1: Zookeeper + UFS (object store)
b. Implementation 2: Raft
2

3. ALLUXIO 3
Stateful distributed
coordination services

4. 4
Source: kubernetes.io
Kubernetes - Container orchestration

5. Kafka - distributed event streaming
5

6. 6
Source: Zhou, Jingyu, et al. "Foundationdb: A distributed unbundled
transactional key value store." Proceedings of the 2021 International
Conference on Management of Data. 2021.
Foundation DB - Transactional Key-Value store

7. 7
Source: Corbett, James C., et al. "Spanner: Google’s globally distributed
database." ACM Transactions on Computer Systems (TOCS) 31.3 (2013): 1-22.
Google Spanner - distributed database

8. 8
Source: Shute, Jeff, et al. "F1: A distributed SQL database that scales." (2013).
Google F1 (Spanner) - SQL Database

9. 9
Source: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html
HDFS - Distributed File System

10. What we have seen
● Metadata
● Configuration
● Cluster health/maintenance
● Naming/service discovery
● Leader Election
10
Small subset of system, but needs strong properties
● High availability
● Fault tolerance
● Strong consistency

11. ALLUXIO 11
Alluxio

12. Alluxio
12

13. Alluxio Architecture
13
● UFS (Under File System)
○ E.g. Amazon S3, HDFS, etc
○ Object stores

14. Fault tolerance
- Workers can be replicated, act as a cache for the various UFS
- Many UFS have high availability, fault tolerance guarantees
- Master becomes single point of failure
14

15. Journal details
15
- Total order log of state changes
- Can recover state by replaying the
operations
- Snapshots to efficiently store state
- Faster recovery
- Smaller size

16. Basic fault tolerance
- Create a fault tolerant journal
- If the master crashes
- Stat a new master
- Replay the journal
- Start serving clients
- Detecting a failure, starting a new node, and replaying the journal takes time
- The system will be unavailable during this time
16

17. Basic high availability
- Run multiple masters
- A primary master will serve requests
- Secondary master(s) will replicate the state of the primary master, and take
over in case of failure
17

18. Basic highly available/ fault tolerant architecture
18

19. Problems to solve
- Ensure a single primary master running at all times
- Journal needs to be
- Fault tolerant
- Masters must agree on a single valid order of journal entries
- Consensus
19

20. ALLUXIO 20
Implementation 1:
Zookeeper + UFS Journal

21. Ensure a single primary master running at a time
- Leader election using Zookeeper
- Apache Zookeeper an open-source server which enables highly reliable
distributed coordination
- Provides a file-system (hierarchical namespace) like abstraction built on top of an Atomic
Broadcast (consensus) protocol
- Run on a cluster of nodes to provide fault tolerance/high availability
21

22. Apache Zookeeper Curator leader election recipe
- Each master subscribes to the leader election recipe
- The recipe will elect one of the nodes leader
- If the leader fails or is unable to be reached due to network issues, the recipe
will elect a new leader
22

23. UFS Journal
- Write journal entries to the UFS
- Use the availability / fault tolerance guarantees of the UFS
23

24. Does leader election solve all our problems?
- Not quite
- The Zookeeper recipe will do its best to ensure only a single node is chosen
leader at a time, but due to asynchrony in the system two nodes may believe
they are leader at the same time
- Multiple nodes may be trying to write to the journal concurrently
- Note leader election is sufficient to provide consensus, but is not a consensus protocol itself
24

25. Defer to the UFS
- Can use the consistency guarantees of the UFS to ensure only a single writer
to the journal
25
- Alluxio Primary/
secondary master state
transition

26. Ensure a single journal writer to the UFS
- Use a mutual exclusion protocol based on log file names
- (1, 12) (13, 20) (21, 300) (301, MAX_INT)
- New primary master must ensure current log file is “complete” before writing
to a new log
- (1, 12) (13, 20) (21, 300) (301, 327)
- Create a new log
- (1, 12) (13, 20) (21, 300) (301, 327) (328, MAX_INT)
26

27. Zookeeper + UFS architecture
27

28. Issues
- Relies on multiple systems
- Each having their own fault tolerance/availability models
- More complicated
- Different UFS have different consistency models and performance
- May not be efficient for appending log entries
28

29. ALLUXIO 29
Implementation 2:
Raft Journal

30. The journal as a (deterministic) state machine
- Input: command
- Append
- State transition (deterministic):
- Add the journal entry to the log
30

31. Raft - replicated state machine
- Clients interact with the state
machine as if it was a single
instance (linearizability)
- Clients send commands to the
state machine and receive
responses
- But the state machine is fault
tolerant and has high
availability
- Apache Ratis (java
implementation of Raft)
31

32. Forget the Journal as a Log
32
Inode ID Inode
metadata
Worker location
0 … …
1 … …
Key-value store state machine commands:
- Write(key, value)
- Remove(key)
- Read(key) -> value

33. The Alluxio metadata as a replicated state machine
- Raft simplifies the task of
implementing the journal.
- It handles snapshotting and recovery
- The journal just becomes a
key-value store keeping track of the
file-system meta-data
- Raft is colocated with the Alluxio
masters
33

34. RocksDB as the key-value store
- Log-structured merge tree
- Efficient inserts
- Key-value store
- Inode tree as a key-value map
- Alluxio provides an additional in memory cache for fast reads
- Efficient snapshots
34

35. Primary master protocol
- Raft will ensure the journal is
consistent and highly available
- Still want a single primary master to
perform updates to UFS and Alluxio
workers and serve clients
- Take advantage of the leader election
built into Raft.
- An additional layer of coordination is
used along with raft to reduce the
likelihood of concurrent primary masters
35

36. Alluxio + Raft architecture
36

37. Advantages
- Simplicity
- No external systems (raft colocated with masters)
- Journal as a key-value store (higher level abstraction)
- Performance
- Journal operations stored locally on masters
- RocksDB as an efficient key-value store
37

38. From our earlier examples
- Kubernetes - etcd - key/value store run on Raft
- Kafka - coordinator running Zookeeper
- Foundation DB - coordinators running Paxos
- Generally a paxos based replicated state machine of a simple data store with
some custom application logic running over this
38

39. ALLUXIO 39
Questions?

40. More information on Alluxio
- Web - alluxio.io
- Github - https://github.com/Alluxio/alluxio
- Slack - alluxio-community.slack.com
40