A modern computing system requires multiple components distributed on different machines to provide scalability, high availability, fault tolerance, and low latency. Therefore, consensus is essential when communicating between nodes to agree on some data value required during computation. There are many examples of consensus around our tooling: - Google Chubby uses Paxos, which is a consensus algorithm. - Kubernetes uses etcd as the backing store for all cluster data. Etcd uses Raft, which is a consensus algorithm. - Hadoop; Kafka uses ZooKeeper for service discovery; leader’s election, ... ZooKeeper uses a Paxos-variant algorithm. - Blockchain cannot exist without consensus algorithms such as Proof-of-work; Proof-of-stake; ... As the result, knowledge of consensus in the distributed system is crucial to understand the behavior of those systems. This presentation will: Brief introduction about the consensus; challenges and achieved result. Raft algorithm - the main consensus algorithm used in most of the recent systems. Some references to get our hand wet on this topic: https://medium.com/@isuruboyagane.16/what-is-consensus-in-distributed-system-6d51d0802b8c https://www.youtube.com/watch?v=5m3eBWKjHtM&ab_channel=HasgeekTV

1. Consensus and
Raft Algorithm
Huynh Quang Thao
Zendesk
Consensus and
Raft Algorithm

2. Part 1: Consensus

3. Introduction to consensus

4. Two General’s Problem
Army 1 Army 2 Result
Not Attack Not Attack Nothing
Not Attack Attack Lost
Attack Not Attack Lost
Attack Attack Win

5. Two General’s Problem
1. Attack on 01/01

6. Two General’s Problem
1. Attack on 01/01
2. Accept

7. Two General’s Problem
1. Attack on 01/01
2. Accept

8. Two General’s Problem
1. Attack on 01/01
2. Accept
3. Accept the “Accept”

9. Two General’s Problem
1. Attack on 01/01
2. Accept
3. Accept the “Accept”

10. Two General’s Problem
1. Attack on 01/01
2. Accept
3. Accept the “Accept”
4. Wait for the reply
2 generals cannot have a consensus on this problem.

11. Byzantine General’s Problem

12. Matching concepts
Message: hello world
1. Attack on 01/01

13. Matching concepts: 2 general problems
Message: hello world
Link failure problem

14. Matching concepts: Byzantine problems
Message: hello world
Node failure problem

15. Matching concepts: Byzantine problems
Message: hello world
Byzantine
Crash-recovery
Crash-stop
Omission
Byzantine
Crash-recovery

16. Distributed System is hard
Message: hello world
- Messages can be dropped anytime
- Messages can be delayed with any amount
- Messages can be received at any orders
- Messages can be modi
f
ied by any strangers
Byzantine
Crash-recovery
Crash-stop
Omission
Byzantine
Crash-recovery

17. Consensus
A distributed consensus ensures a consensus of data among nodes in a distributed system or reaches an
agreement on a proposal
https://computersciencewiki.org

18. FLP Impossibility
The FLP theorem states that in an asynchronous network where messages may be delayed but not lost, there is no
consensus algorithm that is guaranteed to terminate in every execution for all starting conditions, if at least one
node may fail-stop.
Michael [F]ischer, Nancy [L]ynch and Michael [P]aterson - 1985
TL,DR: We cannot develop consensus algorithms that work in all situations in a very generic model
Or
We cannot have any consensus algorithms … :(

19. FLP Impossibility
The FLP theorem states that in an asynchronous network where messages may be delayed but not lost, there is no
consensus algorithm that is guaranteed to terminate in every execution for all starting conditions, if at least one
node may fail-stop.
Michael [F]ischer, Nancy [L]ynch and Michael [P]aterson - 1985
We need to
f
ind a way to relax the model
• asynchronous network : no bound-time on receiving messages. Solution: timeout
• guaranteed to terminate in every execution: allow some situations, the algorithm cannot be terminated (hash
function)
• All staring conditions: force the system to some de
f
ined states before running

20. Protocols Applications
Paxos
- Multi Paxos
- Flexible Paxos
- Cheap Paxos
- PigPaxos
- …
• Google Chubby:
coarse grained lock service
• Google Spanner: Distributed Database
ZooKeeper Atomic Broadcast
Apache ZooKeeper:
• a centralised service for maintaining con
f
iguration
information, naming synchronization.
• Application: Ka
f
ka; Hadoop; …
Raft
• etcd: Distributed Key-value datastore (heart of
Kubernetes)
• CockroachDB (New-SQL Database)
• TIDB
Proof Of Work Blockchain such as Bitcoin; Ethereum

21. Protocols Implementations
Paxos
- Multi Paxos
- Flexible Paxos
- Cheap Paxos
- PigPaxos
- …
• Google Chubby:
coarse grained lock service
• Google Spanner: Distributed Database
ZooKeeper Atomic Broadcast
Apache ZooKeeper:
• a centralised service for maintaining con
f
iguration
information, naming synchronization.
• Application: Ka
f
ka; Hadoop; …
Raft
• etcd: Distributed Key-value datastore (heart of
Kubernetes)
• CockroachDB (New-SQL Database)
• TIDB
Proof Of Work Blockchain such as Bitcoin; Ethereum

22. Part 2: Raft Algorithm

24. Architecture

25. Key Value
Raft
Finite State Machine
Node
Flow diagram when user sends a request to a follower node
Log sequences

26. Key Value
Raft
Key Value
Raft
Key Value
Raft
Leader
Follower 1
Follower 2
call
denied
leader: 127.0.0.2
Flow diagram when user sends a request to a follower node

27. Key Value
Raft
2. call
Key Value
Raft
Key Value
Raft
3. replicate
3. replicate
4. return
to FSM
1. Put K/V
5. return
Leader
Follower 1
Follower 2
waiting
quorum
Flow diagram when user sends a request to a leader node

28. Raft Decomposition
Leader election
- Select one server to act as leader
- Detect crashes, choose new leader
Log replication (normal operation)
• Leader accepts commands from clients, appends to its log
• Leader replicates its log to other servers (overwrites inconsistencies)
Safety
• Keep logs consistent
• Only servers with up-to-date logs can become leader
Source: https://raft.github.io/slides/uiuc2016.pdf

29. Raft State
Follower: passive, issue no requests on their own but simply respond to requests from leaders and candidates (except
redirect requests to leader)
Candidate: Middle state. Try to become a leader
Leader:
• Active node in the cluster.
• Handle requests from clients + Replicate logs to other nodes
• Possibly more than one node think themselves leader

30. Term
• Term is the natural number; acted as a logical timestamp.
• At most one leader per term

31. Log
• Log Entry: data + index + term
• Committed entry: When a log entry is replicated on more
than half of nodes (include itself), log entry will be converted
to commit state.
• When log entry is converted to the commit state, we can
safely apply to FSM and return to clients.

32. Request Type
Raft only has 2 types of request
RequestVote: use for leader election.
AppendEntries:
• Use of heartbeat protocol.
• Log replication: similar as heartbeat, and include log entries need to replicate.

33. Request Type: AppendEntries RPC
Protocol works by round. a.k.a: term
Hence, because of network latency, some messages
come late when other nodes see higher term -> The
request is rejected.
At the same time, node returns the newer term for
candidate to update itself.

34. Request Type: AppendEntries RPC
Follower uses leaderCommit ID to decide which log entry has
not been committed -> commit into FSM.

35. Request Type: AppendEntries RPC
2 parameters to solve the inconsistent log problem
(explain later)

36. Request Type: RequestVote RPC
Protocol works by round. a.k.a:
term
Hence, because of network
latency, some messages come
late when other nodes see
higher term -> The request is
rejected.
At the same time, node returns
the newer term for candidate to
update itself.

37. Request Type: RequestVote RPC
2 important parameters to decide if a
candidate’s log is up-to-date
(Explain later)

38. Leader Election

39. Leader
Follower
Follower
Follower
1
1
1
1
Heartbeat
Heartbeat
Heartbeat
Leader will send heartbeart to every node
For starting, all nodes are followers

40. Leader
Follower
Follower
Follower
1
1
1
1
Heartbeat
Heartbeat
Where are you, bro?
One node doesn’t receive the heartbeat until timeout

41. Leader
Candidate
Follower
Follower
2
1
1
1
Heartbeat
Heartbeat
He ’s dead. I want to become the leader
- Increase term to 1
- Move to the candidate state

42. Candidate
Follower
Follower
2
1
1
1
He ’s dead. I want to become the leader
Candidate will send out the RequestVote with the new term
RequestVote. Term = 2
RequestVote. Term = 2
R
e
q
u
e
s
t
V
o
t
e
.
T
e
r
m
=
2
Leader

43. Candidate
Follower
Follower
2
2
1
1
He ’s dead. I want to become the leader
Because 2 > 1: Accept. -> Update term
Note: With an “up-to-date” condition that we haven’t mentioned yet
R
e
q
u
e
s
t
V
o
t
e
.
T
e
r
m
=
2
Leader

44. Candidate
Follower
Follower
2
2
2
1
He ’s dead. I want to become the leader
Because 2 > 1: Accept. -> Update term. Also move back to the follower state
Note: With an “up-to-date” condition that we haven’t mentioned yet
RequestVote. Term = 2
Follower

45. Candidate
Follower
Follower
2
2
2
1
He ’s dead. I want to become the leader
Some node might not receive the request ….
Follower
RequestVote. Term = 2

46. Candidate
Follower
Follower
2
2
2
1
I received total 3/4 votes (include myself)
Happy case: Candidate receives a majority of votes
Follower
Scenario: Success voting

47. Leader
Follower
Follower
2
2
2
1
I am a leader
Follower
Heartbeat. Term = 2
Heartbeat. Term = 2
h
e
a
r
t
b
e
a
t
.
T
e
r
m
=
2
Scenario: Success voting

48. Candidate
Follower
Follower
2
2
2
1
I received only 2/4 votes (included myself)
Bad case: Candidate receives less than majority of nodes
Follower
Scenario: Unsuccess voting

49. Candidate
Follower
Follower
3
2
2
1
Only the weak give up
Candidate will continue to increase their term and reinitialise the voting
Follower
RequestVote. Term = 3
RequestVote. Term = 3
R
e
q
u
e
s
t
V
o
t
e
.
T
e
r
m
=
3
Scenario: Unsuccess voting

50. Leader
Follower
Follower
2
2
2
2
How long did I sleep :(
Follower
Heartbeat. Term = 2
Heartbeat. Term = 2
h
e
a
r
t
b
e
a
t
.
T
e
r
m
=
2
0
An isolated node after some times, detect timeout and want become a leader
Where is my leader ???
Follower
Scenario: Voting with smaller term

51. Leader
Candidate
Follower
2
2
2
2
Follower
RequestVote. Term = 1
Heartbeat. Term = 2
h
e
a
r
t
b
e
a
t
.
T
e
r
m
=
2
1
An isolated node after some times, detect timeout and want become a leader
I want to become a leader
Follower
Scenario: Voting with smaller term

52. Leader
Candidate
Follower
2
2
2
2
Follower
Heartbeat. Term = 2
h
e
a
r
t
b
e
a
t
.
T
e
r
m
=
2
1
Any request vote that term is less than the current term will be rejected
And return back with a newer term
I want to become a leader
Follower
Rejected. Term = 2
Scenario: Voting with smaller term

53. Leader
Follower
Follower
2
2
2
2
Follower
Heartbeat. Term = 2
h
e
a
r
t
b
e
a
t
.
T
e
r
m
=
2
2
When receiving rejected with larger term, back to the follower state with newer term.
Other guy is a leader :(
Follower
Rejected. Term = 2
I must wait for him
Scenario: Voting with smaller term

54. Leader
Follower
Follower
2
2
1
2
Heartbeat
H
e
a
r
t
b
e
a
t
Candiate
OMG. Look like I just fall asleep
Scenario: 2 nodes think they’re leaders

55. Leader
Follower
Follower
2
2
1
2
Heartbeat
H
e
a
r
t
b
e
a
t
Leader
OMG. Look like I just fall asleep
Back to my work
Term = 1
Bring me some tea
Scenario: 2 nodes think they’re leaders

56. Leader
Follower
Follower
2
2
1
2
H
e
a
r
t
b
e
a
t
Leader
OMG. Look like I just fall asleep
Back to my work
Term = 1
Bring me some tea
WTF. What is this guy thinking?
Every request has term less than current term will be rejected
Scenario: 2 nodes think they’re leaders
Rejected
Term = 2

57. Leader
Follower
Follower
2
2
2
2
H
e
a
r
t
b
e
a
t
Follower
I am not the leader anymore :(
Term = 1
Bring me some tea
Update new term and back to the follower
Scenario: 2 nodes think they’re leaders
Rejected
Term = 2

58. Leader
Follower
Follower
2
2
4
2
Candidate
Heartbeat
h
e
a
r
t
b
e
a
t
Scenario: Full picture of voting
4
I want to be a leader
Vote for me !!!

59. Leader
Follower
Follower
2
2
4
2
Candidate
Heartbeat
h
e
a
r
t
b
e
a
t
Scenario: Full picture of voting
4
I want to be a leader
Vote for me !!!
Voting conditions:
- Candidate’s term >= own’s term
- Candidate’s log must be more “up-to-date”
Let me check carefully

60. Leader
Follower
Follower
2
2
4
4
Candidate
Heartbeat
h
e
a
r
t
b
e
a
t
Scenario: Full picture of voting
4
I want to be a leader
Vote for me !!!
Even rejected, nodes always update to the largest term they have ever seen
Bigger term doesn’t imply
You’re correct
Rejected

61. Leader
Follower
Follower
2
2
4
4
Scenario: Full picture of voting
4
Leader sees a new larger term while sending heartbeat
Heartbeat
Bigger term. Time for me to resign
Follower

62. Follower
Follower
4
2
4
4
Scenario: Full picture of voting
4
Leader moves back to follower state. Even there is no problem in the network
Follower
Follower

63. “up-to-date” condition
candidate's last log entry term > follower's last log entry term
OR
candidate's last log entry term == follower's last log entry term AND candidate's log length > follower's log length

64. https://blog.cloud
f
lare.com/a-byzantine-failure-in-the-real-world/
4

65. Leader Election
• When one node does not receive heartbeat requests from leader, that node
will move from Follower -> Candidate state.
• Candidate will increase its term to 1, then it will send RequestVote for all
nodes in the system.
• When receiving RequestVote request that has a larger term (with the up-to-
date condition): accept and move back to follower
When candidate receives the result:
• Exist one result has larger term: convert back to follower
• Timeout and less than majority of votes: increase term by one and reinitialise the leader election.
• Majority of votes:
• Move from candidate to leader
• Start to send heartbeat to other nodes to maintain the leader status
• Start the log replication phrase

66. Question & Answer
Question: Are there any scenario that the leader election happens without the termination ?
Answer: Yes
• When the leader is down, possibly all followers recognises at the same time and move to the candidate state.
Leader election happens, but no node has the majority.
• After timeout, all candidates at the same time reinitialise the leader election at the new term.
• This process happens again and again. And no node can become a leader.
Solution Raft uses random timeout.
When starting the voting, candidate will randomise a timeout.
Because of random timeout, there will be one node timeout
f
irst and reinitialise leader election
f
irst at the new term.
This node will likely become a leader.

67. Log Replication

68. Node 2
Node 1
M
10
10
Log Matching Property
Statement 1: If 2 entries have same index and term, these 2 entries have same data. We can say they are identical.
Index 4

69. Node 2
Node 1
M
10
10
Log Matching Property
Statement 1: If 2 entries have same index and term, these 2 entries have same data. We can say they are identical.
Statement 2: if 2 log entry are identical at index T, all log entries from the beginning to T are identical.
Index 4

70. 1 3
2
1 3 4 5 6
Leader
Follower
2
M
4 5

71. 1 3
2
1 3 4 5 6
Leader
Follower
2
M
4 5
1 3 4 5 6
Leader
Follower
2
1 3 4 5 6
2
Sending request
Duplicate all logs Brute-force approach

72. 1 3
2
1 3 4 5 6
Leader
Follower
2
M
4 5
Sending request

73. 1 3
2
1 3 4 5 6
Leader
Follower
2
M
4 5
Sending request
Compare 2 log entries at index 5

74. 1 3
2
1 3 4 5 6
Leader
Follower
2
M
4 5
1 3
2
1 3 4 5 6
Leader
Follower
2
4
After updating state
Follower removes inconsistent log
at the index 5
Sending request

75. 1 3
2
1 3 4 5 6
Leader
Follower
2
M
4
Sending request

76. 1 3
2
1 3 4 5 6
Leader
Follower
2
M
4
1 3
2
1 3 4 5 6
Leader
Follower
2
Sending request
After updating state
Follower removes inconsistent log
at the index 4

77. 1 3
2
1 3 4 5 6
Leader
Follower
2
M
Sending request

78. 1 3
2
1 3 4 5 6
Leader
Follower
2
M
1 3
2
1 3 4 5 6
Leader
Follower
2
4 5 6
After updating state
Sending request
- Follower observes all logs are matched
- Follower applies all logs from index 3 -> 6 to the local log

79. 1 3
2
1 3 4 5 6
Leader
Follower
2
M
1 3
2
1 3 4 5 6
Leader
Follower
2
4 5 6
4 5 6
7
7
7
After updating state
Sending request
- Follower observes all logs are matched
- Follower applies log at index 7

80. Other problems
Log Compaction
• Growing log in
f
inity is not feasible in the real world.
• We should provide the mechanism that after updating log into FSM, we can remove the old log.
Example: [x -> 1; x -> 2]
• Challenge: if we accidentally remove log entry that has not replicated to other nodes yet, we cannot
recover.
Con
f
iguration Changes
• Raft paper doesn’t have a speci
f
ication for this.
Byzantine Problem
• Every node must trust other nodes in the system.
• Scenario: one “faulty node” wants to become a leader: always reply “accept” for all RequestVote
request, but at the same time, never accept that request at the local and send RequestVote for other
node with a larger term.

81. References
Paper: In Search of an Understandable Consensus Algorithm
Original paper: Raft in detail and how to implement.
Paper: Raft Re
f
loated: Do We Have Consensus?
Based on the above paper with more explanation.
Paper: Paxos vs Raft: Have we reached consensus on distributed consensus?
Explain again multi-Paxos based on Raft style (more engineering focus). Through this paper, we observe that there is not
much differences between Paxos and raft.
MIT Course 6.824: Distributed System
Good source to learn distributed system fundamentals with assignments
Raft user study: Raft lecture
https://www.youtube.com/watch?v=YbZ3zDzDnrw
Raft user study: Paxos lecture
https://www.youtube.com/watch?v=JEpsBg0AO6o
https://github.com/hashicorp/raft
Production grade implementation for the Raft algorithm.
https://hqt.github.io/2020-10-10-raft-proof/
Raft proof in layman term (Vietnamese version)

82. Bring home …
- Consensus is interesting (don’t?) and hard
- There is no “real” consensus algorithm, must accepted some constraints.
- We should implement a consensus algorithm once, but must use a production-grade libraries.
- Raft is the way to go !!! (Not actually true)
- Bitcoin’s price is all-time high

83. Q&A Session