Consensus Algorithms

CONSENSUS ALGORITHMS
YIFAN XING - 2018
DISTRIBUTED
SYSTEMS
Yifan Xing
Consensus Algorithms
@yifan_xing_e

Poll
Kafka:
A distributed
streaming platform
YIFAN XING - 2018 @yifan_xing_e
Consensus Algorithm
e.g. Raft, Paxos
1

Distributed System
YIFAN XING - 2018
What is a Distributed System?
- Components
- Located on different networked computers
- Communicate and coordinate
- Passing messages to each other
@yifan_xing_e
2

Modern Examples
YIFAN XING - 2018
Web DNS Bittorrent
@yifan_xing_e
3

Fun Fact
YIFAN XING - 2018
American Airline
Central Office
Travel Agents
1920s
Cards for each flight
Mark seats sold on cards
@yifan_xing_e
4

Significant challenges/ characteristics of distributed systems are:
Lack of global Knowledge
Time
Concurrency/ Security
Failures
Challenges
5

Challenge 1: Lack of Global Knowledge
YIFAN XING - 2018
No host has global knowledge
How to exchange StateMachines' state information?
Information up-to-date?
How to detect inconsistency?
@yifan_xing_e
6

Challenge 2: Time
YIFAN XING - 2018
Clock skew
Delay/duplicate messages
How to determine what happened first?
@yifan_xing_e
7

Challenge 3: Consistency/ Security
YIFAN XING - 2018
Often will have concurrent operations on a single object
How to ensure object is in consistent state?
Conflicts
@yifan_xing_e
8

Challenge 4: Failures
YIFAN XING - 2018
Can affect each other
How to tolerate failures of components?
Tolerate: detect, handle, recover
@yifan_xing_e
9
A distributed system is one in which the failure of
a computer you didn't even know existed can
render your own computer unusable.
— Leslie Lamport

Achieve consensus between machines
Fault-tolerance => integrity, availability, etc.
Reliably process and execute client commands
Consensus Algorithms
10
Raft Paxos

Paxos

11
Paxos Background
Paxos Made Simple: Basic Paxos
Lynch & Liskov
Leslie Lamport Proved
The Part Time Parliament
Multi-paxos: Paxos + Complexity
Rejected
Published
No mathematical proof
10

12
Paxos Made Simple

Basic Paxos
YIFAN XING - 2018
Proposer: Propose a value
Prepare: Try to propose
Acceptor: accept/ reject value
@yifan_xing_e
13
Consensus: agree on one value

Basic Paxos
14
Proposer
Acceptor
Choose a
Proposal
Number (n)

Basic Paxos
14
Proposor
Acceptor
Choose a
Proposal
Number (n)
Broadcast
the number
to all servers
Prepare(n)

Basic Paxos
14
Proposor
Acceptor
Choose a
Proposal
Number (n)
Broadcast
the number
to all servers
Prepare(n)
If n > maxProposal:
maxProposal = n
promise
Respond Promise
Won't accept proposal with
n' < n

Basic Paxos
14
Proposor
Acceptor
Choose a
Proposal
Number (n)
Broadcast
the number
to all servers
Prepare(n)
Respond Promise
Won't accept proposal with
n' < n
If majority
Accept(n, )

Basic Paxos: Leader Election
14
Proposor
Acceptor
Broadcast
Accept(n, value)

Basic Paxos
14
Proposor
Acceptor
Broadcast
Accept(n, value)
Respond
If n >= maxProposal:
acceptedProposal = maxProposal = n
acceptedValue = value

Basic Paxos
14
Proposor
Acceptor
Broadcast
Accept(n, value)
Respond
If n >= maxProposal:
acceptedProposal = maxProposal = n
acceptedValue = value
If majority:
if any rejection => n is not largest
repeat from beginning
else:
value chosen

Paxos: Proposal Number
15
S0
- server id: unique
- round number:
increment overtime, shared among all servers
servers keep track of highest round number from msgs
Generate new proposal number:
increment maxRound
concatenate with server id

Multi-Paxos
16
A sequence of instances of Basic Paxos
Each instances of Basic Paxos is a log entry
Implementation of Basic Paxos Concepts

Multi-Paxos
17
Client
cmd

Multi-Paxos
17
Client
cmd
Basic Paxos
Choose cmds (values) in log entries

Multi-Paxos
17
Apply cmd (log entries)
1
cmd
1
cmd
1
cmd
2
cmd
2
cmd
3
cmd
3
cmd
4
cmd

Multi-Paxos
17
Client
result

Paxos: Leader Vs. NonLeader
18
S0
If didn't receive Heartbeat from a Higher ID for >= 2T ms:
act as leader
act as proposer
Server with highest ID
Heartbeat for every T ms
Accept requests from client
Leader/ Distinguished Proposer:

Paxos: Leader Vs. NonLeader
18
S0
Server with highest ID
Heartbeat for every T ms
Accept requests from client
Non-leader:
Redirect client requests to leader
act as acceptor
Leader/ Proposer:

Paxos: Leader
19
Unlikely to have two leaders at the same time
Can handle multiple leaders, however, it won’t work as
efficient because of conflicts

Industry Examples
20

Raft

Raft: Background
YIFAN XING - 2018
Simpler version of Paxos
Equivalent: performance & fault-tolerance
Consistency, Conciseness, Correctness
Why understandability: implemented -> useful, extended/ adapted
to the environment, esp. In DS
Understandability
@yifan_xing_e
21
Designed by Diego Ongaro and John Ousterhout at Stanford

Raft
22
Raft: log (!value)

Raft: Phases
YIFAN XING - 2018
1. Select one machine to be a leader
2. Detect crashes, reelection
@yifan_xing_e
23
Leader Election

Raft: Phases
YIFAN XING - 2018
1. Leader processes commands from clients
2. Replicates logs (consistency and consensus among servers)
@yifan_xing_e
23
Leader Election
Log Replication

Raft: Phases
YIFAN XING - 2018
1. Leader processes commands from clients
2. Replicates logs (consistency and consensus among servers)
Keeps log consistent
Only servers with up-to-date logs can be leader
@yifan_xing_e
Leader Election
Log Replication
Safety
23

Raft: Leader Election
24
Become Candidate
CurrentTerm++
Vote for itself
Send
RequestVoteRPC
to other servers
Become Leader:
- Send heartbeats
- Handle requests
Become Follower:
- Redirect requests
M
ajority
Votes
Timeout
RPC from
Leader

Raft: How to ensure election works?
25
At most one leader per term:
- Each server: one vote per term
- Receive majority to win election (N / 2 + 1)
- Example:
S0 S1
S1S0 S0 S0 S1
S2 S3 S4

26
At most one leader per term:
- Each server: one vote per term
- Receive majority to win election (N / 2 + 1)
- Example:
S0
Leader
S1 S2 S3 S4
S1S0 S0 S0 S1

27
There will eventually be a leader:
- Random election timeout (range 100-300ms)
- Usually, one times out first, and win the majority votes
- If two time out at the same time:
- Split vote -> election timeout -> re-enter election state (increment
term, gather votes)

Raft: Log Replication
28
Client
S0
Log

Raft: Leader Appends Entry to Log
28
Client
Log cmd
S0

Raft: Leader Sends AppendEntryRPC
28
Client
Log cmd
S0
AppendEntryRPC

Raft: Followers Send ACKs
28
Client
Log cmd
cmd
cmd
cmd
cmd
S0
ACK

Raft: Replies Client
28
Client
Log cmd
cmd
cmd
cmd
S0
If majority:
- Entry Committed
- Execute
- Return result
cmd

Raft: Notifies Followers of Committed Entry
28
Client
Log cmd
cmd
cmd
cmd
cmd
S0

Raft: Not Majority?
28
Client
Log cmd
cmd
S0
ACK
If not majority:
- Leader retries until succeed

Raft: Log Entry
29
Term:
- Current Term when
receive cmd
Command:
- cmd to execute
1
cmd
S0
Index

Raft: Consistency
Server crushes => log inconsistency
Goal: log consistency. But how?
30
1
cmd
1
cmd
1
cmd
2
cmd
2
cmd
3
cmd
3
cmd
4
cmd
4
cmd
4
cmd
4
cmd
S0
1
cmd
1
cmd
1
cmd
2
cmd
2
cmd
3
cmd
3
cmd
4
cmd
4
cmd
4
cmd
4
cmd
S2
1
cmd
1
cmd
1
cmd
2
cmd
2
cmd
S1

Log Consistency
Always trust leader’s log
No “holes” in log
Repair inconsistency during log replication process
If a given entry is committed, then all preceding entries are also
committed
31

Log Matching Property
If both index and term match:
1. the two entries store the same cmd
2. all previous entries are identical
For each log entry:
compare index and term
32
1
cmd
1
cmd
2
cmd
2
cmd
2
cmd
1
cmd
1
cmd
2
cmd
2
cmd
2
cmd
S0
S1
1
cmd
1
cmd
2
cmd
3
xxx
3
xxxS2
1 2 3 4 5

Log Replication: Ensure Consistency
Check for consistency when sending AppendEntriesRPC
33
AppendEntryRPC:
CurrentEntry: index, term, cmd
PrecedingEntry: index, term
1
cmd
1
cmd
2
cmd
1 2 3 4 5
2
cmd
2
cmd
S0

Goal: append a new entry to log
34
1
cmd
1
cmd
2
cmd
2
cmd
2
cmdS0
1
cmd
1
cmd
2
cmd
2
cmdS1
AppendEntryRPC:
1 2 3 4 5
Receiver checks its own preceding
index and term

34
1
cmd
1
cmd
2
cmd
2
cmd
2
cmdS0
1
cmd
1
cmd
2
cmd
2
cmd
2
cmdS1
If match => append entry
else => rejects request
1 2 3 4 5

34
1
cmd
1
cmd
1
cmd
2
cmd
2
cmdS0
1
cmd
1
cmd
1
cmd
1
cmdS1
1 2 3 4 5
AppendEntryRPC:

34
1
cmd
1
cmd
1
cmd
2
cmd
2
cmdS0
1
cmd
1
cmd
1
cmd
1
cmdS1
1 2 3 4 5
Do not match: REJECT

34
1
cmd
1
cmd
1
cmd
2
cmd
2
cmdS0
1
cmd
1
cmd
1
cmdS1
2
cmd
2
cmd
1 2 3 4 5
Match: Replicate Log Entry

Leader's Log Completeness
How to guarantee completeness of leader’s log?
35
If an entry is committed, all leaders (current + future) must
store the entry

35
Check candidate (potential leader)’s log completeness
Servers with incomplete logs => will not be elected

35
RequestVoteRPC:
term - candidate’s term
candidateId - candidate requesting vote
lastLogIndex - index of candidate’s last log entry
lastLogTerm - term of candidate’s last log entry

(In)complete Log Example - 1
36
RequestVoteRPC:
term - 2
candidateId - 00001
lastLogIndex - 3
lastLogTerm - 1
Candidate
CurrentStateMachine Info:
term - 3
candidateId - 00002
lastLogIndex - 3
lastLogTerm - 1
Current
StateMachine

36
RequestVoteRPC:
term - 2
candidateId - 00001
lastLogIndex - 3
lastLogTerm - 1
Candidate
term - 3
candidateId - 00002
lastLogIndex - 3
lastLogTerm - 1
Current
StateMachine
REJECT VOTE
Candidate.logTerm < my.logTerm => my log is more complete

36
RequestVoteRPC:
term - 3
candidateId - 00001
lastLogIndex - 3
lastLogTerm - 2
Candidate
term - 3
candidateId - 00002
lastLogIndex - 4
lastLogTerm - 2
Current
StateMachine

36
RequestVoteRPC:
term - 3
candidateId - 00001
lastLogIndex - 3
lastLogTerm - 2
Candidate
term - 3
candidateId - 00002
lastLogIndex - 4
lastLogTerm - 2
Current
StateMachine
REJECT VOTE
Candidate.lastLogIndex < my.lastLogIndex => my log is longer

36
RequestVoteRPC:
term - 3
candidateId - 00001
lastLogIndex - 5
lastLogTerm - 2
Candidate
term - 3
candidateId - 00002
lastLogIndex - 4
lastLogTerm - 2
Current
StateMachine

36
RequestVoteRPC:
term - 3
candidateId - 00001
lastLogIndex - 5
lastLogTerm - 2
Candidate
term - 3
candidateId - 00002
lastLogIndex - 4
lastLogTerm - 2
Current
StateMachine
ACCEPT VOTE
REQUEST
candidate.term >= my.term && candidate.lastLogIdx < my.lastLogIdx => complete

Problems

Messages
37
Rely on messages
High overhead
Messages:
Duplicate
Out of order
Lost
Network Latency

Failures
38
Failures:
Availability
Leader failure
Follower(s) failure
Partition

Security
39
Confidentiality (who can read/ write?)

Test
40
Reliable
Test and debugging is hard

Server problems: e.g. Crushloop
Application-level problem: e.g. Corrupted Data
Other
41

Issues
42
Partitions
Network delays
Packet loss
Duplication
Ordering

43
Byzantine Generals Problem

44
Byzantine Failure

Malicious & Misbehaving Peers
45
Not resilient against malicious or misbehaving peers.
May not be capable of destroying the whole consensus protocol
Abused:
Confidentiality? (only intended parties can read)
Integrity? (messages are authentic)
Correctness is guaranteed ONLY if each peer adheres to the
protocol requirements.
Corrupting the application-level values
Injecting malicious values

46
Byzantine Fault Tolerant Algorithms
Byzantine Paxos algorithms
Byzantine Raft algorithms

Implementing Distributed Systems Protocols/ Consensus Algorithms:
Understandability
Consistency
Correctness, etc.
Don't
Resilient against issues?
Worth it to be resilient?
Take Away
47
Reliability Complexity
Designing Distributed Systems Protocols/ Consensus Algorithms:

Consensus Algorithms

Recommended

Recommended

More Related Content

Similar to Consensus Algorithms

Similar to Consensus Algorithms (8)

Recently uploaded

Recently uploaded (20)

Consensus Algorithms