ZooKeeper allows for dynamic reconfiguration of servers in its ensemble. Manual reconfiguration is problematic as it requires changing the configuration, restarting servers, and can result in data loss. The presented solution allows ZooKeeper to reconfigure itself automatically through a speculative reconfiguration approach. It commits the reconfiguration once quorums of both the old and new ensembles acknowledge it, and gossips the new configuration to ensure all servers sync before activation. This allows reconfigurations to complete without failures in a transparent manner to clients.

1. Dynamic Reconfiguration of ZooKeeper
Alex Shraer
(presented by Benjamin Reed)

2. Why ZooKeeper?
•
Lots of servers
•
Lots of processes
•
High volumes of data
•
Highly complex software systems
•
… mere mortal developers

3. What ZooKeeper gives you
● Simple programming model
● Coordination of distributed processes
● Fast notification of changes
● Elasticity
● Easy setup
● High availability

4. ZooKeeper Configuration
• Membership
• Role of each server
– E.g., follower or observer
• Quorum System spec
– Zookeeper: majority or hierarchical
• Network addresses & ports
• Timeouts, directory paths, etc.

5. Zookeeper - distributed and replicated
ZooKeeper Service
Leader
Server Server Server Server Server
Client Client Client Client Client Client Client Client
• All servers store a copy of the data (in memory)
• A leader is elected at startup
• Reads served by followers, all updates go through leader
• Update acked when a quorum of servers have persisted the
change (on disk)
• Zookeeper uses ZAB - its own atomic broadcast protocol

6. Dynamic Membership Changes
• Necessary in every long-lived system!
• Examples:
– Cloud computing: adapt to changing load, don’t pre-allocate!
– Failures: replacing failed nodes with healthy ones
– Upgrades: replacing out-of-date nodes with up-to-date ones
– Free up storage space: decreasing the number of replicas
– Moving nodes: within the network or the data center
– Increase resilience by changing the set of servers
Example: asynch. replication works as long as > #servers/2 operate:

7. Dynamic Membership Changes
• Necessary in every long-lived system!
• Examples:
– Cloud computing: adapt to changing load, don’t pre-allocate!
– Failures: replacing failed nodes with healthy ones
– Upgrades: replacing out-of-date nodes with up-to-date ones
– Free up storage space: decreasing the number of replicas
– Moving nodes: within the network or the data center
– Increase resilience by changing the set of servers
Example: asynch. replication works as long as > #servers/2 operate:

8. Dynamic Membership Changes
• Necessary in every long-lived system!
• Examples:
– Cloud computing: adapt to changing load, don’t pre-allocate!
– Failures: replacing failed nodes with healthy ones
– Upgrades: replacing out-of-date nodes with up-to-date ones
– Free up storage space: decreasing the number of replicas
– Moving nodes: within the network or the data center
– Increase resilience by changing the set of servers
Example: asynch. replication works as long as > #servers/2 operate:

9. Dynamic Membership Changes
• Necessary in every long-lived system!
• Examples:
– Cloud computing: adapt to changing load, don’t pre-allocate!
– Failures: replacing failed nodes with healthy ones
– Upgrades: replacing out-of-date nodes with up-to-date ones
– Free up storage space: decreasing the number of replicas
– Moving nodes: within the network or the data center
– Increase resilience by changing the set of servers
Example: asynch. replication works as long as > #servers/2 operate:

10. Dynamic Membership Changes
• Necessary in every long-lived system!
• Examples:
– Cloud computing: adapt to changing load, don’t pre-allocate!
– Failures: replacing failed nodes with healthy ones
– Upgrades: replacing out-of-date nodes with up-to-date ones
– Free up storage space: decreasing the number of replicas
– Moving nodes: within the network or the data center
– Increase resilience by changing the set of servers
Example: asynch. replication works as long as > #servers/2 operate:

11. Dynamic Membership Changes
• Necessary in every long-lived system!
• Examples:
– Cloud computing: adapt to changing load, don’t pre-allocate!
– Failures: replacing failed nodes with healthy ones
– Upgrades: replacing out-of-date nodes with up-to-date ones
– Free up storage space: decreasing the number of replicas
– Moving nodes: within the network or the data center
– Increase resilience by changing the set of servers
Example: asynch. replication works as long as > #servers/2 operate:

12. Hazards of Manual Reconfiguration
E
A
C
{A, B, C}
B {A, B, C} D
{A, B, C}
• Goal: add servers E and D

13. Hazards of Manual Reconfiguration
E
A
C
{A, B, C, D, E} {A, B, C, D, E}
B {A, B, C, D, E} D
{A, B, C, D, E}
{A, B, C, D, E}
• Goal: add servers E and D
• Change Configuration

14. Hazards of Manual Reconfiguration
E
A
C
{A, B, C, D, E} {A, B, C, D, E}
B {A, B, C, D, E} D
{A, B, C, D, E}
{A, B, C, D, E}
• Goal: add servers E and D
• Change Configuration
• Restart Servers

15. Hazards of Manual Reconfiguration
E
A
C
{A, B, C, D, E} {A, B, C, D, E}
B {A, B, C, D, E} D
{A, B, C, D, E}
{A, B, C, D, E}
• Goal: add servers E and D
• Change Configuration
• Restart Servers

16. Hazards of Manual Reconfiguration
E
A
C
{A, B, C, D, E} {A, B, C, D, E}
B {A, B, C, D, E} D
{A, B, C, D, E}
{A, B, C, D, E}
• Goal: add servers E and D
• Change Configuration
• Restart Servers

17. Hazards of Manual Reconfiguration
E
A
C
{A, B, C, D, E} {A, B, C, D, E}
B {A, B, C, D, E} D
{A, B, C, D, E}
{A, B, C, D, E}
• Goal: add servers E and D
• Change Configuration
• Restart Servers
• Lost and !

18. 18
Just use a coordination service!
• Zookeeper is the coordination service
– Don’t want to deploy another system to coordinate it!
• Who will reconfigure that system ?
– GFS has 3 levels of coordination services
• More system components -> more management overhead
• Use Zookeeper to reconfigure itself!
– Other systems store configuration information in Zookeeper
– Can we do the same??
– Only if there are no failures

19. Recovery in Zookeeper
C E
B
setData(/x, 5)
D
A

20. Recovery in Zookeeper
C E
B
setData(/x, 5)
D
A

21. Recovery in Zookeeper
C E
B
setData(/x, 5)
D
A

22. Recovery in Zookeeper
C E
B
setData(/x, 5)
D
A

23. This doesn’t work for reconfigurations!
E
C
B
{A, B, C, D, E} {A, B, C, D, E}
setData(/zookeeper/config, {A, B, F})
{A, B, C, D, E} D
remove C, D, E add F
F
{A, B, C, D, E}
A
{A, B, C, D, E}

24. This doesn’t work for reconfigurations!
E
C
B
{A, B, C, D, E} {A, B, C, D, E}
setData(/zookeeper/config, {A, B, F})
{A, B, C, D, E} D
remove C, D, E add F
F
{A, B, C, D, E}
A
{A, B, F}
{A, B, F}

25. This doesn’t work for reconfigurations!
E
C
B
{A, B, C, D, E} {A, B, C, D, E}
setData(/zookeeper/config, {A, B, F})
{A, B, C, D, E} D
remove C, D, E add F
F
{A, B, C, D, E}
A
{A, B, F}
{A, B, F}
• Must persist the decision to reconfigure in the old
config before activating the new config!
• Once such decision is reached, must not allow further
ops to be committed in old config

26. Our Solution
• Correct
• Fully automatic
• No external services or additional components
• Minimal changes to Zookeeper
• Usually unnoticeable to clients
– Pause operations only in rare circumstances
– Clients work with a single configuration
• Rebalances clients across servers in new configuration
• Reconfigures immediately
• Speculative Reconfiguration
– Reconfiguration (and commands that follow it) speculatively sent out by the
primary, similarly to all other updates

27. Principles
● Commit reconfig in a quorum of the old ensemble
– Submit reconfig op just like any other update
● Make sure new ensemble has latest state before
becoming active
– Get quorum of synced followers from new config
– Get acks from both old and new ensembles before committing
updates proposed between reconfig op and activation
– Activate new configuration when reconfig commits
● Once new ensemble active old ensemble cannot commit
or propose new updates
● Gossip activation through leader election and syncing
● Verify configuration id of leader and follower

28. Failure free flow

29. Reconfiguration scenario 1
E
A
C
{A, B, C} {A, B, C}
B {A, B, C} D
{A, B, C}
{A, B, C}
• Goal: add servers E and D

30. Reconfiguration scenario 1
E
A
C
{A, B, C}
B {A, B, C} D
{A, B, C}
• Goal: add servers E and D
• doesn't commit until quorums of
both ensembles ack

31. Reconfiguration scenario 1
E
A
C
{A, B, C} {A, B, C}
B {A, B, C} D
{A, B, C}
{A, B, C}
• Goal: add servers E and D
• doesn't commit until quorums of
both ensembles ack

32. Reconfiguration scenario 1
E
A
C
{A, B, C} {A, B, C}
B {A, B, C} D
{A, B, C}
{A, B, C}
• Goal: add servers E and D
• doesn't commit until quorums of
both ensembles ack

33. Reconfiguration scenario 1
E
A
C
{A, B, C, D, E} {A, B, C, D, E}
B {A, B, C} D
{A, B, C, D, E}
{A, B, C, D, E}
• Goal: add servers E and D
• doesn't commit until quorums of
both ensembles ack

34. Reconfiguration scenario 1
E
A
C
{A, B, C, D, E} {A, B, C, D, E}
B {A, B, C} D
{A, B, C, D, E}
{A, B, C, D, E}
• Goal: add servers E and D
• doesn't commit until quorums of
both ensembles ack
• E and D gossip new configuration
to C

35. Reconfiguration scenario 1
E
A
C
{A, B, C, D, E} {A, B, C, D, E}
B {A, B, C, D, E} D
{A, B, C, D, E}
{A, B, C, D, E}
• Goal: add servers E and D
• doesn't commit until quorums of
both ensembles ack
• E and D gossip new configuration
to C

36. Example - reconfig using CLI
reconfig -add 1=host1.com:1234:1235:observer;1239
-add 2=host2.com:1236:1237:follower;1231 -remove 5
●
Change follower 1 to an observer and change its ports
●
Add follower 2 to the ensemble
●
Remove follower 5 from the ensemble
reconfig -file myNewConfig.txt -v 234547
●
Change the current config to the one in myNewConfig.txt
●
But only if current config version is 234547
getConfig -w -c
●
set a watch on /zookeeper/config
●
-c means we only want the new connection string for clients

37. When it will not work
● Quorum of new ensemble must be in sync
● Another reconfig in progress
● Version condition check fails

38. How do you know you are done
● Write something somewhere

39. The “client side” of reconfiguration
• When system changes, clients need to stay connected
– The usual solution: directory service (e.g., DNS)
• Re-balancing load during reconfiguration is also important!
• Goal: uniform #clients per server with minimal client migration
– Migration should be proportional to change in membership
X 10 X 10 X 10

40. The “client side” of reconfiguration
• When system changes, clients need to stay connected
– The usual solution: directory service (e.g., DNS)
• Re-balancing load during reconfiguration is also important!
• Goal: uniform #clients per server with minimal client migration
– Migration should be proportional to change in membership
X 10 X 10 X 10

41. Our approach - Probabilistic Load Balancing
• Example 1 :
X 10 X 10 X 10

42. Our approach - Probabilistic Load Balancing
• Example 1 :
X 10 X 10 X 10

43. Our approach - Probabilistic Load Balancing
• Example 1 :
X6 X6 X6 X6 X6
– Each client moves to a random new server with probability 0.4
– 1 – 3/5 = 0.4
– Exp. 40% clients will move off of each server

44. Our approach - Probabilistic Load Balancing
• Example 1 :
X6 X6 X6 X6 X6
– Each client moves to a random new server with probability 0.4
– 1 – 3/5 = 0.4
– Exp. 40% clients will move off of each server
●
Example 2 :
X6 X6 X6 X6 X6

45. Our approach - Probabilistic Load Balancing
• Example 1 :
X6 X6 X6 X6 X6
– Each client moves to a random new server with probability 0.4
– 1 – 3/5 = 0.4
– Exp. 40% clients will move off of each server
●
Example 2 :
X6 X6 X6 X6 X6

46. Our approach - Probabilistic Load Balancing
• Example 1 :
X6 X6 X6 X6 X6
– Each client moves to a random new server with probability 0.4
– 1 – 3/5 = 0.4
– Exp. 40% clients will move off of each server
●
Example 2 :
4/18 4/18 10/18
X6 X6 X6 X6 X6
– Connected clients don’t move
– Disconnected clients move to old servers with prob 4/18 and new one with prob
10/18
– Exp. 8 clients will move from A, B, C to D, E and 10 to F

47. Our approach - Probabilistic Load Balancing
• Example 1 :
X6 X6 X6 X6 X6
– Each client moves to a random new server with probability 0.4
– 1 – 3/5 = 0.4
– Exp. 40% clients will move off of each server
●
Example 2 :
4/18 4/18 10/18
X 10 X 10 X 10
– Connected clients don’t move
– Disconnected clients move to old servers with prob 4/18 and new one with prob
10/18
– Exp. 8 clients will move from A, B, C to D, E and 10 to F

48. Current Load Balancing

49. ProbabilisticCurrent Load Balancing
When moving from config. S to S’:
E (load (i, S ' )) = load (i, S ) + ∑ load ( j, S ) ⋅ Pr( j → i ) − load (i, S ) ∑ Pr(i → j )
j∈S ∧ j ≠i j∈S ' ∧ j ≠i
expected #clients #clients
connected to i in S’ connected #clients
(10 in last example) to i in S #clients
moving to i from moving from i to
other servers in S other servers in S’
Solving for Pr we get case-specific probabilities.
Input: each client answers locally
Question 1: Are there more servers now or less ?
Question 2: Is my server being removed?
Output: 1) disconnect or stay connected to my server
if disconnect 2) Pr(connect to one of the old servers)
and Pr(connect to newly added server)

50. Implementation
• Implemented in Zookeeper (Java & C), integration ongoing
– 3 new Zookeeper API calls: reconfig, getConfig, updateServerList
– feature requested since 2008, expected in 3.5.0 release (july 2012)
• Dynamic changes to:
– Membership
– Quorum System
– Server roles
– Addresses & ports
• Reconfiguration modes:
– Incremental (add servers E and D, remove server B)
– Non-incremental (new config = {A, C, D, E})
– Blind or conditioned (reconfig only if current config is #5)
• Subscriptions to config changes
– Client can invoke client-side re-balancing upon change

51. 52
Summary
• Design and implementation of reconfiguration for Apache Zookeeper
– being contributed into Zookeeper codebase
• Much simpler than state of the art, using properties already provided by Zookeeper
• Many nice features:
– Doesn’t limit concurrency
– Reconfigures immediately
– Preserves primary order
– Doesn’t stop client ops
– Zookeeper used by online systems, any delay must be avoided
– Clients work with a single configuration at a time
– No external services
– Includes client-side rebalancing