SpringOne Platform 2017 Michael Klishin, Pivotal; Karl Nilsson, Pivotal Team RabbitMQ has been working on adopting Raft, a distributed consensus protocol, in several components of the system. In this talk we will explain the pros and cons of the migration, how it can affect our users, what kind of trade-offs our team had to face, and whether Raft is actually as simple as that person on Hacker News claim it to be.

1. Raft in RabbitMQ
Karl Nilsson @kjnilsson
Michael Klishin @michaelklishin
1

2. Disclaimer

3. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Safe Harbor Statement
The following is intended to outline the general direction of Pivotal's offerings. It is
intended for information purposes only and may not be incorporated into any
contract. Any information regarding pre-release of Pivotal offerings, future updates
or other planned modifications is subject to ongoing evaluation by Pivotal and is
subject to change. This information is provided without warranty or any kind, express
or implied, and is not a commitment to deliver any material, code, or functionality,
and should not be relied upon in making purchasing decisions regarding Pivotal's
offerings. These purchasing decisions should only be based on features currently
available. The development, release, and timing of any features or functionality
described for Pivotal's offerings in this presentation remain at the sole discretion of
Pivotal. Pivotal has no obligation to update forward looking information in this
presentation.
3

4. What is RabbitMQ?

5. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
What is RabbitMQ
• Messaging broker
5

6. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
What is RabbitMQ
6
Publisher Consumer
Consumer

7. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
What is RabbitMQ
• Messaging broker
• Multi-protocol (AMQP 0-9-1, AMQP 1.0, MQTT, STOMP, …)
• Started in 2006
• Broad ecosystem including Spring support
• Learn more at rabbitmq.com
7

8. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
RabbitMQ nodes can form clusters
• Balance load (connections, traffic, I/O, …) between nodes
• Replicate queue contents
• Tolerate node failures
8

9. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Sections
1. High Availability in RabbitMQ
2. Raft overview
3. RabbitMQ and Raft
4. Raft challenges
5. Implementation
9

10. Where We are Today

11. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
RabbitMQ High Availability
• Replication of data and operations
• Message replication is done at the
queue level
• Called “Queue Mirroring”
• In a cluster of RabbitMQ nodes a
queue can have a mirror on one or
more nodes
• Provides fail-over and redundancy
11
Rabbit 1
Q1
Rabbit 2
Q1
Rabbit 3
Q1
Master Queue
Mirror Queues

12. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
RabbitMQ HA story isn’t perfect!
12

13. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
RabbitMQ Queue Mirroring
• Internally uses a component called “Guaranteed Multicast” to replicate queue
operations and message data
• Provides replication and total ordering of operations
• Ordering matters:
[ENQ + ENQ + DEQ] != [DEQ + ENQ + ENQ]
13

14. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Chain replication
Chain Replication ensures strong consistency and good availability guarantees
in “fail-stop” scenarios.
Chain Replication for Supporting High Throughput and Availability (Robbert
van Renesse, Fred B. Schneider)
http://www.cs.cornell.edu/home/rvr/papers/OSDI04.pdf
14

15. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Chain replication
15
HEAD TAIL
READWRITE

16. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
RabbitMQ Mirrored Queue ring
16
MASTER MIRROR MIRROR MIRROR MIRROR
DELIVERPUBLISH

17. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
RabbitMQ Queue Mirror failure detection
17
MASTER
MIRROR
MIRROR
MIRROR
MIRRORPUBLISH
Needs timely fault detection

18. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
RabbitMQ Mirrored Queue reforms ring
18
MASTER
MIRROR
MIRROR
MIRROR
MIRRORPUBLISH DELIVER

19. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
RabbitMQ Queue reforms ring
19
MASTER
MIRROR
MIRROR
MIRROR
MIRRORPUBLISH DELIVER
I was only gone a minute. Can
I re-join?
It’s complicated…

20. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
RabbitMQ Queue mirror sync
20
MASTER
MIRROR
MIRROR
MIRROR
MIRRORPUBLISH DELIVER

21. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
RabbitMQ documentation wisdom
21
This topology [Ring] through relatively
cheap in queues and connections, is
rather fragile

22. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 22
[DO AS WE SAY NOT AS WE DO]

23. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
RabbitMQ Queue Mirroring
• Works well most of the time
• Requires good failure detection
• Membership changes are expensive
(require queue sync)
• Master election algorithm is
informally specified (at best)
23
We can do better

24. Road to Raft

25. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
What do we need?
• Strong consistency guarantees
• Total order of operations
• Predictable behaviour in response to failure events (well-defined recovery
procedure)
• Safe queue master “fail-over”
• Parallel replication
• Avoid reinventing another distsys wheel
25

26. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Options
• Chain Replication
• Paxos
• Viewstamped Replication
• Raft
26

27. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Raft
• A group of algorithms for reaching consensus in a distributed system
• Similar problem space to RabbitMQ queue mirroring
• Oriented towards implementers
• Proven
• Multiple implementations
• Industry use
• etcd
• Consul
• CockroachDB
• TLA+ specification
• Requires no external dependencies
27

28. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Raft provides
• A state machine log abstraction
• Fits many domains
• Leader-follower model
• State machine log replication
• Consistency-oriented, availability
characteristics covered later
• Total order of operations
• Well-defined algorithms important for implementers
• Leader election
• Safe cluster membership changes
• Durable storage expectations
• Recovery
• Replay log to restore state
• Snapshotting
28
Raft
FOLLOWER
LEADER
FOLLOWER

29. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Raft protocol: replicate entry
29
FOLLOWER
LEADER
FOLLOWER
ENQ ‘A’
LOG
1 ENQ ‘A’
CommitIndex: 0
State = []
AppendEntriesRpc { Entries = [1, ENQ ‘A’]
AppendEntriesRpc { Entries = [1, ENQ ‘A’]

30. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Raft protocol: reply + commit
30
FOLLOWER
LEADER
FOLLOWER
OK
LOG
1 ENQ ‘A’
CommitIndex: 0
State = []
CommitIndex: 1
State = [ENQ ‘A’]
AppendEntriesReply { Success = true }

31. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
A raft
Learn more at raft.github.io
31

32. How Does Raft Compare?

33. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Taking action
What to do when detecting a (potential) failure?
A. Nothing
• most reliable / least useful
B. Try to “fix stuff”
• evict down nodes, reform topology
• communicate changes to other nodes
C. The minimum required
• regain / retain availability and consistency
33

34. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Raft vs Queue Mirroring failure handling
Raft
34
Queue mirroring
Follower
Leader
Follower
Follower
Follower
Mirror
Master
Mirror
Mirror
Mirror

35. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Raft vs Queue Mirroring leader failure
Raft
35
Queue mirroring
Follower
Leader
Follower
Follower
Follower
Mirror
Master
Mirror
Mirror
Mirror
Should I
take
Sure - your log
is up to date.
Go ahead.
Why
I’m the
oldest -
I’ll be
Master!
Ok…

36. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
In response to (potential) failure:
Raft either:
• Does nothing
• Does the minimum required
• ensures consistency
• regains availability
36
RabbitMQ queue mirroring:
• Must always do something

37. Adopting Raft

38. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Raft in RabbitMQ
• Can be adopted in multiple areas incrementally
• Area of focus: queue mirroring
• Coordination, leader election
• Cluster federation
• Shovel
• Delayed message exchange
• Message store data replication
• Messages
• Distributed data and state storage
• Internal metadata store (vhosts, users, permissions, queues, …)
38

39. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Raft in RabbitMQ
• More predictable recovery from node failures
• Improved serializability and linearizability characteristics
• Better scalability of mirrored queues [w.r.t. the number of nodes]
• Write-ahead Log
• Multi-DC support
39

40. Down the Rabbit Hole

41. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Message store replication
• RabbitMQ allows for complex routing of messages
• One message can be routed to many queues (fan-out)
• May consumers may receive the same message
• Consumers may be connected to any RabbitMQ node
• Queue master may exist on any RabbitMQ node
• The queue master delivers messages to consumers
• Sub-optimal use of network
• The same message may be sent over the same socket multiple times
41

42. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Message store replication
• Replicate message bodies independently of queue operations
• All bodies are written and read locally
• Queues only replicate operations + pointers to message data
• Many consumers can read the same message from a local store
42

43. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Raft challenges
• The cost of consensus
• Raft requires “stable storage” (fsync)
• Latency
• Reduced availability
• Requires a quorum
• Cluster formation
• Seeding
• RabbitMQ internal concern
• Single leader
• Scalability
43

44. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Operational changes
• Uneven cluster sizes required / recommended
• 3 nodes can tolerate 1 failure
• 5 nodes can tolerate 2 failures, and so on…
44

45. Announcement time

46. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Ra: a Raft library
• By Team RabbitMQ
• Soon to be open sourced
• ASL2 / MPL1.1 licensed
• Generically usable, not tied to
RabbitMQ
• Tailored for RabbitMQ needs
• Very much a WIP (breaking API
changes are likely)
• https://github.com/rabbitmq/ra
46

47. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Ra: implementation
• Raft cluster per queue
• Many queues = many Raft clusters
• Each node writing to it’s own log file
• Thousands of concurrent fsync operations
• A no-go - we tried it
• Raft is chatty
• High background network usage when idle
47

48. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Ra: implementation
• Shared Write Ahead Log (WAL)
• fsync in batches
• flushed to raft node specific storage periodically
• Storage engine
• Similar to LSM tree and "append-oriented" stores (LevelDB, RocksDB)
• Compaction is radically simpler in our case
• Per RabbitMQ node “heartbeat” process
• Reduce network background usage
48

49. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Ra: implementation testing
• Unit & integration testing
• Property-based testing 🤖
• Correctness is essential
• Deployment testing (BOSH)
• TLA+ spec for log implementation 🤓
49

50. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Summary
• RabbitMQ queue mirroring has fundamental problems
• Raft covers a very similar problem space
• New design promises lots of improvements 😊
• Implementing Raft is non-trivial
• github.com/rabbitmq/ra
• We are still learning
50

51. Announcement time

52. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
RabbitMQ 3.7
• More automation-friendly
• Easier to use config format
• Pluggable peer discovery
• Distributed management plugin
• Per-vhost limits
• Operator policies
• Message store multi-tenancy
• Distributed via Bintray, Package Cloud and GitHub (not rabbitmq.com)
• Learn more at rabbitmq.com/changelog.html.
52

53. Thank you
@kjnilsson
@michaelklishin
rabbitmq-users (a Google group)
53
#springone@s1p