Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
CodeNode
•London•
ORGANISERS
USES CASES FOR
PARTITIONING QUEUES WITH
THE CONSISTENT HASH
EXCHANGE
• Consumers compete to consume from a single totally (FIFO)
ordered sequence of messages
• Unit of parallelism = Consumer
...
• Messages are routed by the hash of the routing key. A given
routing key will always be routed to the same queue.
• Unit ...
•Weakened processing order guarantees
•Large queues and HA configuration
(High volume queues can grow fast. One downstream...
•Guaranteed processing order of dependent messages while
scaling out consumers
•Data Locality Patterns
•Reduce Queue Sizes...
PROCESSING ORDER
AND
PARALLELISM
PARALLEL PROCESSING AND THE SAFETY
SCALE
One Consumer Per Partitioned Queue
with Sequential Processing
Safety
Competing Co...
a=1, a=2, a=3, a=4, a=5, a=6, a=7, a=8, a=9, a=10 ...
DEMO 1: LOSS OF TOTAL ORDERING
Consumer 1
Consumer 2
Consumer 3
Cons...
DEMO 2: MULTIPLE UPDATE SEQUENCE
Consumer 1
Consumer 2
Consumer 3
Consumer 4
InputSeq
(fanout)
Publisher InputSeq
DefaultO...
DEMO 3: MULTIPLE UPDATE SEQUENCE
Consumer 4
Consumer 3
Consumer 2
Consumer 1
InputSeq
(hashing)
Publisher
DefaultOutputSeq...
DATA LOCALITY
PATTERNS
• Use Correlation Id to identify duplicates
• Where to store this state?
• Concurrent lookups return false
DEDUPLICATION
C...
Duplicates always routed to the same consumer
Options:
• Recent Correlation Ids stored in an in-memory map (such as a
Link...
A given consumer always processes the messages of a given entity
(booking, client, order, product etc)
Calculate metrics o...
DEMO 4: DEDUPLICATION
Consumer 4
Consumer 3
Consumer 2
Consumer 1
InputSeq
(hashing)
Publisher
DefaultOutputSeq
Output
Con...
QUEUE MIRRORING
AND
BLOCKING
SYNCHRONISATION
MIRRORED QUEUES - BLOCKING
SYNCHRONIZATION
ONE MONOLITHIC QUEUE
Node 1 Node 2
Orders (master)
1000000
Consumer
Node 3
Orde...
MIRRORED QUEUES - BLOCKING
SYNCHRONIZATION
ONE MONOLITHIC QUEUE, ONE CORE
Node 1 Node 2
Orders (master)
1000000
Consumer
N...
MIRRORED QUEUES - BLOCKING
SYNCHRONIZATION
MULTIPLE QUEUES, MULTIPLE CORES
Node 1 Node 2 Node 3 Node 4 Node 5 Node 6
10000...
MIRRORED QUEUES - BLOCKING
SYNCHRONIZATION
MULTIPLE QUEUES, MULTIPLE CORES
Node 1 Node 2 Node 3 Node 4 Node 5 Node 6
10000...
HARD THINGS ABOUT
USING THE
CONSISTENT HASH
EXCHANGE
• How to do Consumer Queue Assignment?
Rebalanser Project
https://github.com/Rebalanser
• Data Locality Patterns
• Maintai...
App C
Leader
App B
App A
DEMO 5: REBALANSER
Consumer 1
Consumer 2
Consumer 3
Consumer 4
Consumer 5
Bookings.3
Bookings.2
B...
Upcoming SlideShare
Loading in …5
×

The Consistent Hash Exchange: Making RabbitMQ a better broker - Jack Vanlightly

278 views

Published on

Watch full lecture on YouTube: https://www.youtube.com/watch?v=g3Cq9g7sDc4&list=PLDUzG2yLXrU4Lz33ZzSdHyfqdHJ8Zum5A

In this session we'll look at an alternative to the competing consumer pattern by using the Consistent Hash Exchange. We'll see how this exchange enables different messaging patterns such as data locality, message processing order guarantees at scale and helping to avoid large queues which can be difficult to keep synchronized in a HA configuration.

--

The first RabbitMQ Summit connected RabbitMQ users and developers from around the world in London on November 12, 2018. Learn what's happening in and around RabbitMQ, and how top companies utilize RabbitMQ to power their services.

https://www.rabbitmqsummit.com

RabbitMQ Summit was organized by:
- Erlang Solutions, offering world-leading RabbitMQ Consultancy, Support, Health Checks & Tuning solutions https://www.erlang-solutions.com/
- CloudAMQP, offering fully managed RabbitMQ clusters https://www.cloudamqp.com

RabbitMQ Summit 2018 was sponsored by the following companies.

Platinum sponsors:
Pivotal
LShift

Gold sponors:
Trifork
AWS

Silver sponsor:
Cogin Queue Explorer

Published in: Technology
  • Be the first to comment

  • Be the first to like this

The Consistent Hash Exchange: Making RabbitMQ a better broker - Jack Vanlightly

  1. 1. CodeNode •London• ORGANISERS USES CASES FOR PARTITIONING QUEUES WITH THE CONSISTENT HASH EXCHANGE
  2. 2. • Consumers compete to consume from a single totally (FIFO) ordered sequence of messages • Unit of parallelism = Consumer THE COMPETING CONSUMER PATTERN Consumer 1 Consumer 2 Consumer 3 Consumer 4 Consumer 5 Bookings (fanout) Publisher 1 Publisher 2 Publisher 3 Bookings
  3. 3. • Messages are routed by the hash of the routing key. A given routing key will always be routed to the same queue. • Unit of parallelism = Queue • Total Partial order of messages. Ordering of messages according to causal relationships (booking created -> booking modified) ONE CONSUMER PER QUEUE PARTITION PATTERN Consumer 1 Consumer 2 Consumer 3 Consumer 4 Consumer 5 Bookings (hashing) Publisher 1 Publisher 2 Publisher 3 Bookings.3 Bookings.2 Bookings.1 Bookings.5 Bookings.4
  4. 4. •Weakened processing order guarantees •Large queues and HA configuration (High volume queues can grow fast. One downstream outage can cause a queue to balloon quickly) COMPETING CONSUMERS WHAT PROBLEMS CAN ARISE? Consumer 1 Consumer 2 Consumer 3 Consumer 4 Consumer 5 Bookings (fanout) Publisher 1 Publisher 2 Publisher 3 Bookings
  5. 5. •Guaranteed processing order of dependent messages while scaling out consumers •Data Locality Patterns •Reduce Queue Sizes ONE CONSUMER PER QUEUE PARTITION PATTERN WHAT CAN I DO WITH IT? Consumer 1 Consumer 2 Consumer 3 Consumer 4 Consumer 5 Bookings (hashing) Publisher 1 Publisher 2 Publisher 3 Bookings.3 Bookings.2 Bookings.1 Bookings.5 Bookings.4
  6. 6. PROCESSING ORDER AND PARALLELISM
  7. 7. PARALLEL PROCESSING AND THE SAFETY SCALE One Consumer Per Partitioned Queue with Sequential Processing Safety Competing Consumers with Prefetch of 1, short/constant processing time Messages Are Independent Competing Consumers with Prefetch of 1, longer/variable processing times Competing Consumers with Large Prefetch and longer/variable processing time Competing Consumers with Large Prefetch, short/constant processing time
  8. 8. a=1, a=2, a=3, a=4, a=5, a=6, a=7, a=8, a=9, a=10 ... DEMO 1: LOSS OF TOTAL ORDERING Consumer 1 Consumer 2 Consumer 3 Consumer 4 InputSeq (fanout) Publisher InputSeq DefaultOutputSeq Output Consumer a=3, a=1, a=2, a=4, a=5, a=7, a=6, a=10, a=8, a=9 ...
  9. 9. DEMO 2: MULTIPLE UPDATE SEQUENCE Consumer 1 Consumer 2 Consumer 3 Consumer 4 InputSeq (fanout) Publisher InputSeq DefaultOutputSeq Output Consumer a=1, b=1, c=1, d=1, a=2, b=2, c=2, d=2, a=3, b=3, c=3, d=3...
  10. 10. DEMO 3: MULTIPLE UPDATE SEQUENCE Consumer 4 Consumer 3 Consumer 2 Consumer 1 InputSeq (hashing) Publisher DefaultOutputSeq Output Consumer a=1, b=1, c=1, d=1, a=2, b=2, c=2, d=2, a=3, b=3, c=3, d=3... InputSeq.1 InputSeq.2 InputSeq.3 InputSeq.4
  11. 11. DATA LOCALITY PATTERNS
  12. 12. • Use Correlation Id to identify duplicates • Where to store this state? • Concurrent lookups return false DEDUPLICATION COMPETING CONSUMERS Consumer 1 Consumer 2 Consumer 3 Consumer 4 Consumer 5 Bookings Store/ Retrieve
  13. 13. Duplicates always routed to the same consumer Options: • Recent Correlation Ids stored in an in-memory map (such as a LinkedHashMap) • Embedded database • Bloom Filter • Snapshot data periodically DEDUPLICATION ONE CONSUMER PER QUEUE PARTITION Consumer 1 Consumer 2 Consumer 3 Consumer 4 Consumer 5 Bookings.3 Bookings.2 Bookings.1 Bookings.5 Bookings.4
  14. 14. A given consumer always processes the messages of a given entity (booking, client, order, product etc) Calculate metrics over a given time period. • Total revenue from orders in Norway over last 24 hour period • Number of requests per client in last ten minutes Fast resolution of business logic rules: • Client cannot perform more than X actions of type Y within a given period Z WINDOWING AND AGGREGATIONS ONE CONSUMER PER QUEUE PARTITION
  15. 15. DEMO 4: DEDUPLICATION Consumer 4 Consumer 3 Consumer 2 Consumer 1 InputSeq (hashing) Publisher DefaultOutputSeq Output Consumer a=1, b=1, b=1, c=1, d=1, a=2, b=2, c=2, c=2, d=2... InputSeq.1 InputSeq.2 InputSeq.3 InputSeq.4
  16. 16. QUEUE MIRRORING AND BLOCKING SYNCHRONISATION
  17. 17. MIRRORED QUEUES - BLOCKING SYNCHRONIZATION ONE MONOLITHIC QUEUE Node 1 Node 2 Orders (master) 1000000 Consumer Node 3 Orders (mirror) 1000000 Publisher
  18. 18. MIRRORED QUEUES - BLOCKING SYNCHRONIZATION ONE MONOLITHIC QUEUE, ONE CORE Node 1 Node 2 Orders (master) 1000000 Consumer Node 3 Orders (mirror) 0 Publisher Decision Time: To Synchronise or Not Synchronise?
  19. 19. MIRRORED QUEUES - BLOCKING SYNCHRONIZATION MULTIPLE QUEUES, MULTIPLE CORES Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000
  20. 20. MIRRORED QUEUES - BLOCKING SYNCHRONIZATION MULTIPLE QUEUES, MULTIPLE CORES Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 0 0 0
  21. 21. HARD THINGS ABOUT USING THE CONSISTENT HASH EXCHANGE
  22. 22. • How to do Consumer Queue Assignment? Rebalanser Project https://github.com/Rebalanser • Data Locality Patterns • Maintaining State and Consumer Failure • Changing Number of Queues NOTHING FOR FREE
  23. 23. App C Leader App B App A DEMO 5: REBALANSER Consumer 1 Consumer 2 Consumer 3 Consumer 4 Consumer 5 Bookings.3 Bookings.2 Bookings.1 Bookings.5 Bookings.4 Consumer 1 Consumer 2 Consumer 3 Consumer 4 Consumer 5 Bookings.8 Bookings.7 Bookings.6 Bookings.10 Bookings.9 • Leader election • Leader performs assignment • Changes to queues or consumers triggers rebalancing • Works with consensus services such as ZooKeeper or RDBMS with strong serializable transaction support Coordination

×