Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Consistent Hash Exchange: Making RabbitMQ a better broker - Jack Vanlightly

124 views

Published on

Watch full lecture on YouTube: https://www.youtube.com/watch?v=g3Cq9g7sDc4&list=PLDUzG2yLXrU4Lz33ZzSdHyfqdHJ8Zum5A

In this session we'll look at an alternative to the competing consumer pattern by using the Consistent Hash Exchange. We'll see how this exchange enables different messaging patterns such as data locality, message processing order guarantees at scale and helping to avoid large queues which can be difficult to keep synchronized in a HA configuration.

--

The first RabbitMQ Summit connected RabbitMQ users and developers from around the world in London on November 12, 2018. Learn what's happening in and around RabbitMQ, and how top companies utilize RabbitMQ to power their services.

https://www.rabbitmqsummit.com

RabbitMQ Summit was organized by:
- Erlang Solutions, offering world-leading RabbitMQ Consultancy, Support, Health Checks & Tuning solutions https://www.erlang-solutions.com/
- CloudAMQP, offering fully managed RabbitMQ clusters https://www.cloudamqp.com

RabbitMQ Summit 2018 was sponsored by the following companies.

Platinum sponsors:
Pivotal
LShift

Gold sponors:
Trifork
AWS

Silver sponsor:
Cogin Queue Explorer

Published in: Technology
  • Be the first to comment

  • Be the first to like this

The Consistent Hash Exchange: Making RabbitMQ a better broker - Jack Vanlightly

  1. 1. CodeNode •London• ORGANISERS USES CASES FOR PARTITIONING QUEUES WITH THE CONSISTENT HASH EXCHANGE
  2. 2. • Consumers compete to consume from a single totally (FIFO) ordered sequence of messages • Unit of parallelism = Consumer THE COMPETING CONSUMER PATTERN Consumer 1 Consumer 2 Consumer 3 Consumer 4 Consumer 5 Bookings (fanout) Publisher 1 Publisher 2 Publisher 3 Bookings
  3. 3. • Messages are routed by the hash of the routing key. A given routing key will always be routed to the same queue. • Unit of parallelism = Queue • Total Partial order of messages. Ordering of messages according to causal relationships (booking created -> booking modified) ONE CONSUMER PER QUEUE PARTITION PATTERN Consumer 1 Consumer 2 Consumer 3 Consumer 4 Consumer 5 Bookings (hashing) Publisher 1 Publisher 2 Publisher 3 Bookings.3 Bookings.2 Bookings.1 Bookings.5 Bookings.4
  4. 4. •Weakened processing order guarantees •Large queues and HA configuration (High volume queues can grow fast. One downstream outage can cause a queue to balloon quickly) COMPETING CONSUMERS WHAT PROBLEMS CAN ARISE? Consumer 1 Consumer 2 Consumer 3 Consumer 4 Consumer 5 Bookings (fanout) Publisher 1 Publisher 2 Publisher 3 Bookings
  5. 5. •Guaranteed processing order of dependent messages while scaling out consumers •Data Locality Patterns •Reduce Queue Sizes ONE CONSUMER PER QUEUE PARTITION PATTERN WHAT CAN I DO WITH IT? Consumer 1 Consumer 2 Consumer 3 Consumer 4 Consumer 5 Bookings (hashing) Publisher 1 Publisher 2 Publisher 3 Bookings.3 Bookings.2 Bookings.1 Bookings.5 Bookings.4
  6. 6. PROCESSING ORDER AND PARALLELISM
  7. 7. PARALLEL PROCESSING AND THE SAFETY SCALE One Consumer Per Partitioned Queue with Sequential Processing Safety Competing Consumers with Prefetch of 1, short/constant processing time Messages Are Independent Competing Consumers with Prefetch of 1, longer/variable processing times Competing Consumers with Large Prefetch and longer/variable processing time Competing Consumers with Large Prefetch, short/constant processing time
  8. 8. a=1, a=2, a=3, a=4, a=5, a=6, a=7, a=8, a=9, a=10 ... DEMO 1: LOSS OF TOTAL ORDERING Consumer 1 Consumer 2 Consumer 3 Consumer 4 InputSeq (fanout) Publisher InputSeq DefaultOutputSeq Output Consumer a=3, a=1, a=2, a=4, a=5, a=7, a=6, a=10, a=8, a=9 ...
  9. 9. DEMO 2: MULTIPLE UPDATE SEQUENCE Consumer 1 Consumer 2 Consumer 3 Consumer 4 InputSeq (fanout) Publisher InputSeq DefaultOutputSeq Output Consumer a=1, b=1, c=1, d=1, a=2, b=2, c=2, d=2, a=3, b=3, c=3, d=3...
  10. 10. DEMO 3: MULTIPLE UPDATE SEQUENCE Consumer 4 Consumer 3 Consumer 2 Consumer 1 InputSeq (hashing) Publisher DefaultOutputSeq Output Consumer a=1, b=1, c=1, d=1, a=2, b=2, c=2, d=2, a=3, b=3, c=3, d=3... InputSeq.1 InputSeq.2 InputSeq.3 InputSeq.4
  11. 11. DATA LOCALITY PATTERNS
  12. 12. • Use Correlation Id to identify duplicates • Where to store this state? • Concurrent lookups return false DEDUPLICATION COMPETING CONSUMERS Consumer 1 Consumer 2 Consumer 3 Consumer 4 Consumer 5 Bookings Store/ Retrieve
  13. 13. Duplicates always routed to the same consumer Options: • Recent Correlation Ids stored in an in-memory map (such as a LinkedHashMap) • Embedded database • Bloom Filter • Snapshot data periodically DEDUPLICATION ONE CONSUMER PER QUEUE PARTITION Consumer 1 Consumer 2 Consumer 3 Consumer 4 Consumer 5 Bookings.3 Bookings.2 Bookings.1 Bookings.5 Bookings.4
  14. 14. A given consumer always processes the messages of a given entity (booking, client, order, product etc) Calculate metrics over a given time period. • Total revenue from orders in Norway over last 24 hour period • Number of requests per client in last ten minutes Fast resolution of business logic rules: • Client cannot perform more than X actions of type Y within a given period Z WINDOWING AND AGGREGATIONS ONE CONSUMER PER QUEUE PARTITION
  15. 15. DEMO 4: DEDUPLICATION Consumer 4 Consumer 3 Consumer 2 Consumer 1 InputSeq (hashing) Publisher DefaultOutputSeq Output Consumer a=1, b=1, b=1, c=1, d=1, a=2, b=2, c=2, c=2, d=2... InputSeq.1 InputSeq.2 InputSeq.3 InputSeq.4
  16. 16. QUEUE MIRRORING AND BLOCKING SYNCHRONISATION
  17. 17. MIRRORED QUEUES - BLOCKING SYNCHRONIZATION ONE MONOLITHIC QUEUE Node 1 Node 2 Orders (master) 1000000 Consumer Node 3 Orders (mirror) 1000000 Publisher
  18. 18. MIRRORED QUEUES - BLOCKING SYNCHRONIZATION ONE MONOLITHIC QUEUE, ONE CORE Node 1 Node 2 Orders (master) 1000000 Consumer Node 3 Orders (mirror) 0 Publisher Decision Time: To Synchronise or Not Synchronise?
  19. 19. MIRRORED QUEUES - BLOCKING SYNCHRONIZATION MULTIPLE QUEUES, MULTIPLE CORES Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000
  20. 20. MIRRORED QUEUES - BLOCKING SYNCHRONIZATION MULTIPLE QUEUES, MULTIPLE CORES Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 0 0 0
  21. 21. HARD THINGS ABOUT USING THE CONSISTENT HASH EXCHANGE
  22. 22. • How to do Consumer Queue Assignment? Rebalanser Project https://github.com/Rebalanser • Data Locality Patterns • Maintaining State and Consumer Failure • Changing Number of Queues NOTHING FOR FREE
  23. 23. App C Leader App B App A DEMO 5: REBALANSER Consumer 1 Consumer 2 Consumer 3 Consumer 4 Consumer 5 Bookings.3 Bookings.2 Bookings.1 Bookings.5 Bookings.4 Consumer 1 Consumer 2 Consumer 3 Consumer 4 Consumer 5 Bookings.8 Bookings.7 Bookings.6 Bookings.10 Bookings.9 • Leader election • Leader performs assignment • Changes to queues or consumers triggers rebalancing • Works with consensus services such as ZooKeeper or RDBMS with strong serializable transaction support Coordination

×