A somewhat in-depth look
Overview
• Idea was born around the same time AMQP was
first drafted by JP Morgan Chase
• First released in 2006
• Acquired by Sprin...
• Used by quite a few major projects including
‣ Huffington Post (Huff Live)
‣ Indeed (Job Aggregation service)
‣ VMWare (v...
The Basics
Virtual Host
QueuesExchange
Binding
AMQ Model Recap
➡ Queue:
‣ critical_error_queue
➡ Exchange:
‣ errors_exchange
➡ Filter...
QueuesExchange Binding
AMQ Model Recap
Publisher Consumer
‣ creates a relationship between
queues and exchanges
‣ contains...
Fanout Topic Headers System
Exchange
Types
(routing algo)
Direct
Exchange Types
Direct Exchange
“events”
type: direct
Q: user_clicks
X: events
K: user.click
MessagePublisher Consumer
K: user.click
Direct Exchange
“events”
type: direct
Q: user_clicks
X: events
K: user.click
Message
Publisher Consumer
K: user.view
Messa...
Direct Exchange
“events”
type: direct
Q: user_clicks
X: events
K: user.click
Message
Publisher Consumer
K: user.view
Messa...
Direct Exchange
“events”
type: direct
Q: user_clicks
X: events
K: user.click
Message
Publisher Consumer
K: user.view
Messa...
Direct Exchange
“events”
type: direct
Q: user_clicks
X: events
K: user.click
Message
Publisher Consumer
K: user.view
Messa...
Direct Exchange
“events”
type: direct
Q: user_clicks
X: events
K: user.click
Message
Publisher Consumer
K: user.view
Messa...
Direct Exchange
Walkthrough and demo
Topic Exchange
“events”
type: topic
Q: all_errors
X: events
K: *.error
Message
Publisher Consumer
K: sys.info
Message
K: s...
Topic Exchange
“events”
type: topic
Q: all_errors
X: events
K: *.error
Message
Publisher Consumer
K: sys.info
Message
K: s...
Topic Exchange
“events”
type: topic
Q: all_errors
X: events
K: *.error
Message
Publisher Consumer
K: sys.info
Message
K: s...
Topic Exchange
“events”
type: direct
Q: all_errors
X: events
K: *.error
Message
Publisher Consumer
K: sys.info
Message
K: ...
Topic Exchange
Walkthrough and demo
Topic Exchange
“events”
type: topic
Q: all_errors
X: events
K: *.error
Message
Publisher Consumer
K: sys.info
Message
K: s...
Persistence & Durability
exchangeMessagePublisher ConsumerQueues
Persistence & Durability
exchangeMessagePublisher ConsumerQueues
‣persistent
Persistence & Durability
exchangeMessagePublisher ConsumerQueues
‣persistent ‣durable
Persistence & Durability
exchangeMessagePublisher ConsumerQueues
‣persistent ‣durable ‣durable
Persistence & Durability
exchangeMessagePublisher ConsumerQueues
‣persistent ‣durable ‣durable
Component will be recreated...
Persistence & Durability
exchangeMessagePublisher ConsumerQueues
‣persistent ‣durable ‣durable
Component will be recreated...
Administration
Administration
• Management tools via rabbitmq-plugins
• Command:
sudo rabbitmq-plugins enable rabbitmq_management
• Gives...
Administration
• Important tools:
‣ rabbitmqctl commands
- users, perms, queues, bindings
- status, reset
‣ MochiWeb featu...
Distributed RabbitMQ
Clustering & Replication
Distributed RabbitMQ
Clustering Replication HA
Distributed RabbitMQ
Clustering Replication
➡Acts as a single logical unit
➡Consumer sees cluster as
single node
➡Nodes sh...
Distributed RabbitMQ
Clustering Replication
➡Acts as a single logical unit
➡Consumer sees cluster as
single node
➡Nodes sh...
Distributed RabbitMQ
Clustering Replication
➡Acts as a single logical unit
➡Consumer sees cluster as
single node
➡Nodes sh...
Clustering
Clustering
metadata
Clustering
Clustering
metadata
metadata
rabbit01 rabbit03
rabbit02
Cluster
Typical clustering
Metadata is replica...
metadata
Clustering
Clustering
metadata
metadata
rabbit01 rabbit03
rabbit02
Cluster
P
and this is the first time it’s publi...
metadata
Clustering
Clustering
shares_queue
metadata
metadata
rabbit01 rabbit03
rabbit02
Cluster
P
Through a LB, the conne...
metadata
Clustering
Clustering
shares_queue
metadata
metadata
rabbit01 rabbit03
rabbit02
Cluster
Queues are created on the...
metadata
Clustering
Clustering
shares_queue
metadata
metadata
rabbit01 rabbit03
rabbit02
Cluster
P
clicks_queue
Queues are...
metadata
Clustering
Clustering
shares_queue
metadata
metadata
clicks_queue
rabbit01 rabbit03
rabbit02
Cluster
Clustering Notes
• There are three types of nodes:
‣ Stat, Disk and Ram
• All nodes must share the same cookie
• Works lik...
Important
• RabbitMQ was not designed to handle
network partition.
• Nodes in clusters should have LAN-like
reliability
Clustering
Walkthrough and demo
Distributed RabbitMQ
Clustering Replication
➡Acts as a single logical unit
➡Consumer sees cluster as
single node
➡Nodes sh...
HA with Mirrored Queues
metadata
shares_queue
metadata
metadata
shares_queue
rabbit01 rabbit03
rabbit02
Cluster
HA
Needs t...
HA with Mirrored Queues
metadata
shares_queue
metadata
metadata
shares_queue
rabbit01 rabbit03
rabbit02
Cluster
HA
Needs t...
• Set up as a queue policy in RabbitMQ
‣ Create a new policy
‣ Attach a set of policy keys to it to trigger
mirroring
• Po...
HA with Mirrored Queues
Here is an example of what queues look like when it is replicated/mirrored across multiple
nodes. ...
HA with Mirrored Queues
The most powerful part of this approach is that the policy is applied to queues using a regex
patt...
HA with Mirrored Queues
How it would typically look like. The “definition” is the name you gave the policy
Most powerful pa...
HA with Mirrored Queues
you can see here app.info.ha.two and event.share.ha.two is a “+1”
HA with Mirrored Queues
Walkthrough and demo
Distributed RabbitMQ
Clustering Replication
➡Acts as a single logical unit
➡Consumer sees cluster as
single node
➡Nodes sh...
Distributed RabbitMQ
Replication
Distributed RabbitMQ
Replication
SydneyMelbourne
rabbit01 rabbit01
shovel
federation
• consumes and republishes
• exchange...
Distributed RabbitMQ
Replication
shovel
Melbourne
rabbit01
rabbit02
rabbit02
Sydney
syd.rb01
syd.rb03syd.rb03
Idea is to h...
Replication with Shovel
Walkthrough and demo
Q&A
Upcoming SlideShare
Loading in...5
×

RabbitMQ fairly-indepth

4,358

Published on

Following up from AMQP presentation, this is a more in-depth coverage of RabbitMQ with workshop-style walkthrough, covering various aspects of the system.

Published in: Technology

RabbitMQ fairly-indepth

  1. 1. A somewhat in-depth look
  2. 2. Overview
  3. 3. • Idea was born around the same time AMQP was first drafted by JP Morgan Chase • First released in 2006 • Acquired by SpringSource in 2010 (now part of VMWare) • First version built in under 3 months! • Written in Erlang/OTP RabbitMQ History & Facts
  4. 4. • Used by quite a few major projects including ‣ Huffington Post (Huff Live) ‣ Indeed (Job Aggregation service) ‣ VMWare (vFabric) ‣ SoundCloud (transcoding service) ‣ Mozilla (Pulse) RabbitMQ History & Facts
  5. 5. The Basics
  6. 6. Virtual Host QueuesExchange Binding AMQ Model Recap ➡ Queue: ‣ critical_error_queue ➡ Exchange: ‣ errors_exchange ➡ Filter: ‣ *.error.critical Message Publisher Consumer Final overview of what the AMQ Model is about - in a nutshell... * all within a virtual host (completely isolated) * this is protocol at the server’s service level
  7. 7. QueuesExchange Binding AMQ Model Recap Publisher Consumer ‣ creates a relationship between queues and exchanges ‣ contains criteria and properties ‣ can be created at runtime by consumers ‣ store messages ‣ named ‣ bound-able to exchange ‣ criteria ‣ can be created at runtime by consumers ‣ routes messages based on criteria ‣ doesn’t store messages ‣ can inspect message content ‣ can be created at runtime consumers ‣ can create exchanges and queues ‣ can create exchanges and queues
  8. 8. Fanout Topic Headers System Exchange Types (routing algo) Direct Exchange Types
  9. 9. Direct Exchange “events” type: direct Q: user_clicks X: events K: user.click MessagePublisher Consumer K: user.click
  10. 10. Direct Exchange “events” type: direct Q: user_clicks X: events K: user.click Message Publisher Consumer K: user.view Message K: user.click Message K: user.share
  11. 11. Direct Exchange “events” type: direct Q: user_clicks X: events K: user.click Message Publisher Consumer K: user.view Message K: user.click Message K: user.share Q: user_shares X: events K: user.share Q: user_views X: events K: user.click user.share user.view
  12. 12. Direct Exchange “events” type: direct Q: user_clicks X: events K: user.click Message Publisher Consumer K: user.view Message K: user.click Message K: user.share Q: user_shares X: events K: user.share Q: user_views X: events K: user.click user.share user.view
  13. 13. Direct Exchange “events” type: direct Q: user_clicks X: events K: user.click Message Publisher Consumer K: user.view Message K: user.click Message K: user.share Q: user_shares X: events K: user.share Q: user_views X: events K: user.click user.share user.view
  14. 14. Direct Exchange “events” type: direct Q: user_clicks X: events K: user.click Message Publisher Consumer K: user.view Message K: user.click Message K: user.share Q: user_shares X: events K: user.share Q: user_views X: events K: user.click user.share user.view
  15. 15. Direct Exchange Walkthrough and demo
  16. 16. Topic Exchange “events” type: topic Q: all_errors X: events K: *.error Message Publisher Consumer K: sys.info Message K: sys.error Message K: app.info Q: system_error X: events K: sys.error Q: app_logs_sys_error X: events K: app.* sys.error Message K: app.error
  17. 17. Topic Exchange “events” type: topic Q: all_errors X: events K: *.error Message Publisher Consumer K: sys.info Message K: sys.error Message K: app.info Q: system_error X: events K: sys.error Q: app_logs_sys_error X: events K: app.* sys.error Message K: app.error
  18. 18. Topic Exchange “events” type: topic Q: all_errors X: events K: *.error Message Publisher Consumer K: sys.info Message K: sys.error Message K: app.info Q: system_error X: events K: sys.error Q: app_logs_sys_error X: events K: app.* sys.error Message K: app.error
  19. 19. Topic Exchange “events” type: direct Q: all_errors X: events K: *.error Message Publisher Consumer K: sys.info Message K: sys.error Message K: app.info Q: system_error X: events K: sys.error Q: app_logs_sys_error X: events K: app.* sys.error Message K: app.error
  20. 20. Topic Exchange Walkthrough and demo
  21. 21. Topic Exchange “events” type: topic Q: all_errors X: events K: *.error Message Publisher Consumer K: sys.info Message K: sys.error Message K: app.info Q: system_error X: events K: sys.error Q: app_logs_sys_error X: events K: app.* sys.error Message K: app.error
  22. 22. Persistence & Durability exchangeMessagePublisher ConsumerQueues
  23. 23. Persistence & Durability exchangeMessagePublisher ConsumerQueues ‣persistent
  24. 24. Persistence & Durability exchangeMessagePublisher ConsumerQueues ‣persistent ‣durable
  25. 25. Persistence & Durability exchangeMessagePublisher ConsumerQueues ‣persistent ‣durable ‣durable
  26. 26. Persistence & Durability exchangeMessagePublisher ConsumerQueues ‣persistent ‣durable ‣durable Component will be recreated in the event of reboot/restart. Does not mean its content will be retained
  27. 27. Persistence & Durability exchangeMessagePublisher ConsumerQueues ‣persistent ‣durable ‣durable Component will be recreated in the event of reboot/restart. Does not mean its content will be retained Messages will be persisted to disk. In RabbitMQ, messages are actually queued in memory before written to disk and f’syncd For recoverable messages, queue(s) must be durable
  28. 28. Administration
  29. 29. Administration • Management tools via rabbitmq-plugins • Command: sudo rabbitmq-plugins enable rabbitmq_management • Gives you: •rabbitmqctl • MochiWeb: http://localhost:15672
  30. 30. Administration • Important tools: ‣ rabbitmqctl commands - users, perms, queues, bindings - status, reset ‣ MochiWeb features
  31. 31. Distributed RabbitMQ Clustering & Replication
  32. 32. Distributed RabbitMQ Clustering Replication HA
  33. 33. Distributed RabbitMQ Clustering Replication ➡Acts as a single logical unit ➡Consumer sees cluster as single node ➡Nodes shares the same Erlang cookie ➡Automatic metadata replication, etc ➡Requires reliable LAN-like environment ➡Emphasis on Consistency & Availability HA
  34. 34. Distributed RabbitMQ Clustering Replication ➡Acts as a single logical unit ➡Consumer sees cluster as single node ➡Nodes shares the same Erlang cookie ➡Automatic metadata replication, etc ➡Requires reliable LAN-like environment ➡Emphasis on Consistency & Availability ➡Acts like a forwarding agent ➡Nodes doesn’t share any cookie and is standalone ➡Uses AMQP protocol itself for forwarding ➡Suitable for unreliable WAN connections ➡Emphasis on Availability & Partition Tolerance HA
  35. 35. Distributed RabbitMQ Clustering Replication ➡Acts as a single logical unit ➡Consumer sees cluster as single node ➡Nodes shares the same Erlang cookie ➡Automatic metadata replication, etc ➡Requires reliable LAN-like environment ➡Emphasis on Consistency & Availability ➡Acts like a forwarding agent ➡Nodes doesn’t share any cookie and is standalone ➡Uses AMQP protocol itself for forwarding ➡Suitable for unreliable WAN connections ➡Emphasis on Availability & Partition Tolerance HA ➡A combo of both ➡It’s cluster with replication ➡Usually in form of master/ slave ➡Emphasis on Consistency & Availability
  36. 36. Clustering Clustering
  37. 37. metadata Clustering Clustering metadata metadata rabbit01 rabbit03 rabbit02 Cluster Typical clustering Metadata is replicated across all nodes (operations, queues, exchange and bindings declaration, etc) Let’s see what happens when a queue is created... Let’s say we have a publisher...
  38. 38. metadata Clustering Clustering metadata metadata rabbit01 rabbit03 rabbit02 Cluster P and this is the first time it’s publishing message to a newly declared queue
  39. 39. metadata Clustering Clustering shares_queue metadata metadata rabbit01 rabbit03 rabbit02 Cluster P Through a LB, the connection is established with rabbit01. Publish the message, which creates a queue and then message to it. What should happen?
  40. 40. metadata Clustering Clustering shares_queue metadata metadata rabbit01 rabbit03 rabbit02 Cluster Queues are created on the node called by consumer/publisher and not replicated BUT the metadata is replicated.
  41. 41. metadata Clustering Clustering shares_queue metadata metadata rabbit01 rabbit03 rabbit02 Cluster P clicks_queue Queues are created on the node called by consumer/publisher and not replicated
  42. 42. metadata Clustering Clustering shares_queue metadata metadata clicks_queue rabbit01 rabbit03 rabbit02 Cluster
  43. 43. Clustering Notes • There are three types of nodes: ‣ Stat, Disk and Ram • All nodes must share the same cookie • Works like a shard by default, thus adding new node to cluster improves performance • Upgrade is not automated • Best have one stat-node with WebUI, rest with agents
  44. 44. Important • RabbitMQ was not designed to handle network partition. • Nodes in clusters should have LAN-like reliability
  45. 45. Clustering Walkthrough and demo
  46. 46. Distributed RabbitMQ Clustering Replication ➡Acts as a single logical unit ➡Consumer sees cluster as single node ➡Nodes shares the same Erlang cookie ➡Automatic metadata replication, etc ➡Requires reliable LAN-like environment ➡Emphasis on Consistency & Availability ➡Acts like a forwarding agent ➡Nodes doesn’t share any cookie and is standalone ➡Uses AMQP protocol itself for forwarding ➡Suitable for unreliable WAN connections ➡Emphasis on Availability & Partition Tolerance HA ➡A combo of both ➡It’s cluster with replication ➡Usually in form of master/ slave ➡Emphasis on Consistency & Availability Naturally clustering doesn’t really solve high availability problem as it is a separate issue. Let’s look at HA next
  47. 47. HA with Mirrored Queues metadata shares_queue metadata metadata shares_queue rabbit01 rabbit03 rabbit02 Cluster HA Needs to be in a clustered environment before HA can be achieved
  48. 48. HA with Mirrored Queues metadata shares_queue metadata metadata shares_queue rabbit01 rabbit03 rabbit02 Cluster HA Needs to be in a clustered environment before HA can be achieved
  49. 49. • Set up as a queue policy in RabbitMQ ‣ Create a new policy ‣ Attach a set of policy keys to it to trigger mirroring • Policy key: ha-mode • Policy values: all, exactly(N), nodes(str...) HA with Mirrored Queues
  50. 50. HA with Mirrored Queues Here is an example of what queues look like when it is replicated/mirrored across multiple nodes. “+2” means it’s on two other nodes.
  51. 51. HA with Mirrored Queues The most powerful part of this approach is that the policy is applied to queues using a regex pattern. This makes it extremely powerful.
  52. 52. HA with Mirrored Queues How it would typically look like. The “definition” is the name you gave the policy Most powerful part is you can create something similar to namematch-two-only. Basically this means any future queues that ends with *.ha.two will be automatically mirrored across 2 nodes, like the next slide
  53. 53. HA with Mirrored Queues you can see here app.info.ha.two and event.share.ha.two is a “+1”
  54. 54. HA with Mirrored Queues Walkthrough and demo
  55. 55. Distributed RabbitMQ Clustering Replication ➡Acts as a single logical unit ➡Consumer sees cluster as single node ➡Nodes shares the same Erlang cookie ➡Automatic metadata replication, etc ➡Requires reliable LAN-like environment ➡Emphasis on Consistency & Availability ➡Acts like a forwarding agent ➡Nodes doesn’t share any cookie and is standalone ➡Uses AMQP protocol itself for forwarding ➡Suitable for unreliable WAN connections ➡Emphasis on Availability & Partition Tolerance HA ➡A combo of both ➡It’s cluster with replication ➡Usually in form of master/ slave ➡Emphasis on Consistency & Availability
  56. 56. Distributed RabbitMQ Replication
  57. 57. Distributed RabbitMQ Replication SydneyMelbourne rabbit01 rabbit01 shovel federation • consumes and republishes • exchange to exchange • works well across WAN • two main difference: • shovel is lower level • shovel is more flexible Two main plugins - shovel and federation. Both are very very similar but it seems there’s more push for Shovel. Both does the same things listed here...
  58. 58. Distributed RabbitMQ Replication shovel Melbourne rabbit01 rabbit02 rabbit02 Sydney syd.rb01 syd.rb03syd.rb03 Idea is to have local rabbitmq on web nodes re-publishing messages across WAN to a cluster.
  59. 59. Replication with Shovel Walkthrough and demo
  60. 60. Q&A
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×