Distributed messaging with AMQP

1,303 views

Published on

A presentation on the AMQP protocol with a brief live demo of RabbitMQ to illustrate the protocol.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,303
On SlideShare
0
From Embeds
0
Number of Embeds
30
Actions
Shares
0
Downloads
68
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Distributed messaging with AMQP

  1. 1. Distributed Messaging AMQP, RabbitMQ (Quick demo) Tuesday, 10 September 13
  2. 2. Event Logging Current architecture & problems Tuesday, 10 September 13
  3. 3. Event Logging Server Client Client Client MongoDB Tuesday, 10 September 13
  4. 4. Problem Server Client Client Client 80 110 65 MongoDB Tuesday, 10 September 13
  5. 5. Problem Server Client Client Client 80 110 65 255 MongoDB Tuesday, 10 September 13
  6. 6. Problem Server Client Client Client 0 0 0 0 MongoDB Tuesday, 10 September 13
  7. 7. Problem Server Client Client Client 0 0 0 0 MongoDB Tuesday, 10 September 13
  8. 8. Possible Solutions • Server tweaks:TCP conns, ulimit, etc • Multi-master model • Localised queue • ... and quite a few more Tuesday, 10 September 13
  9. 9. Possible Solutions • Server tweaks:TCP conns, ulimit, etc • Multi-master model •Localised queue • ... and quite a few more Tuesday, 10 September 13
  10. 10. Localised Queues • Fast (no TCP latency) • Network partition tolerant • Modular & scalable • Reliable & Durable Tuesday, 10 September 13
  11. 11. Why Localise? Server Client Client Client 80 110 65 MongoDB Q Q Q < 10 Tuesday, 10 September 13
  12. 12. Why Localise? Server Client Client Client 80 110 65 MongoDB Q Q Q 0 Tuesday, 10 September 13
  13. 13. Why Localise? Server Client Client Client 80 110 65 MongoDB Q Q Q 0 Tuesday, 10 September 13
  14. 14. Why Localise? Server Client Client Client 80 110 65 MongoDB Q Q Q < 10 Tuesday, 10 September 13
  15. 15. Why Localise? Server Client Client Client 80 110 65 MongoDB Q Q Q < 10 Client Q Tuesday, 10 September 13
  16. 16. Queue Criteria • Reliable • Durable • Scalable Tuesday, 10 September 13
  17. 17. AMQP Advanced Message Queueing Protocol Tuesday, 10 September 13
  18. 18. Why AMQP? • Stable and mature • Designed for SX markets • Widely supported (good interop) • Modular, Fast & Flexible Tuesday, 10 September 13
  19. 19. Service ModelWire AMQP Overview AMQP Advanced Message Queueing Protocol Tuesday, 10 September 13
  20. 20. Service Model Wire AMQP Overview AMQP Tuesday, 10 September 13
  21. 21. ➡ Model/component spec ‣ Exchange ‣ Message Queue ‣ Binding AMQ Model Wire AMQP Overview AMQP Tuesday, 10 September 13
  22. 22. ➡ Model/component spec ‣ Exchange ‣ Message Queue ‣ Binding AMQ Model Wire AMQP Overview AMQP ➡ Function layer ‣Transactions, Exchange, Queues, etc ➡Transport layer ‣ Protocol format ‣ Data rep/framing ‣ Multiplexing, heart-beating, etc Tuesday, 10 September 13
  23. 23. AMQ Model Wire AMQP Overview AMQP ➡ Function layer ‣Transactions, Exchange, Queues, etc ➡Transport layer ‣ Protocol format ‣ Data rep/framing ‣ Multiplexing, heart-beating, etc ➡ Model/component spec ‣ Exchange ‣ Message Queue ‣ Binding Tuesday, 10 September 13
  24. 24. ExchangeQueuesBindingAMQ Model AMQ Model Tuesday, 10 September 13
  25. 25. QueuesExchange Binding AMQ Model AMQ Model Tuesday, 10 September 13
  26. 26. Virtual Host QueuesExchange Binding AMQ Model AMQ Model Tuesday, 10 September 13
  27. 27. Virtual Host QueuesExchange Binding AMQ Model AMQ Model Publisher Consumer Tuesday, 10 September 13
  28. 28. Virtual Host QueuesExchange Binding AMQ Model Publisher Consumer Tuesday, 10 September 13
  29. 29. Virtual Host QueuesExchange Binding AMQ Model Router Message Publisher Consumer Tuesday, 10 September 13
  30. 30. Virtual Host QueuesExchange Binding AMQ Model Router Message Publisher Consumer Tuesday, 10 September 13
  31. 31. Virtual Host QueuesExchange Binding AMQ Model Router Message Publisher Consumer Tuesday, 10 September 13
  32. 32. Virtual Host QueuesExchange Binding AMQ Model Router ?Message Publisher Consumer Tuesday, 10 September 13
  33. 33. Virtual Host QueuesExchange Binding AMQ Model Message Router ? Publisher Consumer Tuesday, 10 September 13
  34. 34. Virtual Host QueuesExchange Binding AMQ Model Message Router Publisher ➡ Queue: ‣ critical_error_queue ➡ Exchange: ‣ errors_exchange ➡ Filter: ‣ *.error.critical Consumer Tuesday, 10 September 13
  35. 35. Virtual Host QueuesExchange Binding AMQ Model ➡ Queue: ‣ critical_error_queue ➡ Exchange: ‣ errors_exchange ➡ Filter: ‣ *.error.critical Message Router Publisher Consumer Tuesday, 10 September 13
  36. 36. QueuesExchange Binding AMQP Service Model Tuesday, 10 September 13
  37. 37. QueuesExchange Binding AMQP Service Model ‣ routes messages based on criteria ‣ doesn’t store messages ‣ can inspect message content ‣ can be created at runtime consumers Tuesday, 10 September 13
  38. 38. QueuesExchange Binding AMQP Service Model ‣ store messages ‣ named ‣ bound-able to exchange ‣ criteria ‣ can be created at runtime by consumers ‣ routes messages based on criteria ‣ doesn’t store messages ‣ can inspect message content ‣ can be created at runtime consumers Tuesday, 10 September 13
  39. 39. QueuesExchange Binding AMQP Service Model ‣ creates a relationship between queues and exchanges ‣ contains criteria and properties ‣ can be created at runtime by consumers ‣ store messages ‣ named ‣ bound-able to exchange ‣ criteria ‣ can be created at runtime by consumers ‣ routes messages based on criteria ‣ doesn’t store messages ‣ can inspect message content ‣ can be created at runtime consumers Tuesday, 10 September 13
  40. 40. QueuesExchange Binding AMQP Service Model ‣ creates a relationship between queues and exchanges ‣ contains criteria and properties ‣ can be created at runtime by consumers ‣ store messages ‣ named ‣ bound-able to exchange ‣ criteria ‣ can be created at runtime by consumers ‣ routes messages based on criteria ‣ doesn’t store messages ‣ can inspect message content ‣ can be created at runtime consumers Tuesday, 10 September 13
  41. 41. QueuesExchange Binding AMQP Service Model Publisher Consumer ‣ creates a relationship between queues and exchanges ‣ contains criteria and properties ‣ can be created at runtime by consumers ‣ store messages ‣ named ‣ bound-able to exchange ‣ criteria ‣ can be created at runtime by consumers ‣ routes messages based on criteria ‣ doesn’t store messages ‣ can inspect message content ‣ can be created at runtime consumers Tuesday, 10 September 13
  42. 42. QueuesExchange Binding AMQP Service Model Publisher Consumer ‣ creates a relationship between queues and exchanges ‣ contains criteria and properties ‣ can be created at runtime by consumers ‣ store messages ‣ named ‣ bound-able to exchange ‣ criteria ‣ can be created at runtime by consumers ‣ routes messages based on criteria ‣ doesn’t store messages ‣ can inspect message content ‣ can be created at runtime consumers ‣ can create exchanges and queues ‣ can create exchanges and queues Tuesday, 10 September 13
  43. 43. Exchange AMQP Service Model Tuesday, 10 September 13
  44. 44. FanoutTopicHeadersSystem Exchange AMQP Service Model Types (routing algo) Tuesday, 10 September 13
  45. 45. Fanout Topic Headers System Exchange AMQP Service Model Types (routing algo) Direct Tuesday, 10 September 13
  46. 46. Fanout Topic Headers System Exchange AMQP Service Model Types (routing algo) Direct Tuesday, 10 September 13
  47. 47. Exchange AMQP Service Model Direct Type Tuesday, 10 September 13
  48. 48. Exchange ‘events’ AMQP Service Model Direct Type Tuesday, 10 September 13
  49. 49. Exchange ‘events’ AMQP Service Model Direct Type Message Message Message event.user.click event.user.view event.user.share Tuesday, 10 September 13
  50. 50. Exchange ‘events’ AMQP Service Model Direct Type Message Message Message event.user.click event.user.view event.user.share user_click_queue Tuesday, 10 September 13
  51. 51. Exchange ‘events’ Binding AMQP Service Model ➡Queue: ‣ user_click_queue Direct Type Message Message Message event.user.click event.user.view event.user.share user_click_queue Tuesday, 10 September 13
  52. 52. Exchange ‘events’ Binding AMQP Service Model ➡Queue: ‣ user_click_queue ➡Exchange: ‣ events Direct Type Message Message Message event.user.click event.user.view event.user.share user_click_queue Tuesday, 10 September 13
  53. 53. Exchange ‘events’ Binding AMQP Service Model ➡Queue: ‣ user_click_queue ➡Exchange: ‣ events ➡Routing Key: ‣ event.user.click Direct Type Message Message Message event.user.click event.user.view event.user.share user_click_queue Tuesday, 10 September 13
  54. 54. Exchange ‘events’ Binding AMQP Service Model ➡Queue: ‣ user_click_queue ➡Exchange: ‣ events ➡Routing Key: ‣ event.user.click Direct Type Message Message Message event.user.click event.user.view event.user.share PHP Script PHP Script PHP Script user_click_queue Tuesday, 10 September 13
  55. 55. Exchange ‘events’ AMQP Service Model Direct Type Message Message Message event.user.click event.user.view event.user.share PHP Script PHP Script PHP Script user_click_queue ConsumerPublisher/ Producer Tuesday, 10 September 13
  56. 56. Exchange ‘events’ AMQP Service Model Direct Type Message Message Message event.user.click event.user.view event.user.share user_click_queue user_view_queue user_share_queue Tuesday, 10 September 13
  57. 57. Exchange ‘events’ AMQP Service Model Direct Type Message Message Message event.user.click event.user.view event.user.share user_click_queue user_view_queue user_share_queue Message Message Message Tuesday, 10 September 13
  58. 58. Exchange ‘events’ AMQP Service Model What if... Message Message Message event.user.click event.user.view event.user.share user_interaction_queue user_shares_queue Tuesday, 10 September 13
  59. 59. Exchange ‘events’ AMQP Service Model What if... user_interaction_queue user_shares_queue Message Message Message event.user.click event.user.view event.user.share Tuesday, 10 September 13
  60. 60. Exchange ‘events’ AMQP Service Model What if... Message Message Message event.user.click event.user.view event.user.share user_interaction_queue user_shares_queue Tuesday, 10 September 13
  61. 61. Exchange ‘events’ AMQP Service Model Topic Exchange user_interaction_queue Binding Message Message Message event.user.click event.user.view event.user.share Tuesday, 10 September 13
  62. 62. Exchange ‘events’ AMQP Service Model Topic Exchange user_interaction_queue Binding ➡Queue: ‣ user_interaction_queue ➡Exchange: ‣ events ➡Routing Key: ‣ event.user.* Message Message Message event.user.click event.user.view event.user.share Tuesday, 10 September 13
  63. 63. Exchange ‘events’ AMQP Service Model Topic Exchange user_interaction_queue Binding ➡Queue: ‣ user_interaction_queue ➡Exchange: ‣ events ➡Routing Key: ‣ event.user.* Message Message Message event.user.click event.user.view event.user.share PHP Script PHP Script PHP Script Tuesday, 10 September 13
  64. 64. Exchange ‘events’ AMQP Service Model Topic Exchange user_shares_queue Binding Message Message Message event.user.click event.user.view event.user.share Tuesday, 10 September 13
  65. 65. Exchange ‘events’ AMQP Service Model Topic Exchange user_shares_queue Binding ➡Queue: ‣ user_interaction_queue ➡Exchange: ‣ events ➡Routing Key: ‣ event.user.share Message Message Message event.user.click event.user.view event.user.share Tuesday, 10 September 13
  66. 66. Exchange ‘events’ AMQP Service Model Topic Exchange user_shares_queue Binding ➡Queue: ‣ user_interaction_queue ➡Exchange: ‣ events ➡Routing Key: ‣ event.user.share Message Message Message event.user.click event.user.view event.user.share PHP Script PHP Script PHP Script Tuesday, 10 September 13
  67. 67. Message Exchange ‘events’ AMQP Service Model Topic Exchange Message Message Message event.user.click event.user.view event.user.share user_interaction_queue user_shares_queue Message Message event.user.* event.user.share Tuesday, 10 September 13
  68. 68. Message Exchange ‘events’ AMQP Service Model Topic Exchange Message Message Message event.user.click event.user.view event.user.share user_interaction_queue user_shares_queue Message Message Can be done at runtime event.user.* event.user.share Tuesday, 10 September 13
  69. 69. Message Exchange ‘events’ AMQP Service Model Topic Exchange Message Message Message event.user.click event.user.view event.user.share user_interaction_queue user_shares_queue Message Message Click vs Share Conversion? event.user.* event.user.share Tuesday, 10 September 13
  70. 70. Message Exchange ‘events’ AMQP Service Model Topic Exchange Message Message Message event.user.click event.user.view event.user.share user_interaction_queue user_shares_queue Message Message Click vs Share Conversion? event.user.* event.user.share Message click_share_queue Message event.user.share event.user.click Tuesday, 10 September 13
  71. 71. Message Exchange ‘events’ AMQP Service Model Topic Exchange Message Message Message event.user.click event.user.view event.user.share user_interaction_queue user_shares_queue Message live_view_queue event.user.* event.user.share *.*.view Message Message event.staff.view Message Message Staff vs user views? Tuesday, 10 September 13
  72. 72. AMQ Model Wire AMQP Overview AMQP ➡ Function layer ‣Transactions, Exchange, Queues, etc ➡Transport layer ‣ Protocol format ‣ Data rep/framing ‣ Multiplexing, heart-beating, etc ➡ Model/component spec ‣ Exchange ‣ Message Queue ‣ Binding Tuesday, 10 September 13
  73. 73. ➡ Model/component spec ‣ Exchange ‣ Message Queue ‣ Binding AMQ Model Wire AMQP Overview AMQP ➡ Function layer ‣Transactions, Exchange, Queues, etc ➡Transport layer ‣ Protocol format ‣ Data rep/framing ‣ Multiplexing, heart-beating, etc Tuesday, 10 September 13
  74. 74. Wire Wire Protocol Tuesday, 10 September 13
  75. 75. Wire Wire Protocol ‣ Datagram specification ‣ Protocol header ‣ Payload format ‣ Data fields specification ‣ Content framing ‣ Heartbeat frames ‣ Channel multiplexing ‣ Visibility guarantee ‣ Channel closure ‣ Content synchronisation ‣ Content ordering Tuesday, 10 September 13
  76. 76. Wire Protocol Method Payloads Tuesday, 10 September 13
  77. 77. Wire Protocol Method Payloads Tuesday, 10 September 13
  78. 78. Virtual Host QueuesExchange Binding Summary ➡ Queue: ‣ critical_error_queue ➡ Exchange: ‣ errors_exchange ➡ Filter: ‣ *.error.critical Message Router Publisher Consumer Tuesday, 10 September 13
  79. 79. Revisiting event logging A proposed solution Tuesday, 10 September 13
  80. 80. Event Logging Revisited Client Client Client MongoDB Tuesday, 10 September 13
  81. 81. Event Logging Revisited Client Client Client MQ MQ MQ MongoDB Tuesday, 10 September 13
  82. 82. MongoDB Event Logging Revisited MQClient Client Client 80 110 65 MQ MQ MQ Tuesday, 10 September 13
  83. 83. MongoDB Event Logging Revisited MQClient Client Client 80 110 65 MQ MQ MQ Tuesday, 10 September 13
  84. 84. MongoDB Event Logging Revisited MQClient Client Client 80 110 65 MQ MQ MQ MQ Tuesday, 10 September 13
  85. 85. MongoDB Event Logging Revisited MQ Client Client Client 80 110 65 MQ MQ MQ MQ Tuesday, 10 September 13
  86. 86. Message Exchange ‘events’ AMQP Service Model Topic Exchange Message Message Message event.user.click event.user.view event.user.share user_interaction_queue user_shares_queue Message live_view_queue event.user.* event.user.share *.*.view Message Message event.staff.view Message Message Remember this? Tuesday, 10 September 13
  87. 87. MongoDB Event Logging Revisited MQClient Client Client 80 110 65 MQ MQ MQ Reporting SQL Notification Tuesday, 10 September 13
  88. 88. RabbitMQ An AMQP Implementation Tuesday, 10 September 13
  89. 89. Why RabbitMQ? • Mature (born 2006) • Acquired by SpringSource (now part of VMWare) in 2010 • Used by companies like Nokia, Google, Mozilla, vFabric, SecondLife, OpenStack Tuesday, 10 September 13
  90. 90. Overview • Written in Erlang/OTP • Supports other protocols too - STOMP, MQTT • Many library supports - Python, PHP, Ruby, .NET • RESTful API is available Tuesday, 10 September 13
  91. 91. Hands on Demo Tuesday, 10 September 13
  92. 92. Q&A Tuesday, 10 September 13
  93. 93. Coming soon... RabbitMQ in Depth (sort of) Tuesday, 10 September 13

×