Messaging patterns

12,704
-1

Published on

Published in: Technology, Business

Messaging patterns

  1. 1. Messaging Patterns Álvaro Videla - Liip AGMonday, June 6, 2011
  2. 2. About Me • Developer at Liip AG • Blog: http://videlalvaro.github.com/ • Twitter: @old_soundMonday, June 6, 2011
  3. 3. About Me Co-authoring RabbitMQ in Action http://bit.ly/rabbitmqMonday, June 6, 2011
  4. 4. Why Do I need Messaging?Monday, June 6, 2011
  5. 5. An ExampleMonday, June 6, 2011
  6. 6. Implement a Photo GalleryMonday, June 6, 2011
  7. 7. Two Parts:Monday, June 6, 2011
  8. 8. Pretty SimpleMonday, June 6, 2011
  9. 9. ‘Till new requirements arriveMonday, June 6, 2011
  10. 10. The Product OwnerMonday, June 6, 2011
  11. 11. Can we also notify the user friends when she uploads a new image?Monday, June 6, 2011
  12. 12. Can we also notify the user friends when she uploads a new image? I forgot to mention we need it for tomorrow…Monday, June 6, 2011
  13. 13. The Social Media GuruMonday, June 6, 2011
  14. 14. We need to give badges to users for each picture uploadMonday, June 6, 2011
  15. 15. We need to give badges to users for each picture upload and post uploads to TwitterMonday, June 6, 2011
  16. 16. The SysadminMonday, June 6, 2011
  17. 17. Dumb! You’re delivering full size images! The bandwidth bill has tripled!Monday, June 6, 2011
  18. 18. Dumb! You’re delivering full size images! The bandwidth bill has tripled! We need this fixed for yesterday!Monday, June 6, 2011
  19. 19. The Developer in the other teamMonday, June 6, 2011
  20. 20. I need to call your PHP stuff but from PythonMonday, June 6, 2011
  21. 21. I need to call your PHP stuff but from Python And also Java starting next weekMonday, June 6, 2011
  22. 22. The UserMonday, June 6, 2011
  23. 23. I don’t want to wait till your app resizes my image!Monday, June 6, 2011
  24. 24. YouMonday, June 6, 2011
  25. 25. FML!Monday, June 6, 2011
  26. 26. Let’s see the code evolutionMonday, June 6, 2011
  27. 27. First Implementation: %% image_controller handle(PUT, "/user/image", ReqData) -> image_handler:do_upload(ReqData:get_file()), ok.Monday, June 6, 2011
  28. 28. Second Implementation: %% image_controller handle(PUT, "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), resize_image(Image), ok.Monday, June 6, 2011
  29. 29. Third Implementation: %% image_controller handle(PUT, "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), resize_image(Image), notify_friends(ReqData:get_user()), ok.Monday, June 6, 2011
  30. 30. Fourth Implementation: %% image_controller handle(PUT, "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), resize_image(Image), notify_friends(ReqData:get_user()), add_points_to_user(ReqData:get_user()), ok.Monday, June 6, 2011
  31. 31. Final Implementation: %% image_controller handle(PUT, "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), resize_image(Image), notify_friends(ReqData:get_user()), add_points_to_user(ReqData:get_user()), tweet_new_image(User, Image), ok.Monday, June 6, 2011
  32. 32. Can our code scale to new requirements?Monday, June 6, 2011
  33. 33. What ifMonday, June 6, 2011
  34. 34. What if • We need to speed up image conversionMonday, June 6, 2011
  35. 35. What if • We need to speed up image conversion • User notification has to be sent by emailMonday, June 6, 2011
  36. 36. What if • We need to speed up image conversion • User notification has to be sent by email • Stop tweeting about new imagesMonday, June 6, 2011
  37. 37. What if • We need to speed up image conversion • User notification has to be sent by email • Stop tweeting about new images • Resize in different formatsMonday, June 6, 2011
  38. 38. Can we do better?Monday, June 6, 2011
  39. 39. Sure. Using messagingMonday, June 6, 2011
  40. 40. Design Publish / Subscribe PatternMonday, June 6, 2011
  41. 41. First Implementation: %% image_controller handle(PUT, "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), Msg = #msg{user = ReqData:get_user(), image = Image}, publish_message(new_image, Msg).Monday, June 6, 2011
  42. 42. First Implementation: %% image_controller handle(PUT, "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), Msg = #msg{user = ReqData:get_user(), image = Image}, publish_message(new_image, Msg). %% friends notifier on(new_image, Msg) -> notify_friends(Msg.user, Msg.image).Monday, June 6, 2011
  43. 43. First Implementation: %% image_controller handle(PUT, "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), Msg = #msg{user = ReqData:get_user(), image = Image}, publish_message(new_image, Msg). %% friends notifier on(new_image, Msg) -> notify_friends(Msg.user, Msg.image). %% points manager on(new_image, Msg) -> add_points(Msg.user, new_image).Monday, June 6, 2011
  44. 44. First Implementation: %% image_controller handle(PUT, "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), Msg = #msg{user = ReqData:get_user(), image = Image}, publish_message(new_image, Msg). %% friends notifier on(new_image, Msg) -> notify_friends(Msg.user, Msg.image). %% points manager on(new_image, Msg) -> add_points(Msg.user, new_image). %% resizer on(new_image, Msg) -> resize_image(Msg.image).Monday, June 6, 2011
  45. 45. Second Implementation:Monday, June 6, 2011
  46. 46. Second Implementation: %% there’s none.Monday, June 6, 2011
  47. 47. MessagingMonday, June 6, 2011
  48. 48. Messaging • Share data across processesMonday, June 6, 2011
  49. 49. Messaging • Share data across processes • Processes can be part of different appsMonday, June 6, 2011
  50. 50. Messaging • Share data across processes • Processes can be part of different apps • Apps can live in different machinesMonday, June 6, 2011
  51. 51. Messaging • Share data across processes • Processes can be part of different apps • Apps can live in different machines • Communication is AsynchronousMonday, June 6, 2011
  52. 52. Main ConceptsMonday, June 6, 2011
  53. 53. Main Concepts • Messages are sent by ProducersMonday, June 6, 2011
  54. 54. Main Concepts • Messages are sent by Producers • Messages are delivered to ConsumersMonday, June 6, 2011
  55. 55. Main Concepts • Messages are sent by Producers • Messages are delivered to Consumers • Messages goes through a ChannelMonday, June 6, 2011
  56. 56. Messaging and RabbitMQMonday, June 6, 2011
  57. 57. What is RabbitMQ?Monday, June 6, 2011
  58. 58. RabbitMQ • Enterprise Messaging System • Open Source MPL • Written in Erlang/OTP • Commercial Support • Messaging via AMQPMonday, June 6, 2011
  59. 59. Features • Reliable and High Scalable • Easy To install • Easy To Cluster • Runs on: Windows, Solaris, Linux, OSX • AMQP 0.8 - 0.9.1Monday, June 6, 2011
  60. 60. Client Libraries • Java • .NET/C# • Erlang • Ruby, Python, PHP, Perl, AS3, Lisp, Scala, Clojure, HaskellMonday, June 6, 2011
  61. 61. AMQP • Advanced Message Queuing Protocol • Suits Interoperability • Completely Open Protocol • Binary ProtocolMonday, June 6, 2011
  62. 62. Message Flow http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/chap-Messaging_Tutorial-Initial_Concepts.htmlMonday, June 6, 2011
  63. 63. AMQP Model • Exchanges • Message Queues • Bindings • Rules for binding themMonday, June 6, 2011
  64. 64. Exchange Types • Fanout • Direct • TopicMonday, June 6, 2011
  65. 65. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Fanout_Exchange.htmlMonday, June 6, 2011
  66. 66. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Direct_Exchange.htmlMonday, June 6, 2011
  67. 67. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Topic_Exchange.htmlMonday, June 6, 2011
  68. 68. Messaging PatternsMonday, June 6, 2011
  69. 69. There are many messaging patterns http://www.eaipatterns.com/Monday, June 6, 2011
  70. 70. Basic PatternsMonday, June 6, 2011
  71. 71. Competing Consumers How can a messaging client process multiple messages concurrently?Monday, June 6, 2011
  72. 72. Competing Consumers Create multiple Competing Consumers on a single channel so that the consumers can process multiple messages concurrently.Monday, June 6, 2011
  73. 73. Competing ConsumersMonday, June 6, 2011
  74. 74. Publisher Code init(Exchange, Queue) -> #exchange.declare{exchange = Exchange, type = <<"direct">>, durable = true}, #queue.declare{queue = Queue, durable = false}, #queue.bind{queue = Queue, exchange = Exchange}. publish_msg(Exchange, Payload) -> Props = #P_basic{content_type = <<"application/json">>, delivery_mode = 2}, %% persistent publish(Exchange, #amqp_msg{props = Props, payload = Payload}).Monday, June 6, 2011
  75. 75. Consumer Code init_consumer(Exchange, Queue) -> init(Exchange, Queue), #basic.consume{ticket = 0, queue = Queue}. on(#basic.deliver{delivery_tag = DeliveryTag}, #amqp_msg{} = Msg) -> do_something_with_msg(Msg), #basic.ack{delivery_tag = DeliveryTag}.Monday, June 6, 2011
  76. 76. Publish/Subscribe How can the sender broadcast an event to all interested receivers?Monday, June 6, 2011
  77. 77. Publish/Subscribe Send the event on a Publish-Subscribe Channel, which delivers a copy of a particular event to each receiver.Monday, June 6, 2011
  78. 78. Publish/SubscribeMonday, June 6, 2011
  79. 79. Publisher Code init(Exchange, Queue) -> #exchange.declare{exchange = Exchange, type = <<"fanout">>, %% different type durable = true} %% same as before ... publish_msg(Exchange, Payload) -> Props = #P_basic{content_type = <<"application/json">>, delivery_mode = 2}, %% persistent publish(Exchange, #amqp_msg{props = Props, payload = Payload}).Monday, June 6, 2011
  80. 80. Consumer Code A init_consumer(Exchange, ResizeImageQueue) -> init(Exchange, ResizeImageQueue), #basic.consume{queue = ResizeImageQueue}. on(#basic.deliver{delivery_tag = DeliveryTag}, #amqp_msg{} = Msg) -> resize_message(Msg), #basic.ack{delivery_tag = DeliveryTag}.Monday, June 6, 2011
  81. 81. Consumer Code B init_consumer(Exchange, NotifyFriendsQueue) -> init(Exchange, NotifyFriendsQueue), #basic.consume{queue = NotifyFriendsQueue}. on(#basic.deliver{delivery_tag = DeliveryTag}, #amqp_msg{} = Msg) -> notify_friends(Msg), #basic.ack{delivery_tag = DeliveryTag}.Monday, June 6, 2011
  82. 82. Consumer Code C init_consumer(Exchange, LogImageUpload) -> init(Exchange, LogImageUpload), #basic.consume{queue = LogImageUpload}. on(#basic.deliver{delivery_tag = DeliveryTag}, #amqp_msg{} = Msg) -> log_image_upload(Msg), #basic.ack{delivery_tag = DeliveryTag}.Monday, June 6, 2011
  83. 83. Request/Reply When an application sends a message, how can it get a response from the receiver?Monday, June 6, 2011
  84. 84. Request/Reply Send a pair of Request-Reply messages, each on its own channel.Monday, June 6, 2011
  85. 85. Request/ReplyMonday, June 6, 2011
  86. 86. Return Address How does a replier know where to send the reply?Monday, June 6, 2011
  87. 87. Return Address The request message should contain a Return Address that indicates where to send the reply message.Monday, June 6, 2011
  88. 88. Return AddressMonday, June 6, 2011
  89. 89. Correlation Identifier How does a requestor that has received a reply know which request this is the reply for?Monday, June 6, 2011
  90. 90. Correlation Identifier Each reply message should contain a Correlation Identifier, a unique identifier that indicates which request message this reply is for.Monday, June 6, 2011
  91. 91. Correlation IdentifierMonday, June 6, 2011
  92. 92. Putting it all togetherMonday, June 6, 2011
  93. 93. RPC Client init() -> #queue.declare_ok{queue = SelfQueue} = #queue.declare{exclusive = true, auto_delete = true}, #basic.consume{queue = SelfQueue, no_ack = true}, SelfQueue.Monday, June 6, 2011
  94. 94. RPC Client init() -> #queue.declare_ok{queue = SelfQueue} = #queue.declare{exclusive = true, auto_delete = true}, #basic.consume{queue = SelfQueue, no_ack = true}, SelfQueue. request(Payload, RequestId) -> Props = #P_basic{correlation_id = RequestId, reply_to = SelfQueue}, publish(ServerExchange, #amqp_msg{props = Props, payload = Payload}).Monday, June 6, 2011
  95. 95. RPC Client init() -> #queue.declare_ok{queue = SelfQueue} = #queue.declare{exclusive = true, auto_delete = true}, #basic.consume{queue = SelfQueue, no_ack = true}, SelfQueue. request(Payload, RequestId) -> Props = #P_basic{correlation_id = RequestId, reply_to = SelfQueue}, publish(ServerExchange, #amqp_msg{props = Props, payload = Payload}). on(#basic.deliver{}, #amqp_msg{props = Props, payload = Payload}) -> CorrelationId = Props.correlation_id, do_something_with_reply(Payload).Monday, June 6, 2011
  96. 96. RPC Server on(#basic.deliver{}, #amqp_msg{props = Props, payload = Payload}) -> CorrelationId = Props.correlation_id, ReplyTo = Props.reply_to, Reply = process_request(Payload), NewProps = #P_basic{correlation_id = CorrelationId}, publish("", %% anonymous exchange #amqp_msg{props = NewProps, payload = Reply}, ReplyTo). %% routing keyMonday, June 6, 2011
  97. 97. Advanced PatternsMonday, June 6, 2011
  98. 98. Control Bus How can we effectively administer a messaging system that is distributed across multiple platforms and a wide geographic area?Monday, June 6, 2011
  99. 99. Control Bus Use a Control Bus to manage an enterprise integration system.Monday, June 6, 2011
  100. 100. Control Bus • Send Configuration Messages • Start/Stop Services • Inject Test Messages • Collect StatisticsMonday, June 6, 2011
  101. 101. Control BusMonday, June 6, 2011
  102. 102. Control Bus Make Services “Control Bus” EnabledMonday, June 6, 2011
  103. 103. Detour How can you route a message through intermediate steps to perform validation, testing or debugging functions?Monday, June 6, 2011
  104. 104. Detour Construct a Detour with a context-based router controlled via the Control Bus. In one state the router routes incoming messages through additional steps while in the other it routes messages directly to the destination channel.Monday, June 6, 2011
  105. 105. DetourMonday, June 6, 2011
  106. 106. Wire Tap How do you inspect messages that travel on a point-to-point channel?Monday, June 6, 2011
  107. 107. Wire Tap Insert a simple Recipient List into the channel that publishes each incoming message to the main channel and a secondary channel.Monday, June 6, 2011
  108. 108. Wire Tap How do you inspect messages that travel on a point-to-point channel?Monday, June 6, 2011
  109. 109. Wire Tap Insert a simple Recipient List into the channel that publishes each incoming message to the main channel and a secondary channel.Monday, June 6, 2011
  110. 110. Wire TapMonday, June 6, 2011
  111. 111. Smart Proxy How can you track messages on a service that publishes reply messages to the Return Address specified by the requestor?Monday, June 6, 2011
  112. 112. Smart Proxy Use a Smart Proxy to store the Return Address supplied by the original requestor and replace it with the address of the Smart Proxy. When the service sends the reply message route it to the original Return Address.Monday, June 6, 2011
  113. 113. Smart ProxyMonday, June 6, 2011
  114. 114. Credits Pattern graphics and description taken from: http://www.eaipatterns.com/Monday, June 6, 2011
  115. 115. Thanks! @old_sound http://vimeo.com/user1169087 http://www.slideshare.net/old_soundMonday, June 6, 2011
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×