ADD-01RabbitMQ: Messaging in the Cloud          Matthew Sackman         matthew@rabbitmq.com
I  P  C     . . .
I  P  C     . . .     Messaging, messaging, messaging: What does it all mean?
I  P  C     . . .     Messaging, messaging, messaging: What does it all mean?     Wha...
I  P  C     . . .     Messaging, messaging, messaging: What does it all mean?     Wha...
I  P  C     . . .     Messaging, messaging, messaging: What does it all mean?     Wha...
W   ?  M      :     Scalability issues demand greater flexibility from...
RMQ   CT  R   H         Several cloud infrastructures have been built using Rabb...
W  ?              What is Messaging?
W  ?              What is Messaging?              What is a Banana?
W  ?        What can I use Messaging for?
W  ?        What can I use Messaging for?         What can I use a Banana for?
W  ?   What can I use a Messaging protocol for?
U   M PI’      , ?                              Decoupling        ...
U   M PI’      , ?                              Decoupling        ...
U   M PI’      , ?                   Bidirectional Decoupling     ...
U   M PI’      , ?                   Bidirectional Decoupling     ...
U   M PI’      , ?                  Pipelining and Decoupling     ...
U   M PI’      , ?                  Pipelining and Decoupling     ...
U   M PI’      , ?                  Pipelining and Decoupling     ...
U   M PI’      , ?                  Pipelining and Decoupling     ...
U   M PI’      , ?           Work-distribution and Decoupling     ...
U   M PI’      , ?           Work-distribution and Decoupling     ...
U   M PI’      , ?           Work-distribution and Decoupling     ...
U   M PI’      , ?           Work aggregation, distribution and   ...
U   M PT I ’       O               ...
U   M PT I ’       O               ...
AMQP                AMQP:   Advanced Message Queueing Protocol
W AMQP?I’  Advanced,     !    AMQP    . . .          All resources are dyna...
W AMQP?I’  Advanced,     !    AMQP    . . .          All resources are dyna...
AMQP A != C                          Create an exchange,. . .                                    X    ...
AMQP A != C                          . . . create a queue,. . .                          X            ...
AMQP A != C                          . . . add a binding,. . .                          X             ...
AMQP A != C                          . . . all inside a broker.                          X            ...
AMQP A != C    Publish a message, . . .      msg                               X                      ...
AMQP A != C                      . . . it sits in a queue.                           X                ...
AMQP A != C    Publish another message, . . .     msg 2                             X                 ...
AMQP A != C                               . . . it also goes to the queue.                           X...
AMQP A != C    Consuming from the queue retrieves the messages in order.                           X  ...
AMQP A != C    Consuming from the queue retrieves the messages in order.                           X  ...
AMQP A != C    Consuming from the queue retrieves the messages in order.                           X  ...
AMQP A != C    Publish many messages,. . .      F   E   D   C   B   A                                X...
AMQP A != C                          . . . and they are distributed amongst several    consumers on th...
AMQP A != C    We can create a second queue and bind it to the same exchange.                         ...
AMQP A != C    Messages go to every queue bound to a fanout exchange . . .       C   B   A            ...
AMQP A != C    Messages go to every queue bound to a fanout exchange . . .                            ...
AMQP A != C                                                         . . . and the    queues can be con...
AMQP A != C                                                         . . . and the    queues can be con...
AMQP A != C                                                      . . . and the    queues can be consum...
AMQP A != C    We replace “my_exchange” with a direct exchange.                                     od...
AMQP A != C    The routing key of a published message selects which queues the    message goes to.    ...
AMQP A != C    The routing key of a published message selects which queues the    message goes to.    ...
AMQP A != C    The routing key of a published message selects which queues the    message goes to.    ...
AMQP A != C    Messages with no matching bindings are discarded.      C rk = "te                 quila...
AMQP A != C    Messages with no matching bindings are discarded.                                     o...
AMQP A != C    Topic exchanges permit wildcards in the binding key.                                   ...
AMQP A != C    Topic exchanges permit wildcards in the binding key.    Keys are .-separated lists, e.g...
AMQP A != C    Topic exchanges permit wildcards in the binding key.    Keys are .-separated lists, e.g...
AMQP A != C    Topic exchanges permit wildcards in the binding key.    Keys are .-separated lists, e.g...
AMQP A != C    Topic exchanges permit wildcards in the binding key.    Keys are .-separated lists, e.g...
AMQP A != C    Topic exchanges permit wildcards in the binding key.    Keys are .-separated lists, e.g...
AMQP A != C    Topic exchanges permit wildcards in the binding key.    Keys are .-separated lists, e.g...
AMQP A != C    Topic exchanges permit wildcards in the binding key.    Keys are .-separated lists, e.g...
AMQP A != C    Topic exchanges permit wildcards in the binding key.    Keys are .-separated lists, e.g...
AMQP A != C    Topic exchanges permit wildcards in the binding key.    Keys are .-separated lists, e.g...
AMQP A != C    Topic exchanges permit wildcards in the binding key.    Keys are .-separated lists, e.g...
O    AMQPY,                 Errors can be raised for messages that do...
C: W AMQP?I’  Advanced,     !    A O Protocol      . . .        ...
RMQ
I RMQN       RMQ, AMQP & M         RabbitMQ consists of the broker (server) a...
I RMQN       RMQ, AMQP & M         RabbitMQ consists of the broker (server) a...
I RMQN            The broker is written in Erlang: an excellent programming         langua...
RMQ F     A single queue can run up around 30,000 messages per     second, depending on payload, clients, and ...
C     RabbitMQ is a leading implementation of AMQP, and has wide     adoption from a large community across a la...
A . . .C I   P ?    “All our messages are incredibly important and must never ever be lost          ...
A . . .C I   P ?    “All our messages are incredibly important and must never ever be lost          ...
A . . .C I   P ?        “We need guaranteed exactly-once delivery – I want to send 1      message, a...
A . . .C I   P ?        “We need guaranteed exactly-once delivery – I want to send 1      message, a...
T EW !           Thank you             Questions?
Upcoming SlideShare
Loading in …5
×

Rabbit mq messaginginthecloud_v_mworld_2010_ms

394 views
344 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
394
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Rabbit mq messaginginthecloud_v_mworld_2010_ms

  1. 1. ADD-01RabbitMQ: Messaging in the Cloud Matthew Sackman matthew@rabbitmq.com
  2. 2. I  P C     . . .
  3. 3. I  P C     . . . Messaging, messaging, messaging: What does it all mean?
  4. 4. I  P C     . . . Messaging, messaging, messaging: What does it all mean? What is this angular mutant orange Rabbit thing, and why is it dropping AMQP all over the carpet?
  5. 5. I  P C     . . . Messaging, messaging, messaging: What does it all mean? What is this angular mutant orange Rabbit thing, and why is it dropping AMQP all over the carpet? How many balloons does it take to keep a Rabbit in a cloud?
  6. 6. I  P C     . . . Messaging, messaging, messaging: What does it all mean? What is this angular mutant orange Rabbit thing, and why is it dropping AMQP all over the carpet? How many balloons does it take to keep a Rabbit in a cloud? Prizes! Prizes! Prizes!
  7. 7. W   ? M      : Scalability issues demand greater flexibility from application developers Traditional synchronous programming models fair poorly at large scale Cloud computing permits greater dynamic scaling than ever seen before, but applications need to be written well to take advantage of this Messaging enables scaling by decoupling components and adding flexibility
  8. 8. RMQ   CT  R   H Several cloud infrastructures have been built using RabbitMQ as the nervous system of the cloud RabbitMQ is easily installed and used by clients on existing clouds such as Amazon EC2 RabbitMQ is available as an additional component on Heroku RabbitMQ is going to become available in more clouds in the future, e.g. Social, Nebula, VMforce RabbitMQ is just as easy to use to solve problems in clouds as it is to solve problems on the ground
  9. 9. W  ? What is Messaging?
  10. 10. W  ? What is Messaging? What is a Banana?
  11. 11. W  ? What can I use Messaging for?
  12. 12. W  ? What can I use Messaging for? What can I use a Banana for?
  13. 13. W  ? What can I use a Messaging protocol for?
  14. 14. U   M PI’      , ? Decoupling Producer Consumer
  15. 15. U   M PI’      , ? Decoupling Producer Consumer E.g. website passing orders to a credit-card charging engine
  16. 16. U   M PI’      , ? Bidirectional Decoupling Caller Callee
  17. 17. U   M PI’      , ? Bidirectional Decoupling Caller Callee E.g. remote procedure call
  18. 18. U   M PI’      , ? Pipelining and Decoupling Producer Consumer Producer Consumer
  19. 19. U   M PI’      , ? Pipelining and Decoupling Producer Consumer Producer Consumer E.g. combining messages with records in a database
  20. 20. U   M PI’      , ? Pipelining and Decoupling Producer Consumer Producer Consumer E.g. combining messages with records in a database
  21. 21. U   M PI’      , ? Pipelining and Decoupling Producer Consumer Producer Consumer E.g. combining messages with records in a database
  22. 22. U   M PI’      , ? Work-distribution and Decoupling Consumer Producer Consumer
  23. 23. U   M PI’      , ? Work-distribution and Decoupling Consumer Producer Consumer Both duplication and round-robin.
  24. 24. U   M PI’      , ? Work-distribution and Decoupling Consumer Producer Consumer Both duplication and round-robin. Duplication can be used for logging messages at certain points in the system. Round-robin can be used for horizontal scaling.
  25. 25. U   M PI’      , ? Work aggregation, distribution and Decoupling Producer Consumer Producer Consumer
  26. 26. U   M PT I ’    O       Store-and-forward of messages Absorbing spikes of activity (again, decoupling) Routing to different consumers based on message properties
  27. 27. U   M PT I ’    O       Store-and-forward of messages Absorbing spikes of activity (again, decoupling) Routing to different consumers based on message properties G  Messaging protocols as a means of decoupling events is an extremely common and pervasive task There is an enormous range of applications and problems to which using a messaging protocol is an effective solution
  28. 28. AMQP AMQP: Advanced Message Queueing Protocol
  29. 29. W AMQP?I’  Advanced,     ! AMQP    . . . All resources are dynamically created and destroyed by clients as they need them – no static preconfiguration A clean and simple model: just three key nouns to learn Open standard, developed by the AMQP Working Group (we’re members) Lots of client libraries available in many languages, for free
  30. 30. W AMQP?I’  Advanced,     ! AMQP    . . . All resources are dynamically created and destroyed by clients as they need them – no static preconfiguration A clean and simple model: just three key nouns to learn Open standard, developed by the AMQP Working Group (we’re members) Lots of client libraries available in many languages, for free An excellent, freely available, open source broker implementation, called RabbitMQ. . .
  31. 31. AMQP A != C Create an exchange,. . . X "my_exchange" type = fanout
  32. 32. AMQP A != C . . . create a queue,. . . X "my_exchange" "my_queue" type = fanout
  33. 33. AMQP A != C . . . add a binding,. . . X "my_exchange" "my_queue" type = fanout
  34. 34. AMQP A != C . . . all inside a broker. X "my_exchange" "my_queue" type = fanout
  35. 35. AMQP A != C Publish a message, . . . msg X "my_exchange" "my_queue" type = fanout
  36. 36. AMQP A != C . . . it sits in a queue. X msg "my_exchange" "my_queue" type = fanout
  37. 37. AMQP A != C Publish another message, . . . msg 2 X msg "my_exchange" "my_queue" type = fanout
  38. 38. AMQP A != C . . . it also goes to the queue. X msg 2 msg "my_exchange" "my_queue" type = fanout
  39. 39. AMQP A != C Consuming from the queue retrieves the messages in order. X msg 2 msg "my_exchange" "my_queue" type = fanout
  40. 40. AMQP A != C Consuming from the queue retrieves the messages in order. X msg 2 "my_exchange" "my_queue" msg type = fanout
  41. 41. AMQP A != C Consuming from the queue retrieves the messages in order. X "my_exchange" "my_queue" msg 2 type = fanout
  42. 42. AMQP A != C Publish many messages,. . . F E D C B A X "my_exchange" "my_queue" type = fanout
  43. 43. AMQP A != C . . . and they are distributed amongst several consumers on the same queue. A D X E B "my_exchange" "my_queue" F C type = fanout
  44. 44. AMQP A != C We can create a second queue and bind it to the same exchange. X "my_queue_2" "my_exchange" type = fanout "my_queue"
  45. 45. AMQP A != C Messages go to every queue bound to a fanout exchange . . . C B A X "my_queue_2" "my_exchange" type = fanout "my_queue"
  46. 46. AMQP A != C Messages go to every queue bound to a fanout exchange . . . C B A X "my_queue_2" "my_exchange" C B A type = fanout "my_queue"
  47. 47. AMQP A != C . . . and the queues can be consumed from at different rates. C B A X "my_queue_2" "my_exchange" C B A type = fanout "my_queue"
  48. 48. AMQP A != C . . . and the queues can be consumed from at different rates. C X "my_queue_2" C A "my_exchange" type = fanout "my_queue" B
  49. 49. AMQP A != C . . . and the queues can be consumed from at different rates. C X "my_queue_2" "my_exchange" type = fanout "my_queue"
  50. 50. AMQP A != C We replace “my_exchange” with a direct exchange. odka" bk = "v X bk = "b "my_queue_2" eer "my_exchange" " type = direct "my_queue" rk: Routing Key bk: Binding Key
  51. 51. AMQP A != C The routing key of a published message selects which queues the message goes to. A rk = "vo dka" odka" bk = "v X bk = "b "my_queue_2" eer "my_exchange" " type = direct "my_queue" rk: Routing Key bk: Binding Key
  52. 52. AMQP A != C The routing key of a published message selects which queues the message goes to. B r k = "b e er" odka" A bk = "v X bk = "b "my_queue_2" eer "my_exchange" " type = direct "my_queue" rk: Routing Key bk: Binding Key
  53. 53. AMQP A != C The routing key of a published message selects which queues the message goes to. odka" A bk = "v X bk = "b "my_queue_2" eer "my_exchange" " B type = direct "my_queue" rk: Routing Key bk: Binding Key
  54. 54. AMQP A != C Messages with no matching bindings are discarded. C rk = "te quila" odka" A bk = "v X bk = "b "my_queue_2" eer "my_exchange" " B type = direct "my_queue" rk: Routing Key bk: Binding Key
  55. 55. AMQP A != C Messages with no matching bindings are discarded. odka" A bk = "v X bk = "b "my_queue_2" eer "my_exchange" " B type = direct "my_queue" rk: Routing Key bk: Binding Key
  56. 56. AMQP A != C Topic exchanges permit wildcards in the binding key. .*.c" bk = "a X bk = "# "my_queue_2" .e " "my_exchange" type = topic "my_queue" rk: Routing Key bk: Binding Key
  57. 57. AMQP A != C Topic exchanges permit wildcards in the binding key. Keys are .-separated lists, e.g. “stocks.nyse.vmw” * matches any 1 element. # matches zero or more elements. .*.c" bk = "a X bk = "# "my_queue_2" .e " "my_exchange" type = topic "my_queue" rk: Routing Key bk: Binding Key
  58. 58. AMQP A != C Topic exchanges permit wildcards in the binding key. Keys are .-separated lists, e.g. “stocks.nyse.vmw” * matches any 1 element. # matches zero or more elements. A rk = "a" B rk = "a.c" bk = "a .*.c" C rk = "a.b.c" X bk = "# "my_queue_2" .e " D rk = "a.e" "my_exchange" E rk = "b.e" type = topic "my_queue" F rk = "e" rk: Routing Key bk: Binding Key
  59. 59. AMQP A != C Topic exchanges permit wildcards in the binding key. Keys are .-separated lists, e.g. “stocks.nyse.vmw” * matches any 1 element. # matches zero or more elements. A rk = "a" B rk = "a.c" bk = "a .*.c" C rk = "a.b.c" X bk = "# "my_queue_2" .e" D rk = "a.e" "my_exchange" E rk = "b.e" type = topic "my_queue" F rk = "e" rk: Routing Key bk: Binding Key
  60. 60. AMQP A != C Topic exchanges permit wildcards in the binding key. Keys are .-separated lists, e.g. “stocks.nyse.vmw” * matches any 1 element. # matches zero or more elements. B rk = "a.c" bk = "a .*.c" C rk = "a.b.c" X bk = "# "my_queue_2" .e " D rk = "a.e" "my_exchange" E rk = "b.e" type = topic "my_queue" F rk = "e" rk: Routing Key bk: Binding Key
  61. 61. AMQP A != C Topic exchanges permit wildcards in the binding key. Keys are .-separated lists, e.g. “stocks.nyse.vmw” * matches any 1 element. # matches zero or more elements. B rk = "a.c" bk = "a .*.c" C rk = "a.b.c" X bk = "# "my_queue_2" .e" D rk = "a.e" "my_exchange" E rk = "b.e" type = topic "my_queue" F rk = "e" rk: Routing Key bk: Binding Key
  62. 62. AMQP A != C Topic exchanges permit wildcards in the binding key. Keys are .-separated lists, e.g. “stocks.nyse.vmw” * matches any 1 element. # matches zero or more elements. .*.c" bk = "a C rk = "a.b.c" X bk = "# "my_queue_2" .e " D rk = "a.e" "my_exchange" E rk = "b.e" type = topic "my_queue" F rk = "e" rk: Routing Key bk: Binding Key
  63. 63. AMQP A != C Topic exchanges permit wildcards in the binding key. Keys are .-separated lists, e.g. “stocks.nyse.vmw” * matches any 1 element. # matches zero or more elements. .*.c" C bk = "a X bk = "# "my_queue_2" .e " D rk = "a.e" "my_exchange" E rk = "b.e" type = topic "my_queue" F rk = "e" rk: Routing Key bk: Binding Key
  64. 64. AMQP A != C Topic exchanges permit wildcards in the binding key. Keys are .-separated lists, e.g. “stocks.nyse.vmw” * matches any 1 element. # matches zero or more elements. .*.c" C bk = "a X bk = "# "my_queue_2" .e " "my_exchange" D E rk = "b.e" type = topic "my_queue" F rk = "e" rk: Routing Key bk: Binding Key
  65. 65. AMQP A != C Topic exchanges permit wildcards in the binding key. Keys are .-separated lists, e.g. “stocks.nyse.vmw” * matches any 1 element. # matches zero or more elements. .*.c" C bk = "a X bk = "# "my_queue_2" .e " "my_exchange" E D type = topic "my_queue" F rk = "e" rk: Routing Key bk: Binding Key
  66. 66. AMQP A != C Topic exchanges permit wildcards in the binding key. Keys are .-separated lists, e.g. “stocks.nyse.vmw” * matches any 1 element. # matches zero or more elements. .*.c" C bk = "a X bk = "# "my_queue_2" .e " "my_exchange" F E D type = topic "my_queue" rk: Routing Key bk: Binding Key
  67. 67. O    AMQPY,         Errors can be raised for messages that do not get routed to any queues Messages can be consumed so that the broker does not forget about the message until the client explicitly acknowledges the message Messages can be published with a property indicating whether the message should be written to disk Transactions: making publication and acknowledgement of several messages atomic Flow control: e.g. used to stop publishers from overwhelming the broker in extreme situations
  68. 68. C: W AMQP?I’  Advanced,     ! A O Protocol      . . . Makes it easier to have multiple implementations that interoperate at the wire-level Avoids vendor lock-in: easy to rip out and replace individual components with alternative implementations Allows third-parties to write client libraries for other languages Decouples flag-day upgrades for both client libraries and broker Allows third-party traffic analysis tools to inspect and decode interactions between the clients and brokers Promotes similarities in APIs presented by different client libraries
  69. 69. RMQ
  70. 70. I RMQN    RMQ, AMQP & M RabbitMQ consists of the broker (server) and several clients (Java, .Net, plus others) They speak the AMQP protocol to each other Anyone can write (and many have) other clients which also speak AMQP and work with RabbitMQ
  71. 71. I RMQN    RMQ, AMQP & M RabbitMQ consists of the broker (server) and several clients (Java, .Net, plus others) They speak the AMQP protocol to each other Anyone can write (and many have) other clients which also speak AMQP and work with RabbitMQ RabbitMQ is freely available and open source, licensed under the Mozilla Public License v1.1 It’s already in many popular Linux distributions (Ubuntu, Debian, Fedora, Gentoo) and can be easily installed on OS X both through MacPorts and Homebrew
  72. 72. I RMQN    The broker is written in Erlang: an excellent programming language and platform for the task A mere 17k lines of code in the broker Supports clustering for increased scalability Supports several extension points via plugins which are frequently used to extend RabbitMQ, e.g. STOMP, XMPP adaptors; The Shovel; additional exchange types. . . Can work with Pacemaker and associated tools to provide various forms of High Availability
  73. 73. RMQ F A single queue can run up around 30,000 messages per second, depending on payload, clients, and properties of the messages But several queues together can achieve much higher throughput, and still keep the queues empty RabbitMQ 2.0 gains the ability to send messages to disk to free up memory, thus allowing queue depths to grow, bounded only by disk space, not RAM. Queues of 10s of millions of items are easily accommodated Adding extension to RabbitMQ based on feedback from our users, e.g. queue expiry, queue-message TTL, publisher acknowledgements
  74. 74. C RabbitMQ is a leading implementation of AMQP, and has wide adoption from a large community across a large number of languages and problem domains Website and downloads available at http://www.rabbitmq.com/ On Ubuntu and Debian, just an apt-get install rabbitmq-server away. Similarly easy for Fedora and many other Linux distributions For OS X, it’s in MacPorts Full easy-to-install Windows bundles available from the website
  75. 75. A . . .C I   P ? “All our messages are incredibly important and must never ever be lost under any circumstances”.
  76. 76. A . . .C I   P ? “All our messages are incredibly important and must never ever be lost under any circumstances”. Fact: There are always moments at which the message is at a single point of failure.
  77. 77. A . . .C I   P ? “We need guaranteed exactly-once delivery – I want to send 1 message, and know it gets delivered to exactly one consumer, once”.
  78. 78. A . . .C I   P ? “We need guaranteed exactly-once delivery – I want to send 1 message, and know it gets delivered to exactly one consumer, once”. Fact: Provably impossible. Depends on definition of guarantee: you can achieve a high probability of no duplicates and no message loss.
  79. 79. T EW ! Thank you Questions?

×