“RabbitMQ
     python and ruby staring
       at the looking glass”
               - an asynchronous tale -

 •   http://w...
Resources

• http://www.slideshare.net/hungryblank
• http://github.com/hungryblank/rabbitmq_rupy_2009
About me
Time
       GNU/Linux - Dbs
                          Perl
         “systems”
                                   ...
AMQP
Advanced Message Queuing Protocol

   • standard for high performance
     enterprise messaging
   • totally open
   ...
AMQP
     Why is it interesting?
“We use asynchronicity heavily in those 
cases where performance is an issue.
This is gen...
Summary

“AMQP as a case of asynchronous
protocol implementation in python
and ruby”
AMQP - Entities

Producer   msg



Consumer         msg
                       Broker


                       http://www....
AMQP - Entities
Producer   msg

Producer   msg



                       Broker
Producer   msg


Consumer         msg

Con...
AMQP
     Implementations

• Producer/Consumer: libs available for
  python and ruby
• Broker no python or ruby
  implemen...
• RabbitMQ is an implementation of
  AMQP Broker
• It’s opensource
• Written in Erlang
Erlang?
• general-purpose concurrent programming
  language designed by Ericsson
• first version written by J. Armstrong in...
Erlang?
Erlang's main strength is support for
concurrency
“Process communication is done via a
shared-nothing asynchronous...
+ Erlang


It’s messages all the way down
Async Python libs
            txAMQP

• “twisted AMQP”
• based on twisted python
• https://launchpad.net/txamqp
Async Python libs
               amqpev

• based on eventlet
• experimental, not ready for
  production
• http://bitbucket...
Async Python libs
                pika

• based on asyncio
• young but mantained by member of
  RabbitMQ core team
• http:...
Async Ruby libs
              amqp

• amqp gem
• based on EventMachine
• mature and widely used
• http://github.com/tmm1/a...
Async?

• Twisted Python
• Asyncio (python stdlib)
• eventlet (python)
• EventMachine (ruby)
Reactor Pattern

• Non blocking IO and lightweight
  concurrency
• eliminate the complexities of high-
  performance threa...
synchronous          asynchronous
 process      Time   process

   code                 code



 network                  ...
Not only reactor


Some of the libs provide more i.e. eventlet
provides also coroutines
Python, pick one
              pika
• asyncio is part of python stdlib
• easy syntax
• developer member of RabbitMQ team
•...
Code! Producer (ruby)
Consumer (ruby)
Behind the scenes

Publisher

  msg       Exchange   Queue   Consumer
What and where

     RabbitMQ
      (Erlang)              Producer
                          (ruby/python)

              ...
Producer (python)
Producer (python)




 TCP/IP connection
Producer (python)



       channel 1

 TCP/IP connection
Producer (python)



       channel 1

 TCP/IPchannel 2
         connection
Producer (python)



       channel 1

 TCP/IPchannel 2
         connection
       channel n
Producer (python)
Producer (python)
Producer (python)
Producer (python)
Producer (python)
Producer (python)
Producer (python)
Producer (python)
Producer (python)
Producer (python)
Producer (python)
Producer (python)
Publisher

  msg       Exchange   Queue   Consumer
Producer (python)
Publisher

  msg       Exchange   Queue     Consumer



             key based routing
Producer (python)
                       binding by key
Publisher

  msg       Exchange   Queue            Consumer



   ...
Consumer (python)
Things go wrong




If a consumer has a problem we might lose
          one or more messages
                   http://www...
ACKing consumer.rb
ACKing consumer.rb
ACKing consumer.rb
ACKing consumer.py
ACKing consumer.py
ACKing consumer.py
Achieved: decoupling

Publisher

  msg       Exchange   Queue   Consumer
Achieved: easy work
   load distribution
                               Consumer
Publisher

 msg A      Exchange   Queue  ...
Achieved: easy work
   load distribution
                               Consumer
Publisher

            Exchange   Queue  ...
Achieved: easy work
   load distribution
                                     Consumer
Publisher

            Exchange   Q...
Achieved: easy work
   load distribution
                               msg A Consumer
Publisher

            Exchange   Q...
prefetch?




http://www.flickr.com/photos/balakov/327431271
prefetch?


                      tcp ip
                      buffer

http://www.flickr.com/photos/balakov/327431271
prefetch - ruby
prefetch - ruby
prefetch - python
prefetch - python
More routing models

     multicasting
with fanout exchange
Fanout producer.rb
Fanout producer.rb
Fanout consumer.rb
Fanout consumer.rb
Fanout consumer.rb
Fanout consumer.rb
Fanout

                      Queue1   Cons1
Producer

msg A      Exchange   Queue2   Cons2


                      Queue3...
Fanout

                      Queue1   Cons1
Producer
           Exchange   Queue2   Cons2
            msg A

            ...
Fanout

                      msg A Queue1   Cons1
Producer
           Exchange   msg A Queue2   Cons2


                 ...
Fanout producer.py
Fanout producer.py
Fanout producer.py
Fanout consumer.py
Exchanges

• Direct - based on binding routing key
• Fanout - always to all bound queues
• Topic - pattern matching on rou...
Clustering
         RabbitMQ - node A   RabbitMQ - node B

TCP/IP                                           TCP/IP



    ...
Plugins

• introduced with v 1.7.0
• Add functionalities to the broker
• makes a interesting first erlang project
Ecosystem


• 65 RabbitMQ / AMQP projects on github
• + more on bitbucket, and others
Sync?

• easier to add to web apps
• suitable for non performance critical
  messaging
• useful to build cli tools
Sync?

• celldee / bunny - Ruby (github)
• famoseagle / carrot - Ruby (github)
• py-amqplib - Python (http://barryp.org/so...
Distributed systems

• ezmobious / nanite - Ruby (github)
• ask / carrot - Python (github)
• ask / celery - Python (github...
Real time web


• RabbitHub PubSubHubBub
• dansimpson / amqp-js (github)
Thank you!



    ?
Upcoming SlideShare
Loading in...5
×

RabbitMQ with python and ruby RuPy 2009

15,935

Published on

Published in: Technology
2 Comments
37 Likes
Statistics
Notes
  • @asksol I amended the presentation to fix the misclassification you pointed out, thanks for your contribution.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hey! Just want to add that ask/carrot is not actually an AMQP client, it’s an AMQP abstraction on top of different messaging protocols. py-amqlib is the real synchronous client that carrot uses in the background. Currently it supports py-amqplib, in-memory, STOMP and using Redis/SQL database as the message backend. Already working on a pika backend, too. But credit where credit is due, if you want to list something under sync lib, list py-amqplib: http://barryp.org/software/py-amqplib/
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
15,935
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
353
Comments
2
Likes
37
Embeds 0
No embeds

No notes for slide

Transcript of "RabbitMQ with python and ruby RuPy 2009"

  1. 1. “RabbitMQ python and ruby staring at the looking glass” - an asynchronous tale - • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbitmq_rupy_2009 Paolo Negri @hungryblank
  2. 2. Resources • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbitmq_rupy_2009
  3. 3. About me Time GNU/Linux - Dbs Perl “systems” PHP Python Ruby
  4. 4. AMQP Advanced Message Queuing Protocol • standard for high performance enterprise messaging • totally open • platform agnostic • interoperable
  5. 5. AMQP Why is it interesting? “We use asynchronicity heavily in those  cases where performance is an issue. This is generally where we send content  from one peer to another.  We send off  methods as fast as possible without   waiting for confirmations” AMQP Specs [0.8 June 2006]
  6. 6. Summary “AMQP as a case of asynchronous protocol implementation in python and ruby”
  7. 7. AMQP - Entities Producer msg Consumer msg Broker http://www.flickr.com/photos/17258892@N05/2588347668/
  8. 8. AMQP - Entities Producer msg Producer msg Broker Producer msg Consumer msg Consumer msg http://www.flickr.com/photos/17258892@N05/2588347668/
  9. 9. AMQP Implementations • Producer/Consumer: libs available for python and ruby • Broker no python or ruby implementation
  10. 10. • RabbitMQ is an implementation of AMQP Broker • It’s opensource • Written in Erlang
  11. 11. Erlang? • general-purpose concurrent programming language designed by Ericsson • first version written by J. Armstrong in 1986 • distributed • fault tolerant • soft real time • high availability
  12. 12. Erlang? Erlang's main strength is support for concurrency “Process communication is done via a shared-nothing asynchronous message passing: every process has a “mailbox”, a queue of messages sent by other processes, that are not yet consumed” (Wikipedia)
  13. 13. + Erlang It’s messages all the way down
  14. 14. Async Python libs txAMQP • “twisted AMQP” • based on twisted python • https://launchpad.net/txamqp
  15. 15. Async Python libs amqpev • based on eventlet • experimental, not ready for production • http://bitbucket.org/edarc/amqpev
  16. 16. Async Python libs pika • based on asyncio • young but mantained by member of RabbitMQ core team • http://github.com/tonyg/pika
  17. 17. Async Ruby libs amqp • amqp gem • based on EventMachine • mature and widely used • http://github.com/tmm1/amqp/
  18. 18. Async? • Twisted Python • Asyncio (python stdlib) • eventlet (python) • EventMachine (ruby)
  19. 19. Reactor Pattern • Non blocking IO and lightweight concurrency • eliminate the complexities of high- performance threaded network programming
  20. 20. synchronous asynchronous process Time process code code network network operation Free operation use network use network operation operation Callback result result Free Free
  21. 21. Not only reactor Some of the libs provide more i.e. eventlet provides also coroutines
  22. 22. Python, pick one pika • asyncio is part of python stdlib • easy syntax • developer member of RabbitMQ team • fits on slides :-P
  23. 23. Code! Producer (ruby)
  24. 24. Consumer (ruby)
  25. 25. Behind the scenes Publisher msg Exchange Queue Consumer
  26. 26. What and where RabbitMQ (Erlang) Producer (ruby/python) TCP/IP Consumer Exchange Queue (ruby/python)
  27. 27. Producer (python)
  28. 28. Producer (python) TCP/IP connection
  29. 29. Producer (python) channel 1 TCP/IP connection
  30. 30. Producer (python) channel 1 TCP/IPchannel 2 connection
  31. 31. Producer (python) channel 1 TCP/IPchannel 2 connection channel n
  32. 32. Producer (python)
  33. 33. Producer (python)
  34. 34. Producer (python)
  35. 35. Producer (python)
  36. 36. Producer (python)
  37. 37. Producer (python)
  38. 38. Producer (python)
  39. 39. Producer (python)
  40. 40. Producer (python)
  41. 41. Producer (python)
  42. 42. Producer (python)
  43. 43. Producer (python) Publisher msg Exchange Queue Consumer
  44. 44. Producer (python) Publisher msg Exchange Queue Consumer key based routing
  45. 45. Producer (python) binding by key Publisher msg Exchange Queue Consumer key based routing
  46. 46. Consumer (python)
  47. 47. Things go wrong If a consumer has a problem we might lose one or more messages http://www.flickr.com/photos/danzen/2288625136/
  48. 48. ACKing consumer.rb
  49. 49. ACKing consumer.rb
  50. 50. ACKing consumer.rb
  51. 51. ACKing consumer.py
  52. 52. ACKing consumer.py
  53. 53. ACKing consumer.py
  54. 54. Achieved: decoupling Publisher msg Exchange Queue Consumer
  55. 55. Achieved: easy work load distribution Consumer Publisher msg A Exchange Queue Consumer Consumer
  56. 56. Achieved: easy work load distribution Consumer Publisher Exchange Queue Consumer msg A Consumer
  57. 57. Achieved: easy work load distribution Consumer Publisher Exchange Queue msg A Consumer Consumer
  58. 58. Achieved: easy work load distribution msg A Consumer Publisher Exchange Queue Consumer Consumer
  59. 59. prefetch? http://www.flickr.com/photos/balakov/327431271
  60. 60. prefetch? tcp ip buffer http://www.flickr.com/photos/balakov/327431271
  61. 61. prefetch - ruby
  62. 62. prefetch - ruby
  63. 63. prefetch - python
  64. 64. prefetch - python
  65. 65. More routing models multicasting with fanout exchange
  66. 66. Fanout producer.rb
  67. 67. Fanout producer.rb
  68. 68. Fanout consumer.rb
  69. 69. Fanout consumer.rb
  70. 70. Fanout consumer.rb
  71. 71. Fanout consumer.rb
  72. 72. Fanout Queue1 Cons1 Producer msg A Exchange Queue2 Cons2 Queue3 Cons3
  73. 73. Fanout Queue1 Cons1 Producer Exchange Queue2 Cons2 msg A Queue3 Cons3
  74. 74. Fanout msg A Queue1 Cons1 Producer Exchange msg A Queue2 Cons2 msg A Queue3 Cons3
  75. 75. Fanout producer.py
  76. 76. Fanout producer.py
  77. 77. Fanout producer.py
  78. 78. Fanout consumer.py
  79. 79. Exchanges • Direct - based on binding routing key • Fanout - always to all bound queues • Topic - pattern matching on routing key
  80. 80. Clustering RabbitMQ - node A RabbitMQ - node B TCP/IP TCP/IP RabbitMQ - node C TCP/IP
  81. 81. Plugins • introduced with v 1.7.0 • Add functionalities to the broker • makes a interesting first erlang project
  82. 82. Ecosystem • 65 RabbitMQ / AMQP projects on github • + more on bitbucket, and others
  83. 83. Sync? • easier to add to web apps • suitable for non performance critical messaging • useful to build cli tools
  84. 84. Sync? • celldee / bunny - Ruby (github) • famoseagle / carrot - Ruby (github) • py-amqplib - Python (http://barryp.org/software/py-amqplib/) • ...
  85. 85. Distributed systems • ezmobious / nanite - Ruby (github) • ask / carrot - Python (github) • ask / celery - Python (github) • auser / poolparty - Erlang (github)
  86. 86. Real time web • RabbitHub PubSubHubBub • dansimpson / amqp-js (github)
  87. 87. Thank you! ?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×