RabbitMQ with python and ruby RuPy 2009

  • 15,352 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • @asksol I amended the presentation to fix the misclassification you pointed out, thanks for your contribution.
    Are you sure you want to
    Your message goes here
  • Hey! Just want to add that ask/carrot is not actually an AMQP client, it’s an AMQP abstraction on top of different messaging protocols. py-amqlib is the real synchronous client that carrot uses in the background. Currently it supports py-amqplib, in-memory, STOMP and using Redis/SQL database as the message backend. Already working on a pika backend, too. But credit where credit is due, if you want to list something under sync lib, list py-amqplib: http://barryp.org/software/py-amqplib/
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
15,352
On Slideshare
0
From Embeds
0
Number of Embeds
6

Actions

Shares
Downloads
345
Comments
2
Likes
37

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. “RabbitMQ python and ruby staring at the looking glass” - an asynchronous tale - • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbitmq_rupy_2009 Paolo Negri @hungryblank
  • 2. Resources • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbitmq_rupy_2009
  • 3. About me Time GNU/Linux - Dbs Perl “systems” PHP Python Ruby
  • 4. AMQP Advanced Message Queuing Protocol • standard for high performance enterprise messaging • totally open • platform agnostic • interoperable
  • 5. AMQP Why is it interesting? “We use asynchronicity heavily in those  cases where performance is an issue. This is generally where we send content  from one peer to another.  We send off  methods as fast as possible without   waiting for confirmations” AMQP Specs [0.8 June 2006]
  • 6. Summary “AMQP as a case of asynchronous protocol implementation in python and ruby”
  • 7. AMQP - Entities Producer msg Consumer msg Broker http://www.flickr.com/photos/17258892@N05/2588347668/
  • 8. AMQP - Entities Producer msg Producer msg Broker Producer msg Consumer msg Consumer msg http://www.flickr.com/photos/17258892@N05/2588347668/
  • 9. AMQP Implementations • Producer/Consumer: libs available for python and ruby • Broker no python or ruby implementation
  • 10. • RabbitMQ is an implementation of AMQP Broker • It’s opensource • Written in Erlang
  • 11. Erlang? • general-purpose concurrent programming language designed by Ericsson • first version written by J. Armstrong in 1986 • distributed • fault tolerant • soft real time • high availability
  • 12. Erlang? Erlang's main strength is support for concurrency “Process communication is done via a shared-nothing asynchronous message passing: every process has a “mailbox”, a queue of messages sent by other processes, that are not yet consumed” (Wikipedia)
  • 13. + Erlang It’s messages all the way down
  • 14. Async Python libs txAMQP • “twisted AMQP” • based on twisted python • https://launchpad.net/txamqp
  • 15. Async Python libs amqpev • based on eventlet • experimental, not ready for production • http://bitbucket.org/edarc/amqpev
  • 16. Async Python libs pika • based on asyncio • young but mantained by member of RabbitMQ core team • http://github.com/tonyg/pika
  • 17. Async Ruby libs amqp • amqp gem • based on EventMachine • mature and widely used • http://github.com/tmm1/amqp/
  • 18. Async? • Twisted Python • Asyncio (python stdlib) • eventlet (python) • EventMachine (ruby)
  • 19. Reactor Pattern • Non blocking IO and lightweight concurrency • eliminate the complexities of high- performance threaded network programming
  • 20. synchronous asynchronous process Time process code code network network operation Free operation use network use network operation operation Callback result result Free Free
  • 21. Not only reactor Some of the libs provide more i.e. eventlet provides also coroutines
  • 22. Python, pick one pika • asyncio is part of python stdlib • easy syntax • developer member of RabbitMQ team • fits on slides :-P
  • 23. Code! Producer (ruby)
  • 24. Consumer (ruby)
  • 25. Behind the scenes Publisher msg Exchange Queue Consumer
  • 26. What and where RabbitMQ (Erlang) Producer (ruby/python) TCP/IP Consumer Exchange Queue (ruby/python)
  • 27. Producer (python)
  • 28. Producer (python) TCP/IP connection
  • 29. Producer (python) channel 1 TCP/IP connection
  • 30. Producer (python) channel 1 TCP/IPchannel 2 connection
  • 31. Producer (python) channel 1 TCP/IPchannel 2 connection channel n
  • 32. Producer (python)
  • 33. Producer (python)
  • 34. Producer (python)
  • 35. Producer (python)
  • 36. Producer (python)
  • 37. Producer (python)
  • 38. Producer (python)
  • 39. Producer (python)
  • 40. Producer (python)
  • 41. Producer (python)
  • 42. Producer (python)
  • 43. Producer (python) Publisher msg Exchange Queue Consumer
  • 44. Producer (python) Publisher msg Exchange Queue Consumer key based routing
  • 45. Producer (python) binding by key Publisher msg Exchange Queue Consumer key based routing
  • 46. Consumer (python)
  • 47. Things go wrong If a consumer has a problem we might lose one or more messages http://www.flickr.com/photos/danzen/2288625136/
  • 48. ACKing consumer.rb
  • 49. ACKing consumer.rb
  • 50. ACKing consumer.rb
  • 51. ACKing consumer.py
  • 52. ACKing consumer.py
  • 53. ACKing consumer.py
  • 54. Achieved: decoupling Publisher msg Exchange Queue Consumer
  • 55. Achieved: easy work load distribution Consumer Publisher msg A Exchange Queue Consumer Consumer
  • 56. Achieved: easy work load distribution Consumer Publisher Exchange Queue Consumer msg A Consumer
  • 57. Achieved: easy work load distribution Consumer Publisher Exchange Queue msg A Consumer Consumer
  • 58. Achieved: easy work load distribution msg A Consumer Publisher Exchange Queue Consumer Consumer
  • 59. prefetch? http://www.flickr.com/photos/balakov/327431271
  • 60. prefetch? tcp ip buffer http://www.flickr.com/photos/balakov/327431271
  • 61. prefetch - ruby
  • 62. prefetch - ruby
  • 63. prefetch - python
  • 64. prefetch - python
  • 65. More routing models multicasting with fanout exchange
  • 66. Fanout producer.rb
  • 67. Fanout producer.rb
  • 68. Fanout consumer.rb
  • 69. Fanout consumer.rb
  • 70. Fanout consumer.rb
  • 71. Fanout consumer.rb
  • 72. Fanout Queue1 Cons1 Producer msg A Exchange Queue2 Cons2 Queue3 Cons3
  • 73. Fanout Queue1 Cons1 Producer Exchange Queue2 Cons2 msg A Queue3 Cons3
  • 74. Fanout msg A Queue1 Cons1 Producer Exchange msg A Queue2 Cons2 msg A Queue3 Cons3
  • 75. Fanout producer.py
  • 76. Fanout producer.py
  • 77. Fanout producer.py
  • 78. Fanout consumer.py
  • 79. Exchanges • Direct - based on binding routing key • Fanout - always to all bound queues • Topic - pattern matching on routing key
  • 80. Clustering RabbitMQ - node A RabbitMQ - node B TCP/IP TCP/IP RabbitMQ - node C TCP/IP
  • 81. Plugins • introduced with v 1.7.0 • Add functionalities to the broker • makes a interesting first erlang project
  • 82. Ecosystem • 65 RabbitMQ / AMQP projects on github • + more on bitbucket, and others
  • 83. Sync? • easier to add to web apps • suitable for non performance critical messaging • useful to build cli tools
  • 84. Sync? • celldee / bunny - Ruby (github) • famoseagle / carrot - Ruby (github) • py-amqplib - Python (http://barryp.org/software/py-amqplib/) • ...
  • 85. Distributed systems • ezmobious / nanite - Ruby (github) • ask / carrot - Python (github) • ask / celery - Python (github) • auser / poolparty - Erlang (github)
  • 86. Real time web • RabbitHub PubSubHubBub • dansimpson / amqp-js (github)
  • 87. Thank you! ?