RabbitMQ with python and ruby RuPy 2009
Upcoming SlideShare
Loading in...5
×
 

RabbitMQ with python and ruby RuPy 2009

on

  • 21,388 views

 

Statistics

Views

Total Views
21,388
Views on SlideShare
20,622
Embed Views
766

Actions

Likes
37
Downloads
345
Comments
2

13 Embeds 766

http://www.megiteam.pl 546
http://www.slideshare.net 187
http://127.0.0.1 14
http://mtwww.megiteam.pl 5
https://www.megiteam.pl 3
http://beta.megiteam.pl 3
http://megiteam1.megivps.pl 2
http://blog.www.megiteam.pl 1
http://www.linkedin.com 1
http://trunk.ly 1
http://localhost 1
http://www.mefeedia.com 1
http://racoon.cecyf.megivps.pl 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • @asksol I amended the presentation to fix the misclassification you pointed out, thanks for your contribution.
    Are you sure you want to
    Your message goes here
    Processing…
  • Hey! Just want to add that ask/carrot is not actually an AMQP client, it’s an AMQP abstraction on top of different messaging protocols. py-amqlib is the real synchronous client that carrot uses in the background. Currently it supports py-amqplib, in-memory, STOMP and using Redis/SQL database as the message backend. Already working on a pika backend, too. But credit where credit is due, if you want to list something under sync lib, list py-amqplib: http://barryp.org/software/py-amqplib/
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

RabbitMQ with python and ruby RuPy 2009 RabbitMQ with python and ruby RuPy 2009 Presentation Transcript

  • “RabbitMQ python and ruby staring at the looking glass” - an asynchronous tale - • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbitmq_rupy_2009 Paolo Negri @hungryblank
  • Resources • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbitmq_rupy_2009
  • About me Time GNU/Linux - Dbs Perl “systems” PHP Python Ruby
  • AMQP Advanced Message Queuing Protocol • standard for high performance enterprise messaging • totally open • platform agnostic • interoperable
  • AMQP Why is it interesting? “We use asynchronicity heavily in those  cases where performance is an issue. This is generally where we send content  from one peer to another.  We send off  methods as fast as possible without   waiting for confirmations” AMQP Specs [0.8 June 2006]
  • Summary “AMQP as a case of asynchronous protocol implementation in python and ruby”
  • AMQP - Entities Producer msg Consumer msg Broker http://www.flickr.com/photos/17258892@N05/2588347668/
  • AMQP - Entities Producer msg Producer msg Broker Producer msg Consumer msg Consumer msg http://www.flickr.com/photos/17258892@N05/2588347668/
  • AMQP Implementations • Producer/Consumer: libs available for python and ruby • Broker no python or ruby implementation
  • • RabbitMQ is an implementation of AMQP Broker • It’s opensource • Written in Erlang
  • Erlang? • general-purpose concurrent programming language designed by Ericsson • first version written by J. Armstrong in 1986 • distributed • fault tolerant • soft real time • high availability
  • Erlang? Erlang's main strength is support for concurrency “Process communication is done via a shared-nothing asynchronous message passing: every process has a “mailbox”, a queue of messages sent by other processes, that are not yet consumed” (Wikipedia)
  • + Erlang It’s messages all the way down
  • Async Python libs txAMQP • “twisted AMQP” • based on twisted python • https://launchpad.net/txamqp
  • Async Python libs amqpev • based on eventlet • experimental, not ready for production • http://bitbucket.org/edarc/amqpev
  • Async Python libs pika • based on asyncio • young but mantained by member of RabbitMQ core team • http://github.com/tonyg/pika
  • Async Ruby libs amqp • amqp gem • based on EventMachine • mature and widely used • http://github.com/tmm1/amqp/
  • Async? • Twisted Python • Asyncio (python stdlib) • eventlet (python) • EventMachine (ruby)
  • Reactor Pattern • Non blocking IO and lightweight concurrency • eliminate the complexities of high- performance threaded network programming
  • synchronous asynchronous process Time process code code network network operation Free operation use network use network operation operation Callback result result Free Free
  • Not only reactor Some of the libs provide more i.e. eventlet provides also coroutines
  • Python, pick one pika • asyncio is part of python stdlib • easy syntax • developer member of RabbitMQ team • fits on slides :-P
  • Code! Producer (ruby)
  • Consumer (ruby)
  • Behind the scenes Publisher msg Exchange Queue Consumer
  • What and where RabbitMQ (Erlang) Producer (ruby/python) TCP/IP Consumer Exchange Queue (ruby/python)
  • Producer (python)
  • Producer (python) TCP/IP connection
  • Producer (python) channel 1 TCP/IP connection
  • Producer (python) channel 1 TCP/IPchannel 2 connection
  • Producer (python) channel 1 TCP/IPchannel 2 connection channel n
  • Producer (python)
  • Producer (python)
  • Producer (python)
  • Producer (python)
  • Producer (python)
  • Producer (python)
  • Producer (python)
  • Producer (python)
  • Producer (python)
  • Producer (python)
  • Producer (python)
  • Producer (python) Publisher msg Exchange Queue Consumer
  • Producer (python) Publisher msg Exchange Queue Consumer key based routing
  • Producer (python) binding by key Publisher msg Exchange Queue Consumer key based routing
  • Consumer (python)
  • Things go wrong If a consumer has a problem we might lose one or more messages http://www.flickr.com/photos/danzen/2288625136/
  • ACKing consumer.rb
  • ACKing consumer.rb
  • ACKing consumer.rb
  • ACKing consumer.py
  • ACKing consumer.py
  • ACKing consumer.py
  • Achieved: decoupling Publisher msg Exchange Queue Consumer
  • Achieved: easy work load distribution Consumer Publisher msg A Exchange Queue Consumer Consumer
  • Achieved: easy work load distribution Consumer Publisher Exchange Queue Consumer msg A Consumer
  • Achieved: easy work load distribution Consumer Publisher Exchange Queue msg A Consumer Consumer
  • Achieved: easy work load distribution msg A Consumer Publisher Exchange Queue Consumer Consumer
  • prefetch? http://www.flickr.com/photos/balakov/327431271
  • prefetch? tcp ip buffer http://www.flickr.com/photos/balakov/327431271
  • prefetch - ruby
  • prefetch - ruby
  • prefetch - python
  • prefetch - python
  • More routing models multicasting with fanout exchange
  • Fanout producer.rb
  • Fanout producer.rb
  • Fanout consumer.rb
  • Fanout consumer.rb
  • Fanout consumer.rb
  • Fanout consumer.rb
  • Fanout Queue1 Cons1 Producer msg A Exchange Queue2 Cons2 Queue3 Cons3
  • Fanout Queue1 Cons1 Producer Exchange Queue2 Cons2 msg A Queue3 Cons3
  • Fanout msg A Queue1 Cons1 Producer Exchange msg A Queue2 Cons2 msg A Queue3 Cons3
  • Fanout producer.py
  • Fanout producer.py
  • Fanout producer.py
  • Fanout consumer.py
  • Exchanges • Direct - based on binding routing key • Fanout - always to all bound queues • Topic - pattern matching on routing key
  • Clustering RabbitMQ - node A RabbitMQ - node B TCP/IP TCP/IP RabbitMQ - node C TCP/IP
  • Plugins • introduced with v 1.7.0 • Add functionalities to the broker • makes a interesting first erlang project
  • Ecosystem • 65 RabbitMQ / AMQP projects on github • + more on bitbucket, and others
  • Sync? • easier to add to web apps • suitable for non performance critical messaging • useful to build cli tools
  • Sync? • celldee / bunny - Ruby (github) • famoseagle / carrot - Ruby (github) • py-amqplib - Python (http://barryp.org/software/py-amqplib/) • ...
  • Distributed systems • ezmobious / nanite - Ruby (github) • ask / carrot - Python (github) • ask / celery - Python (github) • auser / poolparty - Erlang (github)
  • Real time web • RabbitHub PubSubHubBub • dansimpson / amqp-js (github)
  • Thank you! ?