RabbitMQ with python and ruby RuPy 2009

“RabbitMQ python and ruby staring at the looking glass” - an asynchronous tale - • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbitmq_rupy_2009 Paolo Negri @hungryblank

Resources • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbitmq_rupy_2009

About me Time GNU/Linux - Dbs Perl “systems” PHP Python Ruby

AMQP Advanced Message Queuing Protocol • standard for high performance enterprise messaging • totally open • platform agnostic • interoperable

AMQP Why is it interesting? “We use asynchronicity heavily in those cases where performance is an issue. This is generally where we send content from one peer to another. We send off methods as fast as possible without waiting for conﬁrmations” AMQP Specs [0.8 June 2006]

Summary “AMQP as a case of asynchronous protocol implementation in python and ruby”

AMQP - Entities Producer msg Consumer msg Broker http://www.ﬂickr.com/photos/17258892@N05/2588347668/

AMQP - Entities Producer msg Producer msg Broker Producer msg Consumer msg Consumer msg http://www.ﬂickr.com/photos/17258892@N05/2588347668/

AMQP Implementations • Producer/Consumer: libs available for python and ruby • Broker no python or ruby implementation

• RabbitMQ is an implementation of AMQP Broker • It’s opensource • Written in Erlang

Erlang? • general-purpose concurrent programming language designed by Ericsson • ﬁrst version written by J. Armstrong in 1986 • distributed • fault tolerant • soft real time • high availability

Erlang? Erlang's main strength is support for concurrency “Process communication is done via a shared-nothing asynchronous message passing: every process has a “mailbox”, a queue of messages sent by other processes, that are not yet consumed” (Wikipedia)

+ Erlang It’s messages all the way down

Async Python libs txAMQP • “twisted AMQP” • based on twisted python • https://launchpad.net/txamqp

Async Python libs amqpev • based on eventlet • experimental, not ready for production • http://bitbucket.org/edarc/amqpev

Async Python libs pika • based on asyncio • young but mantained by member of RabbitMQ core team • http://github.com/tonyg/pika

Async Ruby libs amqp • amqp gem • based on EventMachine • mature and widely used • http://github.com/tmm1/amqp/

Async? • Twisted Python • Asyncio (python stdlib) • eventlet (python) • EventMachine (ruby)

Reactor Pattern • Non blocking IO and lightweight concurrency • eliminate the complexities of high- performance threaded network programming

synchronous asynchronous process Time process code code network network operation Free operation use network use network operation operation Callback result result Free Free

Not only reactor Some of the libs provide more i.e. eventlet provides also coroutines

Python, pick one pika • asyncio is part of python stdlib • easy syntax • developer member of RabbitMQ team • ﬁts on slides :-P

Code! Producer (ruby)

Consumer (ruby)

Behind the scenes Publisher msg Exchange Queue Consumer

What and where RabbitMQ (Erlang) Producer (ruby/python) TCP/IP Consumer Exchange Queue (ruby/python)

Producer (python)

Producer (python) TCP/IP connection

Producer (python) channel 1 TCP/IP connection

Producer (python) channel 1 TCP/IPchannel 2 connection

Producer (python) channel 1 TCP/IPchannel 2 connection channel n

Producer (python)

Producer (python) Publisher msg Exchange Queue Consumer

Producer (python) Publisher msg Exchange Queue Consumer key based routing

Producer (python) binding by key Publisher msg Exchange Queue Consumer key based routing

Consumer (python)

Things go wrong If a consumer has a problem we might lose one or more messages http://www.ﬂickr.com/photos/danzen/2288625136/

ACKing consumer.rb

ACKing consumer.py

Achieved: decoupling Publisher msg Exchange Queue Consumer

Achieved: easy work load distribution Consumer Publisher msg A Exchange Queue Consumer Consumer

Achieved: easy work load distribution Consumer Publisher Exchange Queue Consumer msg A Consumer

Achieved: easy work load distribution Consumer Publisher Exchange Queue msg A Consumer Consumer

Achieved: easy work load distribution msg A Consumer Publisher Exchange Queue Consumer Consumer

prefetch? http://www.ﬂickr.com/photos/balakov/327431271

prefetch? tcp ip buffer http://www.ﬂickr.com/photos/balakov/327431271

prefetch - ruby

prefetch - python

More routing models multicasting with fanout exchange

Fanout producer.rb

Fanout consumer.rb

Fanout Queue1 Cons1 Producer msg A Exchange Queue2 Cons2 Queue3 Cons3

Fanout Queue1 Cons1 Producer Exchange Queue2 Cons2 msg A Queue3 Cons3

Fanout msg A Queue1 Cons1 Producer Exchange msg A Queue2 Cons2 msg A Queue3 Cons3

Fanout producer.py

Fanout consumer.py

Exchanges • Direct - based on binding routing key • Fanout - always to all bound queues • Topic - pattern matching on routing key

Clustering RabbitMQ - node A RabbitMQ - node B TCP/IP TCP/IP RabbitMQ - node C TCP/IP

Plugins • introduced with v 1.7.0 • Add functionalities to the broker • makes a interesting ﬁrst erlang project

Ecosystem • 65 RabbitMQ / AMQP projects on github • + more on bitbucket, and others

Sync? • easier to add to web apps • suitable for non performance critical messaging • useful to build cli tools

Sync? • celldee / bunny - Ruby (github) • famoseagle / carrot - Ruby (github) • py-amqplib - Python (http://barryp.org/software/py-amqplib/) • ...

Distributed systems • ezmobious / nanite - Ruby (github) • ask / carrot - Python (github) • ask / celery - Python (github) • auser / poolparty - Erlang (github)

Real time web • RabbitHub PubSubHubBub • dansimpson / amqp-js (github)

Thank you! ?

RabbitMQ with python and ruby RuPy 2009

2 comments

Notes on slide 1

13 Favorites