Plone, rabbit mq and messaging that just works

Plone, RabbitMQand messagingthat just worksAsko Soukka & Jukka Ojaniemi

Speakers backgroundUniversity of Jyväskyläthe largest Plone user in Finland● 83 Plone sites● 2.5 TB data served through Plone● 700 content editors

Demo (serializing writes) RabbitMQ Browser A M1 M1 Browser B Queue Queue Plone M1 M1

Why message queues?

Why message queues?We needed a reliable, effective andscalable solution for: ● running asynchronous tasks ● being a common integration point between services (decoupling).

Why message queues?”None of the existing ZODB-based mechanismreally scales. When you are really in need for areal and working queuing mechanism onewould use some external queuing solution (e.g.something AMQP compatible like RabbitMQ).” – Andreas Jung

Advanced Message QueueProtocol (AMQP)Think of it as an highly advancedpost office which can delivermessage to multiple recipientsbased on address.

AMQP jargon 1/2● Message consists of label and payload.● Producer is a program which sends messages to exchange.● Consumer is a program which mostly waits to receive messages.

AMQP jargon 2/2● Exchange receives the messages from producers and pushes them to queues.● Bindings are how the messages get routed from the exchange to a queue.● Queue is a buffer that stores messages.

Basic messagingWork queues M1 Message broker Consumer M2 M2 M2 M1 Exchange M1 M1 Producer Queue (direct) M2 Consumer

Basic messagingPublish/Subscribe queues M1 Message broker M1 Consumer Queue M1 Exchange M1 Producer (fanout) M1 Queue M1 Consumer

Basic messagingRPC implementation Message broker Exchange M1 M1 RPC queue M1Client/producer M1 R1 Server R1 Reply_to R1 Exchange R1 queue

RabbitMQ● Implements AMQP (Advanced Message Queue Protocol) open standard● Written in Erlang ○ made for messaging ○ easy to cluster ○ fast (enough)● Industry tested and reliable

Sounds complicated?

Example withcollective.zamqpPublish and subscribe announcements M1 Message broker M1 Site B Queue M1 Exchange M1 Site A (fanout) M1 Queue M1 Site C

Produce messages withcollective.zamqpfrom zope.component import getUtilityfrom collective.zamqp.interfaces import IProducerproducer = getUtility( IProducer, name="announcer")producer.register() # for transactionproducer.publish("Hello World!")

Handle messages withcollective.zamqpfrom five import grokfrom collective.zamqp.interfaces import IMessageArrivedEvent@grok.subscribe(IAnnouncement, IMessageArrivedEvent)def handleMessage(message, event): logger.info(message.body) message.ack()

Overview ofcollective.zamqp Message handlers (as event subscribers)Producers Consumers IMessageArrivedEvent stamp messages produce messages Consuming Servers AMQP Broker Connections consume Message broker messages

Connections incollective.zamqpbuildout.cfg:[instance]zope-conf-additional = %import collective.zamqp <amqp-broker-connection> connection_id example ... </amqp-broker-connection>

Producers incollective.zamqpclass Announcer(Producer): grok.name("announcements") connection_id = "example" exchange = "announcements" exchange_type = "fanout" serializer = "text/plain" durable = False

Consuming servers incollective.zamqpbuildout.cfg:zope-conf-additional = %import collective.zamqp .... <amqp-consuming-server> connection_id example site_id Plone </amqp-consuming-server>

Consumers incollective.zamqpclass Announcements(Consumer): grok.name("announcements") connection_id = "example" exchange = "announcements" exchange_type = "fanout" queue = "" # anonymous temp. queue durable = False marker = IAnnouncement

Message handlers incollective.zamqp@grok.subscribe(IAnnouncement, IMessageArrivedEvent)def handleMessage(message, event): logger.info(message.body) message.ack()

Overview ofcollective.zamqp Message handlers (as event subscribers)Producers Consumers IMessageArrivedEvent stamp messages produce messages Consuming Servers AMQP Broker Connections consume Message broker messages

Its so simple! It really is!

Real world exampleMoniviestin video publishing service● Plone 4.1 site● Distributed encoder servers (48 CPUs combined)● First version used XML-RPC based communication - lots of error handling code.● New version uses AMQP-messaging.● Very fast and scalable. Recovers from network outages.

Encoder Mediainfomemcached FS RabbitMQ CherryPy upload service Plone

Thank youc.zamqp and examples are available at:● https://github.com/datakurre/collective.zamqp● https://github.com/datakurre/collective.zamqpdemos● https://github.com/datakurre/chatbehavior● https://github.com/datakurre/pubsubannouncementsother relevant links:● https://www.rabbitmq.com/getstarted.html● https://www.amqp.org/about/examples● http://www.manning.com/videla/ (”RabbitMQ in Action”)

Plone, rabbit mq and messaging that just works

by datakurre on Oct 12, 2012

Accessibility

Upload Details

Usage Rights

Statistics