1. Plone, RabbitMQand messagingthat just worksAsko Soukka & Jukka Ojaniemi
2. Speakers backgroundUniversity of Jyväskyläthe largest Plone user in Finland● 83 Plone sites● 2.5 TB data served through Plone● 700 content editors
3. Demo (serializing writes) RabbitMQ Browser A M1 M1 Browser B Queue Queue Plone M1 M1
4. Why message queues?
5. Why message queues?We needed a reliable, effective andscalable solution for: ● running asynchronous tasks ● being a common integration point between services (decoupling).
6. Why message queues?”None of the existing ZODB-based mechanismreally scales. When you are really in need for areal and working queuing mechanism onewould use some external queuing solution (e.g.something AMQP compatible like RabbitMQ).” – Andreas Jung
7. Advanced Message QueueProtocol (AMQP)Think of it as an highly advancedpost office which can delivermessage to multiple recipientsbased on address.
8. AMQP jargon 1/2● Message consists of label and payload.● Producer is a program which sends messages to exchange.● Consumer is a program which mostly waits to receive messages.
9. AMQP jargon 2/2● Exchange receives the messages from producers and pushes them to queues.● Bindings are how the messages get routed from the exchange to a queue.● Queue is a buffer that stores messages.
13. RabbitMQ● Implements AMQP (Advanced Message Queue Protocol) open standard● Written in Erlang ○ made for messaging ○ easy to cluster ○ fast (enough)● Industry tested and reliable
14. Sounds complicated?
15. Example withcollective.zamqpPublish and subscribe announcements M1 Message broker M1 Site B Queue M1 Exchange M1 Site A (fanout) M1 Queue M1 Site C
16. Produce messages withcollective.zamqpfrom zope.component import getUtilityfrom collective.zamqp.interfaces import IProducerproducer = getUtility( IProducer, name="announcer")producer.register() # for transactionproducer.publish("Hello World!")
26. Real world exampleMoniviestin video publishing service● Plone 4.1 site● Distributed encoder servers (48 CPUs combined)● First version used XML-RPC based communication - lots of error handling code.● New version uses AMQP-messaging.● Very fast and scalable. Recovers from network outages.
27. Encoder Mediainfomemcached FS RabbitMQ CherryPy upload service Plone
28. Thank youc.zamqp and examples are available at:● https://github.com/datakurre/collective.zamqp● https://github.com/datakurre/collective.zamqpdemos● https://github.com/datakurre/chatbehavior● https://github.com/datakurre/pubsubannouncementsother relevant links:● https://www.rabbitmq.com/getstarted.html● https://www.amqp.org/about/examples● http://www.manning.com/videla/ (”RabbitMQ in Action”)