This document discusses using RabbitMQ and message queues with Plone. It provides an overview of message queues and how they can provide reliable, scalable solutions for running asynchronous tasks and integrating services. It introduces AMQP and describes how collective.zamqp can be used to easily produce, consume, and handle messages with Plone. Examples are given of using collective.zamqp to implement publish/subscribe and request/response patterns between Plone sites.

1. Plone, RabbitMQ
and messaging
that just works
Asko Soukka & Jukka Ojaniemi

2. Speakers' background
University of Jyväskylä
the largest Plone user in Finland
● 83 Plone sites
● 2.5 TB data served through Plone
● 700 content editors

3. Demo (serializing writes)
RabbitMQ
Browser A M1
M1
Browser B Queue Queue Plone
M1 M1

4. Why message queues?

5. Why message queues?
We needed a reliable, effective and
scalable solution for:
● running asynchronous tasks
● being a common integration point
between services (decoupling).

6. Why message queues?
”None of the existing ZODB-based mechanism
really scales. When you are really in need for a
real and working queuing mechanism one
would use some external queuing solution (e.g.
something AMQP compatible like RabbitMQ).”
– Andreas Jung

7. Advanced Message Queue
Protocol (AMQP)
Think of it as an highly advanced
post office which can deliver
message to multiple recipients
based on address.

8. AMQP jargon 1/2
● Message consists of label and payload.
● Producer is a program which sends
messages to exchange.
● Consumer is a program which mostly waits
to receive messages.

9. AMQP jargon 2/2
● Exchange receives the messages from
producers and pushes them to queues.
● Bindings are how the messages get routed
from the exchange to a queue.
● Queue is a buffer that stores messages.

10. Basic messaging
Work queues
M1
Message broker Consumer
M2 M2 M2
M1 Exchange M1 M1
Producer Queue
(direct)
M2
Consumer

11. Basic messaging
Publish/Subscribe queues
M1
Message broker
M1 Consumer
Queue
M1 Exchange M1
Producer
(fanout) M1
Queue
M1
Consumer

12. Basic messaging
RPC implementation
Message broker
Exchange M1 M1
RPC queue M1
Client/producer M1
R1 Server
R1 Reply_to R1 Exchange R1
queue

13. RabbitMQ
● Implements AMQP (Advanced Message
Queue Protocol) open standard
● Written in Erlang
○ made for messaging
○ easy to cluster
○ fast (enough)
● Industry tested and reliable

14. Sounds complicated?

15. Example with
collective.zamqp
Publish and subscribe announcements
M1
Message broker
M1 Site B
Queue
M1 Exchange M1
Site A
(fanout) M1
Queue
M1
Site C

16. Produce messages with
collective.zamqp
from zope.component import getUtility
from collective.zamqp.interfaces import IProducer
producer = getUtility(
IProducer, name="announcer")
producer.register() # for transaction
producer.publish("Hello World!")

17. Handle messages with
collective.zamqp
from five import grok
from collective.zamqp.interfaces import IMessageArrivedEvent
@grok.subscribe(IAnnouncement,
IMessageArrivedEvent)
def handleMessage(message, event):
logger.info(message.body)
message.ack()

18. Overview of
collective.zamqp
Message handlers
(as event subscribers)
Producers Consumers
IMessageArrivedEvent
stamp
messages
produce messages
Consuming Servers
AMQP Broker Connections
consume
Message broker messages

19. Connections in
collective.zamqp
buildout.cfg:
[instance]
zope-conf-additional =
%import collective.zamqp
<amqp-broker-connection>
connection_id example
...
</amqp-broker-connection>

20. Producers in
collective.zamqp
class Announcer(Producer):
grok.name("announcements")
connection_id = "example"
exchange = "announcements"
exchange_type = "fanout"
serializer = "text/plain"
durable = False

21. Consuming servers in
collective.zamqp
buildout.cfg:
zope-conf-additional =
%import collective.zamqp
....
<amqp-consuming-server>
connection_id example
site_id Plone
</amqp-consuming-server>

22. Consumers in
collective.zamqp
class Announcements(Consumer):
grok.name("announcements")
connection_id = "example"
exchange = "announcements"
exchange_type = "fanout"
queue = "" # anonymous temp. queue
durable = False
marker = IAnnouncement

23. Message handlers in
collective.zamqp
@grok.subscribe(IAnnouncement,
IMessageArrivedEvent)
def handleMessage(message, event):
logger.info(message.body)
message.ack()

24. Overview of
collective.zamqp
Message handlers
(as event subscribers)
Producers Consumers
IMessageArrivedEvent
stamp
messages
produce messages
Consuming Servers
AMQP Broker Connections
consume
Message broker messages

25. It's so simple!
It really is!

26. Real world example
Moniviestin video publishing service
● Plone 4.1 site
● Distributed encoder servers (48 CPUs
combined)
● First version used XML-RPC based
communication - lots of error handling code.
● New version uses AMQP-messaging.
● Very fast and scalable. Recovers from
network outages.

27. Encoder
Mediainfo
memcached
FS
RabbitMQ
CherryPy
upload
service
Plone

28. Thank you
c.zamqp and examples are available at:
● https://github.com/datakurre/collective.zamqp
● https://github.com/datakurre/collective.zamqpdemos
● https://github.com/datakurre/chatbehavior
● https://github.com/datakurre/pubsubannouncements
other relevant links:
● https://www.rabbitmq.com/getstarted.html
● https://www.amqp.org/about/examples
● http://www.manning.com/videla/ (”RabbitMQ in Action”)