Plone, rabbit mq and messaging that just works


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Plone, rabbit mq and messaging that just works

  1. 1. Plone, RabbitMQand messagingthat just worksAsko Soukka & Jukka Ojaniemi
  2. 2. Speakers backgroundUniversity of Jyväskyläthe largest Plone user in Finland● 83 Plone sites● 2.5 TB data served through Plone● 700 content editors
  3. 3. Demo (serializing writes) RabbitMQ Browser A M1 M1 Browser B Queue Queue Plone M1 M1
  4. 4. Why message queues?
  5. 5. Why message queues?We needed a reliable, effective andscalable solution for: ● running asynchronous tasks ● being a common integration point between services (decoupling).
  6. 6. Why message queues?”None of the existing ZODB-based mechanismreally scales. When you are really in need for areal and working queuing mechanism onewould use some external queuing solution (e.g.something AMQP compatible like RabbitMQ).” – Andreas Jung
  7. 7. Advanced Message QueueProtocol (AMQP)Think of it as an highly advancedpost office which can delivermessage to multiple recipientsbased on address.
  8. 8. AMQP jargon 1/2● Message consists of label and payload.● Producer is a program which sends messages to exchange.● Consumer is a program which mostly waits to receive messages.
  9. 9. AMQP jargon 2/2● Exchange receives the messages from producers and pushes them to queues.● Bindings are how the messages get routed from the exchange to a queue.● Queue is a buffer that stores messages.
  10. 10. Basic messagingWork queues M1 Message broker Consumer M2 M2 M2 M1 Exchange M1 M1 Producer Queue (direct) M2 Consumer
  11. 11. Basic messagingPublish/Subscribe queues M1 Message broker M1 Consumer Queue M1 Exchange M1 Producer (fanout) M1 Queue M1 Consumer
  12. 12. Basic messagingRPC implementation Message broker Exchange M1 M1 RPC queue M1Client/producer M1 R1 Server R1 Reply_to R1 Exchange R1 queue
  13. 13. RabbitMQ● Implements AMQP (Advanced Message Queue Protocol) open standard● Written in Erlang ○ made for messaging ○ easy to cluster ○ fast (enough)● Industry tested and reliable
  14. 14. Sounds complicated?
  15. 15. Example withcollective.zamqpPublish and subscribe announcements M1 Message broker M1 Site B Queue M1 Exchange M1 Site A (fanout) M1 Queue M1 Site C
  16. 16. Produce messages withcollective.zamqpfrom zope.component import getUtilityfrom collective.zamqp.interfaces import IProducerproducer = getUtility( IProducer, name="announcer")producer.register() # for transactionproducer.publish("Hello World!")
  17. 17. Handle messages withcollective.zamqpfrom five import grokfrom collective.zamqp.interfaces import IMessageArrivedEvent@grok.subscribe(IAnnouncement, IMessageArrivedEvent)def handleMessage(message, event): message.ack()
  18. 18. Overview ofcollective.zamqp Message handlers (as event subscribers)Producers Consumers IMessageArrivedEvent stamp messages produce messages Consuming Servers AMQP Broker Connections consume Message broker messages
  19. 19. Connections incollective.zamqpbuildout.cfg:[instance]zope-conf-additional = %import collective.zamqp <amqp-broker-connection> connection_id example ... </amqp-broker-connection>
  20. 20. Producers incollective.zamqpclass Announcer(Producer):"announcements") connection_id = "example" exchange = "announcements" exchange_type = "fanout" serializer = "text/plain" durable = False
  21. 21. Consuming servers incollective.zamqpbuildout.cfg:zope-conf-additional = %import collective.zamqp .... <amqp-consuming-server> connection_id example site_id Plone </amqp-consuming-server>
  22. 22. Consumers incollective.zamqpclass Announcements(Consumer):"announcements") connection_id = "example" exchange = "announcements" exchange_type = "fanout" queue = "" # anonymous temp. queue durable = False marker = IAnnouncement
  23. 23. Message handlers incollective.zamqp@grok.subscribe(IAnnouncement, IMessageArrivedEvent)def handleMessage(message, event): message.ack()
  24. 24. Overview ofcollective.zamqp Message handlers (as event subscribers)Producers Consumers IMessageArrivedEvent stamp messages produce messages Consuming Servers AMQP Broker Connections consume Message broker messages
  25. 25. Its so simple! It really is!
  26. 26. Real world exampleMoniviestin video publishing service● Plone 4.1 site● Distributed encoder servers (48 CPUs combined)● First version used XML-RPC based communication - lots of error handling code.● New version uses AMQP-messaging.● Very fast and scalable. Recovers from network outages.
  27. 27. Encoder Mediainfomemcached FS RabbitMQ CherryPy upload service Plone
  28. 28. Thank youc.zamqp and examples are available at:●●●● relevant links:●●● (”RabbitMQ in Action”)