Evented applications with RabbitMQ and CakePHP

Evented Applicationswith RabbitMQ , Python, and CakePHP

What the wibbleﬁsh?All about oﬄoading work, and doing things in‘realtime’Make applications that complete tasks in adistributed/scalable fashion.Doing work outside the request/response cycleof a UI/API.

Work to ofﬂoadNoti cations (SMS, Email, Postal mail).Logging.Import & export.Process image & video.Reporting.Webhooks.

Isn’t cron good enough?Cron tabs are great with one server.Cron doesn’t scale to multiple machines.Cron doesn’t solve the whole problem space.

Problems with CronIsn’t suitable for ‘real-time’ results.Requires a set schedule. You might spin upprocesses that do nothing. Or you might nothave enough capacity.Parallelizing is a pain.

Queues to the rescue!Queues accept and dispatch events.Events can contain almost anything*.Events can be roughly equal to ‘jobs’.A single event can trigger lots of downstreamwork.

Architectural beneﬁtsSeparation of concerns.Collection of mini/small applications.No monolithic applications.Decoupled, and easier to test.Easy to scale. Busy service = more of them.

Queue technologiesRabbitMQZeroMQSQSCeleryGearman

RabbitMQWritten in Erlang.Open source and Free.Really Fast.Socket protocol that is has cheap connections.Data persists after a crash.

Queue basicsProducers create events.Consumers eat messages, and do work.messages == events.

Queue BasicsProducer

Queue BasicsProducer Consumer

Queue Basics ConsumerProducer Consumer

Queue Basics ConsumerProducer Consumer Consumer

ExchangesExchanges abstract queues from the producer.Route messages to queues, that are bound byrouting keys.Allows consumers to listen to speci c routingkeys.Multiple consumers can listen to the sameevents.

ExchangesProducer

Exchanges page_view ConsumerProducer

Exchanges page_view Consumer import ConsumerProducer

Exchanges page_view Consumer import ConsumerProducer Consumer error, page_view

Topic subscriptionConsumers can also listen to topics. Which arekeys separated by dots.e.g page_view.search, page_view.updateConsumer could listen to page_view.*

ProducingCan create messages with many tools.PHP, python, node, ruby, etc.Messages can contain any stringy data (JSON)

Data for messagesOnly string-y things can be stored in messages.JSON encoded data in message.Key to value stored in Redis/Memcache.Keep messages small. RabbitMQ performsbetter with small messages.

ConsumingUse a daemon process.Daemons in PHP can be done, but they kindasuck.Usually use sleep(2) or other workaround.sleep(2) moves you away from ‘real time’.

Aw bugger..

Use a better toolOther tools are much better for makingdaemons. Python (pika, kombu, sparkplug) NodeJS (node-amqp) Ruby (carrot, ruby-amqp)

Managing mixed applicationsMany ways to get it done.Avoid duplicating code.Thin daemons + worker processes.

Thin daemonsPython/node daemon accepts messages.Delegate to PHP worker processes.Keeps PHP’s fork + die work ow.Re-use application code by calling shells.

Thin daemonsPHP

Thin daemons PyPHP

Thin daemons PyPHP Py

Thin daemons PyPHP Py Py

Thin daemons Py PHPPHP Py Py

Thin daemons Py PHPPHP Py PHP Py

Thin daemons Py PHPPHP Py PHP Py PHP

Demo Time

OverviewUse CakeLog to re events.Use CakePHP shell’s for actual work.Thin python daemon for processing messages.

Python + sparkplugPython module for creating queue listeners.Created by a developer at FreshBooks.Ini con guration le de nes behaviour.

What we’ll try:User events logged during application.Log events to the database.

Insert code

Limitations of using shellsDaemon and shell processes have to be on thesame box.This can be limiting as you scale, depends onhow you structure applications.

SolutionsUse HTTP.Have thin daemons dispatch back to the front-end application using HTTP.Use Basic/Digest authentication.Use checksums.

HTTP daemonsPHP

HTTP daemonsPHP Py

HTTP daemons WebPHP Py

HTTP daemons WebPHP Py Web

HTTP daemons WebPHP Py Web Web

Thanks; Questions?

http://lanyrd.com	66
http://twitter.com	5
http://localhost	2
https://twitter.com	1

Evented applications with RabbitMQ and CakePHP

by markstory on Sep 05, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 74

Statistics

Evented applications with RabbitMQ and CakePHP — Presentation Transcript