Evented applications with RabbitMQ and CakePHP
Upcoming SlideShare
Loading in...5
×
 

Evented applications with RabbitMQ and CakePHP

on

  • 7,686 views

Talk given at CakeFest 2011 about using RabbitMQ, CakePHP and pyhon to create soft realtime applications.

Talk given at CakeFest 2011 about using RabbitMQ, CakePHP and pyhon to create soft realtime applications.

Statistics

Views

Total Views
7,686
Views on SlideShare
7,481
Embed Views
205

Actions

Likes
7
Downloads
70
Comments
1

5 Embeds 205

http://lanyrd.com 161
https://twitter.com 35
http://twitter.com 5
http://localhost 3
https://si0.twimg.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Evented applications with RabbitMQ and CakePHP Evented applications with RabbitMQ and CakePHP Presentation Transcript

  • Evented Applicationswith RabbitMQ , Python, and CakePHP
  • What the wibblefish?All about offloading work, and doing things in‘realtime’Make applications that complete tasks in adistributed/scalable fashion.Doing work outside the request/response cycleof a UI/API.
  • Work to offloadNoti cations (SMS, Email, Postal mail).Logging.Import & export.Process image & video.Reporting.Webhooks.
  • Isn’t cron good enough?Cron tabs are great with one server.Cron doesn’t scale to multiple machines.Cron doesn’t solve the whole problem space.
  • Problems with CronIsn’t suitable for ‘real-time’ results.Requires a set schedule. You might spin upprocesses that do nothing. Or you might nothave enough capacity.Parallelizing is a pain.
  • Queues to the rescue!Queues accept and dispatch events.Events can contain almost anything*.Events can be roughly equal to ‘jobs’.A single event can trigger lots of downstreamwork.
  • Architectural benefitsSeparation of concerns.Collection of mini/small applications.No monolithic applications.Decoupled, and easier to test.Easy to scale. Busy service = more of them.
  • Queue technologiesRabbitMQZeroMQSQSCeleryGearman
  • RabbitMQWritten in Erlang.Open source and Free.Really Fast.Socket protocol that is has cheap connections.Data persists after a crash.
  • Queue basicsProducers create events.Consumers eat messages, and do work.messages == events.
  • Queue BasicsProducer
  • Queue BasicsProducer
  • Queue BasicsProducer Consumer
  • Queue Basics ConsumerProducer Consumer
  • Queue Basics ConsumerProducer Consumer Consumer
  • ExchangesExchanges abstract queues from the producer.Route messages to queues, that are bound byrouting keys.Allows consumers to listen to speci c routingkeys.Multiple consumers can listen to the sameevents.
  • ExchangesProducer
  • ExchangesProducer
  • Exchanges page_view ConsumerProducer
  • Exchanges page_view Consumer import ConsumerProducer
  • Exchanges page_view Consumer import ConsumerProducer Consumer error, page_view
  • Topic subscriptionConsumers can also listen to topics. Which arekeys separated by dots.e.g page_view.search, page_view.updateConsumer could listen to page_view.*
  • ProducingCan create messages with many tools.PHP, python, node, ruby, etc.Messages can contain any stringy data (JSON)
  • Data for messagesOnly string-y things can be stored in messages.JSON encoded data in message.Key to value stored in Redis/Memcache.Keep messages small. RabbitMQ performsbetter with small messages.
  • ConsumingUse a daemon process.Daemons in PHP can be done, but they kindasuck.Usually use sleep(2) or other workaround.sleep(2) moves you away from ‘real time’.
  • Aw bugger..
  • Use a better toolOther tools are much better for makingdaemons. Python (pika, kombu, sparkplug) NodeJS (node-amqp) Ruby (carrot, ruby-amqp)
  • Managing mixed applicationsMany ways to get it done.Avoid duplicating code.Thin daemons + worker processes.
  • Thin daemonsPython/node daemon accepts messages.Delegate to PHP worker processes.Keeps PHP’s fork + die work ow.Re-use application code by calling shells.
  • Thin daemonsPHP
  • Thin daemonsPHP
  • Thin daemons PyPHP
  • Thin daemons PyPHP Py
  • Thin daemons PyPHP Py Py
  • Thin daemons Py PHPPHP Py Py
  • Thin daemons Py PHPPHP Py PHP Py
  • Thin daemons Py PHPPHP Py PHP Py PHP
  • Demo Time
  • OverviewUse CakeLog to re events.Use CakePHP shell’s for actual work.Thin python daemon for processing messages.
  • Python + sparkplugPython module for creating queue listeners.Created by a developer at FreshBooks.Ini con guration le de nes behaviour.
  • What we’ll try:User events logged during application.Log events to the database.
  • Insert code
  • Limitations of using shellsDaemon and shell processes have to be on thesame box.This can be limiting as you scale, depends onhow you structure applications.
  • SolutionsUse HTTP.Have thin daemons dispatch back to the front-end application using HTTP.Use Basic/Digest authentication.Use checksums.
  • HTTP daemonsPHP
  • HTTP daemonsPHP
  • HTTP daemonsPHP Py
  • HTTP daemons WebPHP Py
  • HTTP daemons WebPHP Py Web
  • HTTP daemons WebPHP Py Web Web
  • Thanks; Questions?