0
Evented Applicationswith RabbitMQ , Python, and CakePHP
What the wibblefish?All about offloading work, and doing things in‘realtime’Make applications that complete tasks in adistrib...
Work to offloadNoti cations (SMS, Email, Postal mail).Logging.Import & export.Process image & video.Reporting.Webhooks.
Isn’t cron good enough?Cron tabs are great with one server.Cron doesn’t scale to multiple machines.Cron doesn’t solve the ...
Problems with CronIsn’t suitable for ‘real-time’ results.Requires a set schedule. You might spin upprocesses that do nothi...
Queues to the rescue!Queues accept and dispatch events.Events can contain almost anything*.Events can be roughly equal to ...
Architectural benefitsSeparation of concerns.Collection of mini/small applications.No monolithic applications.Decoupled, an...
Queue technologiesRabbitMQZeroMQSQSCeleryGearman
RabbitMQWritten in Erlang.Open source and Free.Really Fast.Socket protocol that is has cheap connections.Data persists aft...
Queue basicsProducers create events.Consumers eat messages, and do work.messages == events.
Queue BasicsProducer
Queue BasicsProducer
Queue BasicsProducer                  Consumer
Queue Basics                          ConsumerProducer                    Consumer
Queue Basics                          ConsumerProducer                    Consumer                          Consumer
ExchangesExchanges abstract queues from the producer.Route messages to queues, that are bound byrouting keys.Allows consum...
ExchangesProducer
ExchangesProducer
Exchanges               page_view                           ConsumerProducer
Exchanges               page_view                           Consumer                  import    ConsumerProducer
Exchanges                 page_view                              Consumer                     import    ConsumerProducer  ...
Topic subscriptionConsumers can also listen to topics. Which arekeys separated by dots.e.g page_view.search, page_view.upd...
ProducingCan create messages with many tools.PHP, python, node, ruby, etc.Messages can contain any stringy data (JSON)
Data for messagesOnly string-y things can be stored in messages.JSON encoded data in message.Key to value stored in Redis/...
ConsumingUse a daemon process.Daemons in PHP can be done, but they kindasuck.Usually use sleep(2) or other workaround.slee...
Aw bugger..
Use a better toolOther tools are much better for makingdaemons. Python (pika, kombu, sparkplug) NodeJS (node-amqp) Ruby (c...
Managing mixed applicationsMany ways to get it done.Avoid duplicating code.Thin daemons + worker processes.
Thin daemonsPython/node daemon accepts messages.Delegate to PHP worker processes.Keeps PHP’s fork + die work ow.Re-use app...
Thin daemonsPHP
Thin daemonsPHP
Thin daemons               PyPHP
Thin daemons               PyPHP            Py
Thin daemons               PyPHP            Py               Py
Thin daemons               Py    PHPPHP            Py               Py
Thin daemons               Py    PHPPHP            Py    PHP               Py
Thin daemons               Py    PHPPHP            Py    PHP               Py    PHP
Demo Time
OverviewUse CakeLog to re events.Use CakePHP shell’s for actual work.Thin python daemon for processing messages.
Python + sparkplugPython module for creating queue listeners.Created by a developer at FreshBooks.Ini con guration le de n...
What we’ll try:User events logged during application.Log events to the database.
Insert code
Limitations of using shellsDaemon and shell processes have to be on thesame box.This can be limiting as you scale, depends...
SolutionsUse HTTP.Have thin daemons dispatch back to the front-end application using HTTP.Use Basic/Digest authentication....
HTTP daemonsPHP
HTTP daemonsPHP
HTTP daemonsPHP            Py
HTTP daemons                     WebPHP            Py
HTTP daemons                     WebPHP            Py      Web
HTTP daemons                     WebPHP            Py      Web                      Web
Thanks; Questions?
Upcoming SlideShare
Loading in...5
×

Evented applications with RabbitMQ and CakePHP

7,818

Published on

Talk given at CakeFest 2011 about using RabbitMQ, CakePHP and pyhon to create soft realtime applications.

Published in: Technology
1 Comment
8 Likes
Statistics
Notes
No Downloads
Views
Total Views
7,818
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
80
Comments
1
Likes
8
Embeds 0
No embeds

No notes for slide

Transcript of "Evented applications with RabbitMQ and CakePHP"

  1. 1. Evented Applicationswith RabbitMQ , Python, and CakePHP
  2. 2. What the wibblefish?All about offloading work, and doing things in‘realtime’Make applications that complete tasks in adistributed/scalable fashion.Doing work outside the request/response cycleof a UI/API.
  3. 3. Work to offloadNoti cations (SMS, Email, Postal mail).Logging.Import & export.Process image & video.Reporting.Webhooks.
  4. 4. Isn’t cron good enough?Cron tabs are great with one server.Cron doesn’t scale to multiple machines.Cron doesn’t solve the whole problem space.
  5. 5. Problems with CronIsn’t suitable for ‘real-time’ results.Requires a set schedule. You might spin upprocesses that do nothing. Or you might nothave enough capacity.Parallelizing is a pain.
  6. 6. Queues to the rescue!Queues accept and dispatch events.Events can contain almost anything*.Events can be roughly equal to ‘jobs’.A single event can trigger lots of downstreamwork.
  7. 7. Architectural benefitsSeparation of concerns.Collection of mini/small applications.No monolithic applications.Decoupled, and easier to test.Easy to scale. Busy service = more of them.
  8. 8. Queue technologiesRabbitMQZeroMQSQSCeleryGearman
  9. 9. RabbitMQWritten in Erlang.Open source and Free.Really Fast.Socket protocol that is has cheap connections.Data persists after a crash.
  10. 10. Queue basicsProducers create events.Consumers eat messages, and do work.messages == events.
  11. 11. Queue BasicsProducer
  12. 12. Queue BasicsProducer
  13. 13. Queue BasicsProducer Consumer
  14. 14. Queue Basics ConsumerProducer Consumer
  15. 15. Queue Basics ConsumerProducer Consumer Consumer
  16. 16. ExchangesExchanges abstract queues from the producer.Route messages to queues, that are bound byrouting keys.Allows consumers to listen to speci c routingkeys.Multiple consumers can listen to the sameevents.
  17. 17. ExchangesProducer
  18. 18. ExchangesProducer
  19. 19. Exchanges page_view ConsumerProducer
  20. 20. Exchanges page_view Consumer import ConsumerProducer
  21. 21. Exchanges page_view Consumer import ConsumerProducer Consumer error, page_view
  22. 22. Topic subscriptionConsumers can also listen to topics. Which arekeys separated by dots.e.g page_view.search, page_view.updateConsumer could listen to page_view.*
  23. 23. ProducingCan create messages with many tools.PHP, python, node, ruby, etc.Messages can contain any stringy data (JSON)
  24. 24. Data for messagesOnly string-y things can be stored in messages.JSON encoded data in message.Key to value stored in Redis/Memcache.Keep messages small. RabbitMQ performsbetter with small messages.
  25. 25. ConsumingUse a daemon process.Daemons in PHP can be done, but they kindasuck.Usually use sleep(2) or other workaround.sleep(2) moves you away from ‘real time’.
  26. 26. Aw bugger..
  27. 27. Use a better toolOther tools are much better for makingdaemons. Python (pika, kombu, sparkplug) NodeJS (node-amqp) Ruby (carrot, ruby-amqp)
  28. 28. Managing mixed applicationsMany ways to get it done.Avoid duplicating code.Thin daemons + worker processes.
  29. 29. Thin daemonsPython/node daemon accepts messages.Delegate to PHP worker processes.Keeps PHP’s fork + die work ow.Re-use application code by calling shells.
  30. 30. Thin daemonsPHP
  31. 31. Thin daemonsPHP
  32. 32. Thin daemons PyPHP
  33. 33. Thin daemons PyPHP Py
  34. 34. Thin daemons PyPHP Py Py
  35. 35. Thin daemons Py PHPPHP Py Py
  36. 36. Thin daemons Py PHPPHP Py PHP Py
  37. 37. Thin daemons Py PHPPHP Py PHP Py PHP
  38. 38. Demo Time
  39. 39. OverviewUse CakeLog to re events.Use CakePHP shell’s for actual work.Thin python daemon for processing messages.
  40. 40. Python + sparkplugPython module for creating queue listeners.Created by a developer at FreshBooks.Ini con guration le de nes behaviour.
  41. 41. What we’ll try:User events logged during application.Log events to the database.
  42. 42. Insert code
  43. 43. Limitations of using shellsDaemon and shell processes have to be on thesame box.This can be limiting as you scale, depends onhow you structure applications.
  44. 44. SolutionsUse HTTP.Have thin daemons dispatch back to the front-end application using HTTP.Use Basic/Digest authentication.Use checksums.
  45. 45. HTTP daemonsPHP
  46. 46. HTTP daemonsPHP
  47. 47. HTTP daemonsPHP Py
  48. 48. HTTP daemons WebPHP Py
  49. 49. HTTP daemons WebPHP Py Web
  50. 50. HTTP daemons WebPHP Py Web Web
  51. 51. Thanks; Questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×