Talk given at CakeFest 2011 about using RabbitMQ, CakePHP and pyhon to create soft realtime applications.

1. Evented Applications
with RabbitMQ , Python, and CakePHP

2. What the wibblefish?
All about offloading work, and doing things in
‘realtime’
Make applications that complete tasks in a
distributed/scalable fashion.
Doing work outside the request/response cycle
of a UI/API.

3. Work to offload
Noti cations (SMS, Email, Postal mail).
Logging.
Import & export.
Process image & video.
Reporting.
Webhooks.

4. Isn’t cron good enough?
Cron tabs are great with one server.
Cron doesn’t scale to multiple machines.
Cron doesn’t solve the whole problem space.

5. Problems with Cron
Isn’t suitable for ‘real-time’ results.
Requires a set schedule. You might spin up
processes that do nothing. Or you might not
have enough capacity.
Parallelizing is a pain.

6. Queues to the rescue!
Queues accept and dispatch events.
Events can contain almost anything*.
Events can be roughly equal to ‘jobs’.
A single event can trigger lots of downstream
work.

7. Architectural benefits
Separation of concerns.
Collection of mini/small applications.
No monolithic applications.
Decoupled, and easier to test.
Easy to scale. Busy service = more of them.

8. Queue technologies
RabbitMQ
ZeroMQ
SQS
Celery
Gearman

9. RabbitMQ
Written in Erlang.
Open source and Free.
Really Fast.
Socket protocol that is has cheap connections.
Data persists after a crash.

10. Queue basics
Producers create events.
Consumers eat messages, and do work.
messages == events.

11. Queue Basics
Producer

12. Queue Basics
Producer

13. Queue Basics
Producer Consumer

14. Queue Basics
Consumer
Producer Consumer

15. Queue Basics
Consumer
Producer Consumer
Consumer

16. Exchanges
Exchanges abstract queues from the producer.
Route messages to queues, that are bound by
routing keys.
Allows consumers to listen to speci c routing
keys.
Multiple consumers can listen to the same
events.

17. Exchanges
Producer

18. Exchanges
Producer

19. Exchanges
page_view
Consumer
Producer

20. Exchanges
page_view
Consumer
import Consumer
Producer

21. Exchanges
page_view
Consumer
import Consumer
Producer
Consumer
error,
page_view

22. Topic subscription
Consumers can also listen to topics. Which are
keys separated by dots.
e.g page_view.search, page_view.update
Consumer could listen to page_view.*

23. Producing
Can create messages with many tools.
PHP, python, node, ruby, etc.
Messages can contain any stringy data (JSON)

24. Data for messages
Only string-y things can be stored in messages.
JSON encoded data in message.
Key to value stored in Redis/Memcache.
Keep messages small. RabbitMQ performs
better with small messages.

25. Consuming
Use a daemon process.
Daemons in PHP can be done, but they kinda
suck.
Usually use sleep(2) or other workaround.
sleep(2) moves you away from ‘real time’.

26. Aw bugger..

27. Use a better tool
Other tools are much better for making
daemons.
Python (pika, kombu, sparkplug)
NodeJS (node-amqp)
Ruby (carrot, ruby-amqp)

28. Managing mixed applications
Many ways to get it done.
Avoid duplicating code.
Thin daemons + worker processes.

29. Thin daemons
Python/node daemon accepts messages.
Delegate to PHP worker processes.
Keeps PHP’s fork + die work ow.
Re-use application code by calling shells.

30. Thin daemons
PHP

31. Thin daemons
PHP

32. Thin daemons
Py
PHP

33. Thin daemons
Py
PHP Py

34. Thin daemons
Py
PHP Py
Py

35. Thin daemons
Py PHP
PHP Py
Py

36. Thin daemons
Py PHP
PHP Py PHP
Py

37. Thin daemons
Py PHP
PHP Py PHP
Py PHP

38. Demo Time

39. Overview
Use CakeLog to re events.
Use CakePHP shell’s for actual work.
Thin python daemon for processing messages.

40. Python + sparkplug
Python module for creating queue listeners.
Created by a developer at FreshBooks.
Ini con guration le de nes behaviour.

41. What we’ll try:
User events logged during application.
Log events to the database.

42. Insert code

43. Limitations of using shells
Daemon and shell processes have to be on the
same box.
This can be limiting as you scale, depends on
how you structure applications.

44. Solutions
Use HTTP.
Have thin daemons dispatch back to the front-
end application using HTTP.
Use Basic/Digest authentication.
Use checksums.

45. HTTP daemons
PHP

46. HTTP daemons
PHP

47. HTTP daemons
PHP Py

48. HTTP daemons
Web
PHP Py

49. HTTP daemons
Web
PHP Py Web

50. HTTP daemons
Web
PHP Py Web
Web

51. Thanks; Questions?