Message Queues & Offline Processing with PHP

UNCON
Message Queues &Offline Processing in PHP
ByMarcel Esser
Code Samurai, CROSCON
Contact
+1 202 730 9728
+49 176 785 69729
marcel.esser@croscon.com

Offline Processing Explained
What is offline processing?
Short answer: Running things in the background.
Long answer: Offline processing is the act of queuing, prioritizing, dispatching, and executing jobs described by a message packet, received from some initiating process.
Practical answer: Do the heavy stuff in a worker thread.

Offline Processing Explained
Now
Later
Web Request
Dispatcher
Message
ApplicationData
Store
Message
Dispatcher
Worker

Message Queue Solutions

Amazon SQS
Proprietary Web Service. 0.01USD/10,000.00 msgs + bw.
Dead simple.
Handles all locking.
Messages up to 8kb in size.
Messages expire after 4 days.
Many permissions options.
Automatically distributed.
Runs on a 3rd party.
5 functions (CreateQueue, SendMessage, ReceiveMessage, ChangeMessageVisibility, and DeleteMessage)
SOAP or RESTful.

gearman
Open source C daemon.
Socket-based client libraries in several languages.
PHP client library is nice and clean.
Can be configured for redundancy.
Fast.
No Persistence. Now with minty-fresh persistence.

dropr
Open source PHP daemon.
“trunk in production”
Separate client and poll services.
De-centralized
PHP-centric
Storage and transport are modular. FS and cURL are implemented.

Apache ActiveMQ
Open source Java daemon.
Big.
Fairly Java-centric.
De-centralized
PHP-centric
Storage and transport are modular. FS and cURL are implemented.
Uses big words like ‘transport connectors’ and ‘network connectors’ and has big XML configuraton files.
Just fail. Don’t even ask me.

beanstalkd
Open source C daemon.
Fast.
Can be persisted by writing events to a binlog.
Can be distributed.
Still pretty raw.
Client libraries for many languages.
Little bit slanted towards Rails.

My Homebrew (version 1, PHP)
Proprietary PHP library.
Pretty fast.
Implemented as a memcached linked list.
Not even remotely difficult to write.
Can scale up pretty well if done with some consideration.
Used memcache for locking.
I don’t feel comfortable taking PHP out of it’s web role in that way.

My Homebrew (version 2, C)
Proprietary C daemon.
Very, very fast. Non-blocking epoll server. C10K and then some.
Used mutexes for locking.
Highly specialized for the role; not multi-purpose.
Difficult to maintain.
Difficult to write (properly).
Difficult to deploy.

Zend Queue / Zend Job Queue
Largely a collection of adaptors to other message queues, plus a job-based queue system.
Why? Existing APIs + at command

Message Formats

Message Formats
You need something other than XML and JSON because…?

Oh, and btw.
In case you forgot…

Shameless Plug
Hi, my name is Marcel Esser. I work for CROSCON.
We do:
Bespoke application development
Customized service monitoring
MyCourt productivity software
Info?
marcel.esser@croscon.com
202.730.9728

http://distriwatch.de	438
http://www.berejeb.com	43
http://www.slideshare.net	25
http://www.devetdesign.com	7

Message Queues & Offline Processing with PHP

by marcelesser on Oct 28, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 513

Statistics

Message Queues & Offline Processing with PHP — Presentation Transcript