Slides from J and Beyond 2014 presentation on asynchronous message queues in Joomla. Asynchronous message queues are an important component of many distributed computing environments. This presentation will show how queues can be used to help deliver scalable and robust web applications. A lightweight library for message queuing in Joomla will be presented and some possible applications in areas such as search indexing and image processing will be explored. Chris is a Joomla developer with a fondness for the British national sport of queuing. He's also a member of the Joomla Production Leadership Team.

1. “Leave it to me,
I'm British,
I know how to queue”
Chris Davenport
Joomla Production Leadership Team
J and Beyond 2014

2. Leave it to me, I'm British, I know how to queue.

3. Leave it to me, I'm British, I know how to queue.

4. Leave it to me, I'm British, I know how to queue.
“to form a line while waiting for something”

5. Leave it to me, I'm British, I know how to queue.
Queues are bad?

6. Leave it to me, I'm British, I know how to queue.
We hate queues

7. Leave it to me, I'm British, I know how to queue.
Especially when it's raining

8. Leave it to me, I'm British, I know how to queue.
Why make me wait!?

9. Leave it to me, I'm British, I know how to queue.
The aim of this presentation

10. Leave it to me, I'm British, I know how to queue.
The aim of this presentation
Some queues are a “good thing”

11. Leave it to me, I'm British, I know how to queue.
Sometimes it's good to wait

12. Leave it to me, I'm British, I know how to queue.
Synchronisation is good

13. Leave it to me, I'm British, I know how to queue.
Good queues
Bad queues

14. Leave it to me, I'm British, I know how to queue.
Queue messages, not people

15. Leave it to me, I'm British, I know how to queue.
Why queue at all?

16. Leave it to me, I'm British, I know how to queue.
1: Image processing

17. Leave it to me, I'm British, I know how to queue.
2: Sending documents

18. Leave it to me, I'm British, I know how to queue.
Queue messages, not people
Asynchronous message queues

19. Leave it to me, I'm British, I know how to queue.
What is a message queue?

20. Leave it to me, I'm British, I know how to queue.
What is a message queue?
Client
(Producer)
Queue

21. Leave it to me, I'm British, I know how to queue.
What is a message queue?
Server
(Consumer)
Queue

22. Leave it to me, I'm British, I know how to queue.
1: Image processing

23. Leave it to me, I'm British, I know how to queue.
1: Image processing
Takes time.
Queue: But not the user's time.
Requires a lot of memory.
Queue: But not the user's memory.
Demand spikes can threaten the server.
Queue: Naturally throttled.

24. Leave it to me, I'm British, I know how to queue.
2: Sending documents

25. Leave it to me, I'm British, I know how to queue.
It's about decoupling
●
Decoupling in space
●
Decoupling in time
●
Decoupling synchronisation

26. Leave it to me, I'm British, I know how to queue.
Space decoupling
●
Different processes
●
Different machines
●
Different locations

27. Leave it to me, I'm British, I know how to queue.
Time decoupling
●
Heavy tasks may be deferred
●
Task streams may be throttled
●
Can cope with unreliable systems
●
Can cope with unreliable networks

28. Leave it to me, I'm British, I know how to queue.
Synchronisation decoupling
●
Clients should never block.
●
Redundancy
●
Scalability

29. Leave it to me, I'm British, I know how to queue.
Queue patterns

30. Leave it to me, I'm British, I know how to queue.
Request - Reply
Producer Q
12345
Consumer

31. Leave it to me, I'm British, I know how to queue.
Request - Reply
Producer ConsumerQ 1
2345

32. Leave it to me, I'm British, I know how to queue.
Request - Reply
Producer Q
Q
R1
Consumer
2345

33. Leave it to me, I'm British, I know how to queue.
Request - Reply
Producer Q
Q
R1
Consumer2
345

34. Leave it to me, I'm British, I know how to queue.
Request - Reply
Producer Q
Q? R1
Consumer2
345

35. Leave it to me, I'm British, I know how to queue.
Request - Reply
Producer Q
Q?
Consumer
Consumer
Consumer
12345
Consumer

36. Leave it to me, I'm British, I know how to queue.
Request - Reply
Producer Q
Q?
2345
Consumer
Consumer
Consumer
Consumer1

37. Leave it to me, I'm British, I know how to queue.
Request - Reply
Producer Q
Q?
345
Consumer
Consumer
Consumer
Consumer1
2

38. Leave it to me, I'm British, I know how to queue.
Request - Reply
Producer Q
Q?
45
Consumer
Consumer
Consumer
Consumer1
2
3

39. Leave it to me, I'm British, I know how to queue.
Request - Reply
Producer Q
Q?
5
Consumer
Consumer
Consumer
Consumer1
2
3
4

40. Leave it to me, I'm British, I know how to queue.
Request - Reply
Producer Q
Q?
R1
Consumer
Consumer
Consumer
Consumer
2
3
4
5

41. Leave it to me, I'm British, I know how to queue.
Request - Reply
Producer Q
Q?
R1
Consumer
Consumer
Consumer
Consumer5
2
3
4

42. Leave it to me, I'm British, I know how to queue.
Pipeline
Producer Q Consumer
12345
ConsumerQ
ConsumerQ

43. Leave it to me, I'm British, I know how to queue.
Pipeline
Producer Q Consumer1
2345
ConsumerQ
ConsumerQ

44. Leave it to me, I'm British, I know how to queue.
Pipeline
Producer Q
Consumer
Consumer
Q
1'
2
345
ConsumerQ

45. Leave it to me, I'm British, I know how to queue.
Pipeline
Producer Q
Consumer
Consumer
Q 1'
2'
3
45
ConsumerQ

46. Leave it to me, I'm British, I know how to queue.
Pipeline
Producer Q
Consumer
Consumer
Q 2'
3'
4
5
ConsumerQ
1''

47. Leave it to me, I'm British, I know how to queue.
Pipeline
Producer Q
Consumer
Consumer
Q 3'
4'
5
ConsumerQ
2''
1''

48. Leave it to me, I'm British, I know how to queue.
Publish - Subscribe
Producer Q
Consumer
Consumer
Consumer
Consumer
12345

49. Leave it to me, I'm British, I know how to queue.
Publish - Subscribe
Producer Q
2345
Consumer
Consumer
Consumer
Consumer1
1
1
1

50. Leave it to me, I'm British, I know how to queue.
Queue packages

51. Leave it to me, I'm British, I know how to queue.
Queue packages
AMQP – Apache Qpid, OpenAMQ, RabbitMQ
ZeroMQ
IBM WebsphereMQ
Amazon SQS
MSMQ
MQTT

52. Leave it to me, I'm British, I know how to queue.
Queue characteristics
Locality
Persistent or non-persistent
Messages have an “in-flight” period or not
Preserve message order or not
Eliminate duplicates or not
Queue full behaviour
Message delivery guarantees

53. Leave it to me, I'm British, I know how to queue.
Using queues in Joomla

54. Leave it to me, I'm British, I know how to queue.
Using queues in Joomla
Most require special server software
Not available on most shared hosts

55. Leave it to me, I'm British, I know how to queue.
Using queues in Joomla
Need a native PHP implementation
Optional adapters for
mainstream message queues

56. Leave it to me, I'm British, I know how to queue.
Queuing in Joomla
GitHub: chrisdavenport/joomla-mq

57. Leave it to me, I'm British, I know how to queue.
Queuing in Joomla
Provide a very basic standard interface to a choice
of message queues.
Currently supports: Array, filesystem, database,
Unix System V, Amazon SQS.
Documented and unit tested.
Pull requests welcome.

58. Leave it to me, I'm British, I know how to queue.
JMq sending a message
$config = new JRegistry($configData);
$q = new JMqQueue($config);
$q->send('myqueue', 'Hello World');

59. Leave it to me, I'm British, I know how to queue.
JMq receiving a message
$config = new JRegistry($configData);
$q = new JMqQueue($config);
$msg = $q->receive('myqueue');
// Process the message here.
$q->delete('myqueue', $msg->messageid);

60. Leave it to me, I'm British, I know how to queue.
Problem...
Generally needs some kind of cron.
Low-cost hosts will need to implement some kind
of “pseudo-cron” system.
Might be possible to use a plugin event to
transparently process a queue on hosts that do
not allow cron.

61. Leave it to me, I'm British, I know how to queue.
Joomla use-cases
Image resizing and watermarking.
Smart Search indexing.
Hit counters.
Ecommerce.
Throttling.
Monitoring, logging, clickstream analysis.
Statistics and report generation.

62. Leave it to me, I'm British, I know how to queue.
Convinced?

63. Leave it to me, I'm British, I know how to queue.
Queue Tips

64. Leave it to me, I'm British, I know how to queue.
Queues are cheap
So feel free to use lots of them

65. Leave it to me, I'm British, I know how to queue.
Avoid fat messages
Message systems can rarely handle
large messages efficiently

66. Leave it to me, I'm British, I know how to queue.
Guaranteed delivery
There's no such thing
Murphy's Law will always win in the end

67. Leave it to me, I'm British, I know how to queue.
Duplicate messages
Assume that some messages
may be duplicated

68. Leave it to me, I'm British, I know how to queue.
Message ordering
Don't assume messages will arrive
in the same order they were sent

69. Leave it to me, I'm British, I know how to queue.
Avoid transactions
Atomic transactions across message queues
are fraught with difficulty, so avoid them
If you absolutely must have transactions
then lookup Raft and Paxos algorithms

70. Leave it to me, I'm British, I know how to queue.
Error handling
Invalid or undeliverable messages
should be forwarded to a dead-letter queue
Process failures should be
reported to an error queue

71. Leave it to me, I'm British, I know how to queue.
Questions?

72. Leave it to me, I'm British, I know how to queue.
Image credits 1
http://commons.wikimedia.org/wiki/File:Queuing_z01.jpg?uselang=en-gb
http://commons.wikimedia.org/wiki/File:Form_an_orderly_queue_to_the_left_%286180460759%29.jpg
http://commons.wikimedia.org/wiki/File:Angry_woman.jpg
http://commons.wikimedia.org/wiki/File:20110529_London_38.JPG
http://commons.wikimedia.org/wiki/File:Take_off_queue,_Heathrow,_10_Sept._2010_-_Flickr_-_PhillipC.jpg
http://commons.wikimedia.org/wiki/File:DigitalPicture.jpg
http://commons.wikimedia.org/wiki/File:Contrast_improvement.jpg
http://commons.wikimedia.org/wiki/File:Base-isolation.gif

73. Leave it to me, I'm British, I know how to queue.
Image credits 2
http://commons.wikimedia.org/wiki/File:Traffic_queues_likely.JPG
http://commons.wikimedia.org/wiki/File:Motorway_queue_near_Lisburn_%282%29_-_geograph.org.uk_-
_1380275.jpg
http://commons.wikimedia.org/wiki/File:Queue_for_the_Banksy_Exhibition_-_geograph.org.uk_-
_1429992.jpg
http://commons.wikimedia.org/wiki/File:1925_ES%26ARobinson_Invoice_with_illustrations.jpg?
uselang=en-gb
http://commons.wikimedia.org/wiki/File:Cirencester_..._snazzy_trolley..jpg
http://commons.wikimedia.org/wiki/File:Copertina_facebook.jpg
http://commons.wikimedia.org/wiki/File:Q_Tips_plain_BG.jpg?uselang=en-gb
http://commons.wikimedia.org/wiki/File:Bus_queue_in_Redhill_Road,_Cobham_-_geograph.org.uk_-
_1800438.jpg

74. Leave it to me, I'm British, I know how to queue.
Image credits 3
http://commons.wikimedia.org/wiki/File:Queue_at_Pashupatinath01.JPG
http://commons.wikimedia.org/wiki/File:Cracow_queue_1986.jpg (short queue)
http://commons.wikimedia.org/wiki/File:Newport_Staplers_Road_traffic_queue_4.JPG
http://commons.wikimedia.org/wiki/File:Queue_outside_a_soup_kitchenat_Klostertorv_in_
%C3%85lborg_1943.jpg
http://commons.wikimedia.org/wiki/File:Bathroom_Scale.jpg?uselang=en-gb
http://commons.wikimedia.org/wiki/File:Super_Cheap_Auto.jpg
http://commons.wikimedia.org/wiki/File:Clones_id.jpg
http://commons.wikimedia.org/wiki/File:Blue_Angels_all_in_a_line_%282501423714%29.jpg

75. Leave it to me, I'm British, I know how to queue.
Image credits 4
http://commons.wikimedia.org/wiki/File:Three_in_a_line_-_geograph.org.uk_-_652726.jpg
http://commons.wikimedia.org/wiki/File:Viruses.jpg