Successfully reported this slideshow.
Mike Willbanks | Barnes & NobleMessage Queues : A Primer
Housekeeping…•  Talk   –  Slides will be posted after the talk.•  Me   –  Sr. Web Architect Manager at NOOK      Developer...
Agenda•    Message Queues?•    Protocols and Software•    Picking a Message Queue•    Best Practices
A Definition“Message queues and mailboxes are software-engineeringcomponents used for interprocess communication, or for i...
What is messaging?“Messaging describes the sending and receiving of data (in theform of messages) between systems. Message...
Producer                 Messages                 Consumer                            Messages                            ...
Describing Message Queuesl    Pub/Subl    FIFO bufferl    Push / Pulll    A way to communicate between      applicatio...
Why to use a Message Queuel    Offload Heavy Workl    Integration with Legacy Systemsl    Asynchronous Processingl    ...
Unix Foundationsl    “Write programs that work together.”l    “Do it in the background.”
Why it mattersl    Web systems need to be geared to run things      asynchronously.l    Distribution of loadl    System...
You’ve seen them before; they are used in most applications tohelp them scale.MESSAGE QUEUE EXAMPLES
When to make use of message queues in case you’re wondering.GENERIC USE CASES
NotificationsEmail, SMS, Push Messaging….
Photo ProcessingThumbnails, Resizing, Watermarking, Converting…
Video ProcessingResampling, Audio Overlay, Type Conversion…
AnalyticsWeb Server Logs, Log Aggregation, PHP Errors, etc.
IntegrationsSave local first; push second.
AMQPSTOMPXMPPVendor SpecificPROTOCOLS
AMQPAdvanced Message Queuing Protocol
Overview of AMQPl    AMQP Working Group (Community and      Vendor)l    Platform agnostic protocol.l    Completely open...
How it WorksProducer, Exchange, Queue, Consumer
How it Worksl    AMQP utilizes exchanges, queues and      bindings.l    An exchange are routers with routing tables.l  ...
Exchanges
Understanding Exchangesl    Fanout Exchange       l  No routing keys involved. Any message that is           sent to the...
Implementations  www.rabbitmq.com  Very popular and common message queue owned by  VMware.  qpid.apache.org  Long standing...
Building a Queuel    An exchange, queue and bindings must be defined      first. Publishing can then commence after.     ...
Persistence?l    Default behavior is no persistence.l    How important are the messages?l    Just about all items have ...
PECL AMQPl    Extension compatible with AMQP specification      0-9-1.l    pecl install amqp
AMQP Client
AMQP Worker
STOMPSimple (or Streaming) Text Orientated Messaging Protocol
Overviewl    Simple protocol      l    Behaviors follow very simple commands.l    Most message queues can communicate o...
/queue/                                            msg              Connect           Send                Disconnect      ...
Sever Implementations       activemq.apache.org       One of the oldest message queues existing;       a apache foundation...
PECL Stompl    pecl install stompl    That was easy J
STOMP Client
STOMP Worker
XMPPExtensible Messaging and Presence Protocol(Although not really a “Message Queue”)
Overviewl    Best for real-time data.l    Leveraging pub/sub can turn it into more of a generic message      system.l  ...
Publish                                      to, id,                                     message                          ...
Publish
Message
Various others; including Gearman, ZeroMQ, etc.SPECIAL MESSAGE QUEUES
Overviewl    There are job servers available that are more      flexible or more specific.      l    Extreme Flexibility...
ZeroMQThe ultimate in message queue flexibility.Socket library that acts as a concurrency framework.Contains a PHP extensi...
Several Types of QueuesRequest / ReplyPublish / SubscribeParallel PipelineFair QueuingAnd more…
ZeroMQ Client Example
ZeroMQ Worker Example
Gearmanl    Application framework for farming out work.l    Job sever for asynchronous or synchronous      messages.
Gearmanl    Pass a job to the job serverl    Worker receives the job and processesl    Ability to persist if enabled; d...
Gearman Client Example
Gearman Worker Example
Beanstalkdl    Asynchronous Job Queuel    Good scheduling system (aka delays).l    User land PHP clients.
Application Code                         Beanstalkd Client API                             (Pheanstalk)                   ...
Pheanstalk Client
Pheanstalk Worker
What you want to look for.PICKING A MESSAGE QUEUE
The Trifecta                                                  l?                                        Horizonta         ...
Standards•  A recognized standard?  –  AMQP? STOMP? XMPP?•  How many developers?  –  Will an unfortunate event kill off th...
Delivery Policies Store Message                  Saved?Receive Message                 TTL?Process Message                ...
Message Policies•    Handling in-active messages?•    Messages that failed processing?•    Time to live?•    Retry?•    Ab...
Routing Policies•    Fanout?•    Direct?•    Topic?•    Broadcast?•    Etc?
Security Policies•  Require authentication?  –  LDAP/AD integration?•  Connection Restrictions?•  Require logging?  –  Com...
Rules of the road my friends…BEST PRACTICES
Messages•  Formatting  –  JSON or XML are great options.     •  Please no serialized PHP objects.•  Message Size  –  Only ...
Workers•  Dumb as possible!  –  Single message type     •  Easier to scale.  –  Single operation     •  Easy to debug and ...
PHP Daemons•  Prevent Memory Leaks  –  Detection  –  Cycle Workers•  Handle Signals  –  Properly shutdown!  –  Watch out f...
Database   Worker Server                       Web Server                       Message QueueEnvironment IntegrationWeb Se...
Database                                                             Web Server                  Worker Server  Worker Ser...
SupervisordProcess monitoring, logging and more!
Installing Supervisord•  Requires: python-setuptools         sudo easy_install supervisor!•  Generate Configuration      s...
These slides will be posted to SlideShare & SpeakerDeck.   SpeakerDeck: http://speakerdeck.com/u/mwillbanks   Slideshare: ...
Message Queues : A Primer - International PHP Conference Fall 2012
Message Queues : A Primer - International PHP Conference Fall 2012
Message Queues : A Primer - International PHP Conference Fall 2012
Message Queues : A Primer - International PHP Conference Fall 2012
Message Queues : A Primer - International PHP Conference Fall 2012
Upcoming SlideShare
Loading in …5
×

Message Queues : A Primer - International PHP Conference Fall 2012

9,996 views

Published on

Message Queues - A Primer is an introduction to message queues and how they work with PHP.

Published in: Technology

Message Queues : A Primer - International PHP Conference Fall 2012

  1. 1. Mike Willbanks | Barnes & NobleMessage Queues : A Primer
  2. 2. Housekeeping…•  Talk –  Slides will be posted after the talk.•  Me –  Sr. Web Architect Manager at NOOK Developer –  Prior MNPHP Organizer –  Open Source Contributor –  Where you can find me: •  Twitter: mwillbanks G+: Mike Willbanks •  IRC (freenode): mwillbanks Blog: http://blog.digitalstruct.com •  GitHub: https://github.com/mwillbanks
  3. 3. Agenda•  Message Queues?•  Protocols and Software•  Picking a Message Queue•  Best Practices
  4. 4. A Definition“Message queues and mailboxes are software-engineeringcomponents used for interprocess communication, or for inter-thread communication within the same process. They use a queuefor messaging – the passing of control or of content.”
  5. 5. What is messaging?“Messaging describes the sending and receiving of data (in theform of messages) between systems. Messages are exchangedbetween programs or applications, similar to the way peoplecommunicate by email but with guarantees on delivery, speed,security and the absence of spam.”
  6. 6. Producer Messages Consumer Messages Task MessagesGeneral AnatomyProducer creates a message and pushes it to the queue; theconsumer reads from the queue and processes the message.
  7. 7. Describing Message Queuesl  Pub/Subl  FIFO bufferl  Push / Pulll  A way to communicate between applications / systems.l  A way to decouple components.l  A way to offload work.
  8. 8. Why to use a Message Queuel  Offload Heavy Workl  Integration with Legacy Systemsl  Asynchronous Processingl  Parallel Processingl  Process consistencyl  Scalability
  9. 9. Unix Foundationsl  “Write programs that work together.”l  “Do it in the background.”
  10. 10. Why it mattersl  Web systems need to be geared to run things asynchronously.l  Distribution of loadl  System integrity
  11. 11. You’ve seen them before; they are used in most applications tohelp them scale.MESSAGE QUEUE EXAMPLES
  12. 12. When to make use of message queues in case you’re wondering.GENERIC USE CASES
  13. 13. NotificationsEmail, SMS, Push Messaging….
  14. 14. Photo ProcessingThumbnails, Resizing, Watermarking, Converting…
  15. 15. Video ProcessingResampling, Audio Overlay, Type Conversion…
  16. 16. AnalyticsWeb Server Logs, Log Aggregation, PHP Errors, etc.
  17. 17. IntegrationsSave local first; push second.
  18. 18. AMQPSTOMPXMPPVendor SpecificPROTOCOLS
  19. 19. AMQPAdvanced Message Queuing Protocol
  20. 20. Overview of AMQPl  AMQP Working Group (Community and Vendor)l  Platform agnostic protocol.l  Completely open, interoperable and broadly applicable.l  Many severs available and many client libraries.
  21. 21. How it WorksProducer, Exchange, Queue, Consumer
  22. 22. How it Worksl  AMQP utilizes exchanges, queues and bindings.l  An exchange are routers with routing tables.l  A binding defines the routing rules.l  A queue is where the messages wait for a consumer.
  23. 23. Exchanges
  24. 24. Understanding Exchangesl  Fanout Exchange l  No routing keys involved. Any message that is sent to the exchange is sent to all queues bound to that exchange.l  Direct Exchange l  Routing keys involved. A queue binds to the exchange to request messages that match a routing key exactly.l  Topic Exchange l  Routing keys involved. A queue binds to the exchange to request messages that match a routing key pattern.
  25. 25. Implementations www.rabbitmq.com Very popular and common message queue owned by VMware. qpid.apache.org Long standing project; apache foundation. www.openamq.org Long standing project; ZeroMQ partner, no news since 2009.
  26. 26. Building a Queuel  An exchange, queue and bindings must be defined first. Publishing can then commence after. l  Create the queue l  Create the exchange l  Bind to the queue.
  27. 27. Persistence?l  Default behavior is no persistence.l  How important are the messages?l  Just about all items have a level of persistence if you would like them to survive on reboot. l  Mark exchanges, queues and messages as DURABLE.
  28. 28. PECL AMQPl  Extension compatible with AMQP specification 0-9-1.l  pecl install amqp
  29. 29. AMQP Client
  30. 30. AMQP Worker
  31. 31. STOMPSimple (or Streaming) Text Orientated Messaging Protocol
  32. 32. Overviewl  Simple protocol l  Behaviors follow very simple commands.l  Most message queues can communicate over STOMP.
  33. 33. /queue/ msg Connect Send Disconnect /queue/ S msg T O Connect Subscribe Disconnect M P P H S P E R V Read E R AckHow It Worksl  When you send in a message, you tell it which queue to go to.l  When you subscribe you request a queue.
  34. 34. Sever Implementations activemq.apache.org One of the oldest message queues existing; a apache foundation project activemq.apache.org/apollo Next generation ActiveMQ www.rabbitmq.com Very popular and common message queue owned by VMware. www.jboss.org/hornetq Supported by Red Hat Middle Ware division, picking up steam.
  35. 35. PECL Stompl  pecl install stompl  That was easy J
  36. 36. STOMP Client
  37. 37. STOMP Worker
  38. 38. XMPPExtensible Messaging and Presence Protocol(Although not really a “Message Queue”)
  39. 39. Overviewl  Best for real-time data.l  Leveraging pub/sub can turn it into more of a generic message system.l  Multiple libraries l  JAXL - https://github.com/abhinavsingh/JAXL l  Xmpp - https://github.com/alexmace/Xmpp
  40. 40. Publish to, id, message Subscribe id, from, to, message Sub1 Sub2 Sub3XEP-0060: Publish-Subscribel  Specification for implementing Publish Subscribe models.l  Extension to the original XMPP specification.
  41. 41. Publish
  42. 42. Message
  43. 43. Various others; including Gearman, ZeroMQ, etc.SPECIAL MESSAGE QUEUES
  44. 44. Overviewl  There are job servers available that are more flexible or more specific. l  Extreme Flexibility l  Job Severs l  Cloud Messaging
  45. 45. ZeroMQThe ultimate in message queue flexibility.Socket library that acts as a concurrency framework.Contains a PHP extension.
  46. 46. Several Types of QueuesRequest / ReplyPublish / SubscribeParallel PipelineFair QueuingAnd more…
  47. 47. ZeroMQ Client Example
  48. 48. ZeroMQ Worker Example
  49. 49. Gearmanl  Application framework for farming out work.l  Job sever for asynchronous or synchronous messages.
  50. 50. Gearmanl  Pass a job to the job serverl  Worker receives the job and processesl  Ability to persist if enabled; default is in-memory.
  51. 51. Gearman Client Example
  52. 52. Gearman Worker Example
  53. 53. Beanstalkdl  Asynchronous Job Queuel  Good scheduling system (aka delays).l  User land PHP clients.
  54. 54. Application Code Beanstalkd Client API (Pheanstalk) Beanstalkd Server Beanstalkd Worker API (Pheanstalk) Worker Application CodeBeanstalkdl  Pass a job to the job serverl  Worker receives the job and processesl  Ability to persist to binlog; default is in-memory
  55. 55. Pheanstalk Client
  56. 56. Pheanstalk Worker
  57. 57. What you want to look for.PICKING A MESSAGE QUEUE
  58. 58. The Trifecta l? Horizonta Vertical? Scale Messages Messages Messages Messages Messages Messages Messages Messages Messages ry? ed? In-MemoR a w Sp e ce? es? Performance Durability Persisten# Messag n? Replicatio
  59. 59. Standards•  A recognized standard? –  AMQP? STOMP? XMPP?•  How many developers? –  Will an unfortunate event kill off the product?•  Undocumented protocol? –  Forget about it!
  60. 60. Delivery Policies Store Message Saved?Receive Message TTL?Process Message ACK?
  61. 61. Message Policies•  Handling in-active messages?•  Messages that failed processing?•  Time to live?•  Retry?•  Ability to check message status?
  62. 62. Routing Policies•  Fanout?•  Direct?•  Topic?•  Broadcast?•  Etc?
  63. 63. Security Policies•  Require authentication? –  LDAP/AD integration?•  Connection Restrictions?•  Require logging? –  Compliance –  Intrusion Detection
  64. 64. Rules of the road my friends…BEST PRACTICES
  65. 65. Messages•  Formatting –  JSON or XML are great options. •  Please no serialized PHP objects.•  Message Size –  Only as large as necessary. •  Don’t send a binary object through the queue.
  66. 66. Workers•  Dumb as possible! –  Single message type •  Easier to scale. –  Single operation •  Easy to debug and far more flexible.
  67. 67. PHP Daemons•  Prevent Memory Leaks –  Detection –  Cycle Workers•  Handle Signals –  Properly shutdown! –  Watch out for OS service kills•  Sleeping is good J
  68. 68. Database Worker Server Web Server Message QueueEnvironment IntegrationWeb Server -> Message QueueWorker Server -> Message Queue
  69. 69. Database Web Server Worker Server Worker Server Message Queue Send to 1 Message QueueHigh AvailabilityInsert multiple message queue servers + multiple worker nodes.Each worker node can connect to as many message queue serversas necessary.
  70. 70. SupervisordProcess monitoring, logging and more!
  71. 71. Installing Supervisord•  Requires: python-setuptools sudo easy_install supervisor!•  Generate Configuration sudo echo_supervisord_conf > /etc/ supervisord.conf! !
  72. 72. These slides will be posted to SlideShare & SpeakerDeck. SpeakerDeck: http://speakerdeck.com/u/mwillbanks Slideshare: http://www.slideshare.net/mwillbanks Twitter: mwillbanks G+: Mike Willbanks IRC (freenode): mwillbanks Blog: http://blog.digitalstruct.com GitHub: https://github.com/mwillbanksQUESTIONS?

×