Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Webinar: Queues with RabbitMQ - Lorna Mitchell

560 views

Published on

Queues are a great addition to any application that has some tasks that need processing asynchronously. This could be sending a confirmation email, resizing an avatar, or recalculating a running total of some kind; in all those cases it would be cool to send the response back to the user and then sort out that task later. This session looks at how to use a RabbitMQ job queue in your application. It also looks at how to design elegant and robust long-running workers that will consume the jobs from the queue and process them. This session is ideal for technical leads, developers and architects alike.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Webinar: Queues with RabbitMQ - Lorna Mitchell

  1. 1. Queues With RabbitMQ   Lorna Mitchell, IBM
  2. 2. Introducing Queues Use queues to: • asynchronously process tasks in your (existing) application • e.g. sending email, processing uploads • provide loose coupling points • e.g. delegate heavy tasks, split ownership/technology • enable parts of the system to scale appropriately • e.g. an event in one system causes a thousand other actions @lornajane
  3. 3. Queue Tools A selection of queue-ish tools • RabbitMQ* http://www.rabbitmq.com/ • Gearman http://gearman.org/ • Beanstalkd http://kr.github.io/beanstalkd/ • Kafka https://kafka.apache.org/ • Redis https://redis.io/   * RabbitMQ is used in today's examples @lornajane
  4. 4. Getting To Know RabbitMQ @lornajane
  5. 5. Management Plugin https://www.rabbitmq.com/management.html @lornajane
  6. 6. Getting To Know RabbitMQ Some vocabulary • broker: the RabbitMQ instance • exchange: where to send the messages to • queue: where messages wait to be processed • binding key: rules for which messages go into this queue • message: the data to be processed • routing key: message route information • consumer: worker script to processes the messages @lornajane
  7. 7. RabbitMQ Exchanges Exchanges are the routing logic of RabbitMQ   Messages go to exchanges and the exchanges put them into the correct queues for storage @lornajane
  8. 8. Types Of Exchange Direct: a given routing key puts messages onto the matching queue(s)   Topic: queues are bound by key, and messages are routed to as many queues as their routing key matches   Fanout: messages go to all queues bound to this exchange @lornajane
  9. 9. Default Exchange There is a default exchange in RabbitMQ   Its name is '' and routing is on queue name @lornajane
  10. 10. RabbitMQ Queues Queues can: • have wildcards in binding keys • be durable (messages have their own durability) • have a maximum length • be configured with a "dead letter exchange"   If producer needs feedback, create another queue to put status updates into @lornajane
  11. 11. Example App: Webhooks @lornajane
  12. 12. Example App: Webhooks (code: https://github.com/ibm-watson-data-lab/guestbook )   A simple guestbook application is extended to allow webhook notifications of new comments. @lornajane
  13. 13. Example App: Webhooks @lornajane
  14. 14. Example App: Webhooks @lornajane
  15. 15. Example App: Webhooks   @lornajane
  16. 16. Processing Messages @lornajane
  17. 17. Creating Workers Workers are disposable!   • if things go wrong, exit • separate tool to monitor/restart as needed • beware long-running process hazards • everything processed "at least once" (but maybe more than once, and in any order...) @lornajane
  18. 18. Independent Workers @lornajane
  19. 19. Independent Workers For best results: • awesome, aggregated logging • monitoring: queue size, worker uptime, processing time • miniminum viable dependencies @lornajane
  20. 20. Completed Messages • acknowledge when messages are processed successfully • can acknowledge failure • reject the message • optionally: requeue @lornajane
  21. 21. Dead Letter Queues • reject without requeue • exceed TTL • queue length exceeded @lornajane
  22. 22. Retries Implement your own logic to handle retries   Create a new message with: • all the existing message contents • plus some metadata such as retry count or backoff time @lornajane
  23. 23. Feedback Mechanisms Rabbit is fire-and-forget; work is delegated   Common pattern: return queue to put updates into for the original producer then to consume. @lornajane
  24. 24. Example App: Webhooks   @lornajane
  25. 25. Example App: Webhooks @lornajane
  26. 26. Queues With RabbitMQ @lornajane
  27. 27. Queues With RabbitMQ   Queues are awesome for scalability and robustness   RabbitMQ is open source, lightweight and fast   Queues help us meet the requirements for modern applications @lornajane
  28. 28. Thanks! Resources: • RabbitMQ: https://www.rabbitmq.com/ • Resilience: https://lrnja.net/rabbitfail • Blog: https://lornajane.net • Code: https://github.com/ibm-watson-data-lab/guestbook   • @lornajane • lorna.mitchell@uk.ibm.com @lornajane

×