Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

RabbitMQ Data Ingestion

12,241 views

Published on

This talk explains the use of RabbitMQ Federation to build a distributed system with RabbitMQ

Published in: Business, Technology
  • Be the first to comment

RabbitMQ Data Ingestion

  1. 1. Building a Distributed Data Ingestion System with RabbitMQ Alvaro Videla - RabbitMQ
  2. 2. Alvaro Videla • • • • • Developer Advocate at Pivotal / RabbitMQ! Co-Author of RabbitMQ in Action! Creator of the RabbitMQ Simulator! Blogs about RabbitMQ Internals: http://videlalvaro.github.io/internals.html! @old_sound — alvaro@rabbitmq.com — github.com/videlalvaro

  3. 3. About Me Co-authored! ! RabbitMQ in Action! http://bit.ly/rabbitmq
  4. 4. About this Talk • Exploratory Talk • A ‘what could be done’ talk instead of ‘this is how you do it’
  5. 5. Agenda • Intro to RabbitMQ • The Problem • Solution Proposal • Improvements
  6. 6. What is RabbitMQ
  7. 7. RabbitMQ
  8. 8. RabbitMQ
  9. 9. RabbitMQ • Multi Protocol Messaging Server
  10. 10. RabbitMQ • Multi Protocol Messaging Server! • Open Source (MPL)
  11. 11. RabbitMQ • Multi Protocol Messaging Server! • Open Source (MPL)! • Polyglot
  12. 12. RabbitMQ • Multi Protocol Messaging Server! • Open Source (MPL)! • Polyglot! • Written in Erlang/OTP
  13. 13. Multi Protocol http://bit.ly/rmq-protocols
  14. 14. Polyglot
  15. 15. Polyglot
  16. 16. Polyglot • Java
  17. 17. Polyglot • Java! • node.js
  18. 18. Polyglot • Java! • node.js! • Erlang
  19. 19. Polyglot • Java! • node.js! • Erlang! • PHP
  20. 20. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby
  21. 21. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby! • .Net
  22. 22. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby! • .Net! • Haskell
  23. 23. Polyglot Even COBOL!!!11
  24. 24. Some users of RabbitMQ
  25. 25. Some users of RabbitMQ • Instagram
  26. 26. Some users of RabbitMQ • • Instagram! Indeed.com
  27. 27. Some users of RabbitMQ • • • Instagram! Indeed.com! Telefonica
  28. 28. Some users of RabbitMQ • • • • Instagram! Indeed.com! Telefonica! Mercado Libre
  29. 29. Some users of RabbitMQ • • • • • Instagram! Indeed.com! Telefonica! Mercado Libre! NHS
  30. 30. Some users of RabbitMQ • • • • • • Instagram! Indeed.com! Telefonica! Mercado Libre! NHS! Mozilla
  31. 31. http://www.rabbitmq.com/download.html Unix - Mac - Windows
  32. 32. Messaging with RabbitMQ A demo with the RabbitMQ Simulator https://github.com/RabbitMQSimulator/RabbitMQSimulator
  33. 33. http://tryrabbitmq.com
  34. 34. RabbitMQ Simulator
  35. 35. The Problem
  36. 36. Distributed Application App App App App
  37. 37. Distributed Application App App App App
  38. 38. Ad-hoc solution
  39. 39. A process that replicates data to the remote server
  40. 40. Possible issues • Remote server is offline • • • Prevent unbounded local buffers Prevent message loss Prevent unnecessary message replication • No need for those messages on remote server • Messages that became stale
  41. 41. Can we do better?
  42. 42. RabbitMQ Federation
  43. 43. RabbitMQ Federation • Supports replication across different administrative domains • Supports mix of Erlang and RabbitMQ versions • Supports Network Partitions • Specificity - not everything has to be federated
  44. 44. RabbitMQ Federation
  45. 45. RabbitMQ Federation
  46. 46. RabbitMQ Federation
  47. 47. RabbitMQ Federation • It’s a RabbitMQ Plugin
  48. 48. RabbitMQ Federation • It’s a RabbitMQ Plugin • Internally uses Queues and Exchanges Decorators
  49. 49. RabbitMQ Federation • It’s a RabbitMQ Plugin • Internally uses Queues and Exchanges Decorators • Managed using Parameters and Policies
  50. 50. Enabling the Plugin rabbitmq-plugins enable rabbitmq_federation
  51. 51. Enabling the Plugin rabbitmq-plugins enable rabbitmq_federation rabbitmq-plugins enable rabbitmq_federation_management
  52. 52. Federating an Exchange rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}'
  53. 53. Federating an Exchange rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}' ! rabbitmqctl set_policy --apply-to exchanges federate-me "^amq." '{"federation-upstream-set":"all"}'
  54. 54. Federating an Exchange
  55. 55. Configuring Federation
  56. 56. Config Options rabbitmqctl set_parameter federation-upstream name ‘json-object’
  57. 57. Config Options rabbitmqctl set_parameter federation-upstream name ‘json-object’ ! json-object: { ‘uri’: ‘amqp://server-name/’, ‘prefetch-count’: 1000, ‘reconnect-delay’: 1, ‘ack-mode’: on-confirm } http://www.rabbitmq.com/federation-reference.html
  58. 58. Prevent unbound buffers expires: N // ms. message-ttl: N // ms.
  59. 59. Prevent message forwarding max-hops: N
  60. 60. Speed vs No Message Loss ack-mode: on-confirm ack-mode: on-publish ack-mode: no-ack
  61. 61. AMQP URI: amqp://user:pass@host:10000/vhost http://www.rabbitmq.com/uri-spec.html
  62. 62. Config can be applied via • CLI using rabbitmqctl • HTTP API • RabbitMQ Management Interface
  63. 63. RabbitMQ Federation
  64. 64. Scaling the Setup
  65. 65. The Problem
  66. 66. The Problem • Queues contents live in the node where the Queue was declared
  67. 67. The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every connected node
  68. 68. The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every connected node • Queues are an Erlang process (tied to one core)
  69. 69. The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every connected node • Queues are an Erlang process (tied to one core) • Adding more nodes doesn’t really help
  70. 70. Sharded Queues
  71. 71. Pieces of the Puzzle • consistent hash exchange • good ol’ queues
  72. 72. Sharded Queues
  73. 73. Sharded Queues
  74. 74. Sharded Queues
  75. 75. Sharded Queues • Declare Queues with name: nodename.queuename.index
  76. 76. Sharded Queues • Declare Queues with name: nodename.queuename.index • Bind the queues to a consistent hash exchange
  77. 77. Sharded Queues • Declare Queues with name: nodename.queuename.index • Bind the queues to a consistent hash exchange • Get the consumer to randomly build the queue name
  78. 78. We need more scale!
  79. 79. Federated Queues
  80. 80. Federated Queues • Load-balance messages across federated queues • Only moves messages when needed
  81. 81. Federating a Queue rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}'
  82. 82. Federating a Queue rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}' ! rabbitmqctl set_policy --apply-to queues federate-me "^images." '{"federation-upstream-set":"all"}'
  83. 83. With RabbitMQ we can
  84. 84. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP
  85. 85. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation
  86. 86. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation • Scale up using Sharding
  87. 87. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation • Scale up using Sharding • Load balance consumers with Federated Queues
  88. 88. Credits world map: wikipedia.org federation diagrams: rabbitmq.com
  89. 89. Questions?
  90. 90. Thanks Alvaro Videla - @old_sound

×