Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Building a Distributed Data Ingestion System with RabbitMQ Alvaro Videla - RabbitMQ
Alvaro Videla • • • • • Developer Advocate at Pivotal / RabbitMQ! Co-Author of RabbitMQ in Action! Creator of the RabbitM...
About Me Co-authored! ! RabbitMQ in Action! http://bit.ly/rabbitmq
About this Talk • Exploratory Talk • A ‘what could be done’ talk instead of ‘this is how you do it’
Agenda • Intro to RabbitMQ • The Problem • Solution Proposal • Improvements
What is RabbitMQ
RabbitMQ
RabbitMQ
RabbitMQ • Multi Protocol Messaging Server
RabbitMQ • Multi Protocol Messaging Server! • Open Source (MPL)
RabbitMQ • Multi Protocol Messaging Server! • Open Source (MPL)! • Polyglot
RabbitMQ • Multi Protocol Messaging Server! • Open Source (MPL)! • Polyglot! • Written in Erlang/OTP
Multi Protocol http://bit.ly/rmq-protocols
Polyglot
Polyglot
Polyglot • Java
Polyglot • Java! • node.js
Polyglot • Java! • node.js! • Erlang
Polyglot • Java! • node.js! • Erlang! • PHP
Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby
Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby! • .Net
Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby! • .Net! • Haskell
Polyglot Even COBOL!!!11
Some users of RabbitMQ
Some users of RabbitMQ • Instagram
Some users of RabbitMQ • • Instagram! Indeed.com
Some users of RabbitMQ • • • Instagram! Indeed.com! Telefonica
Some users of RabbitMQ • • • • Instagram! Indeed.com! Telefonica! Mercado Libre
Some users of RabbitMQ • • • • • Instagram! Indeed.com! Telefonica! Mercado Libre! NHS
Some users of RabbitMQ • • • • • • Instagram! Indeed.com! Telefonica! Mercado Libre! NHS! Mozilla
http://www.rabbitmq.com/download.html Unix - Mac - Windows
Messaging with RabbitMQ A demo with the RabbitMQ Simulator https://github.com/RabbitMQSimulator/RabbitMQSimulator
http://tryrabbitmq.com
RabbitMQ Simulator
The Problem
Distributed Application App App App App
Distributed Application App App App App
Ad-hoc solution
A process that replicates data to the remote server
Possible issues • Remote server is offline • • • Prevent unbounded local buffers Prevent message loss Prevent unnecessa...
Can we do better?
RabbitMQ Federation
RabbitMQ Federation • Supports replication across different administrative domains • Supports mix of Erlang and RabbitM...
RabbitMQ Federation
RabbitMQ Federation
RabbitMQ Federation
RabbitMQ Federation • It’s a RabbitMQ Plugin
RabbitMQ Federation • It’s a RabbitMQ Plugin • Internally uses Queues and Exchanges Decorators
RabbitMQ Federation • It’s a RabbitMQ Plugin • Internally uses Queues and Exchanges Decorators • Managed using Parame...
Enabling the Plugin rabbitmq-plugins enable rabbitmq_federation
Enabling the Plugin rabbitmq-plugins enable rabbitmq_federation rabbitmq-plugins enable rabbitmq_federation_management
Federating an Exchange rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3...
Federating an Exchange rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3...
Federating an Exchange
Configuring Federation
Config Options rabbitmqctl set_parameter federation-upstream name ‘json-object’
Config Options rabbitmqctl set_parameter federation-upstream name ‘json-object’ ! json-object: { ‘uri’: ‘amqp://server-na...
Prevent unbound buffers expires: N // ms. message-ttl: N // ms.
Prevent message forwarding max-hops: N
Speed vs No Message Loss ack-mode: on-confirm ack-mode: on-publish ack-mode: no-ack
AMQP URI: amqp://user:pass@host:10000/vhost http://www.rabbitmq.com/uri-spec.html
Config can be applied via • CLI using rabbitmqctl • HTTP API • RabbitMQ Management Interface
RabbitMQ Federation
Scaling the Setup
The Problem
The Problem • Queues contents live in the node where the Queue was declared
The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every...
The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every...
The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every...
Sharded Queues
Pieces of the Puzzle • consistent hash exchange • good ol’ queues
Sharded Queues
Sharded Queues
Sharded Queues
Sharded Queues • Declare Queues with name: nodename.queuename.index
Sharded Queues • Declare Queues with name: nodename.queuename.index • Bind the queues to a consistent hash exchange
Sharded Queues • Declare Queues with name: nodename.queuename.index • Bind the queues to a consistent hash exchange • ...
We need more scale!
Federated Queues
Federated Queues • Load-balance messages across federated queues • Only moves messages when needed
Federating a Queue rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":36000...
Federating a Queue rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":36000...
With RabbitMQ we can
With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP
With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using ...
With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using ...
With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using ...
Credits world map: wikipedia.org federation diagrams: rabbitmq.com
Questions?
Thanks Alvaro Videla - @old_sound
Upcoming SlideShare
Loading in …5
×

RabbitMQ Data Ingestion

12,241 views

Published on

This talk explains the use of RabbitMQ Federation to build a distributed system with RabbitMQ

Published in: Business, Technology
no profile picture user

  • Be the first to comment

Show More

RabbitMQ Data Ingestion

  1. 1. Building a Distributed Data Ingestion System with RabbitMQ Alvaro Videla - RabbitMQ
  2. 2. Alvaro Videla • • • • • Developer Advocate at Pivotal / RabbitMQ! Co-Author of RabbitMQ in Action! Creator of the RabbitMQ Simulator! Blogs about RabbitMQ Internals: http://videlalvaro.github.io/internals.html! @old_sound — alvaro@rabbitmq.com — github.com/videlalvaro

  3. 3. About Me Co-authored! ! RabbitMQ in Action! http://bit.ly/rabbitmq
  4. 4. About this Talk • Exploratory Talk • A ‘what could be done’ talk instead of ‘this is how you do it’
  5. 5. Agenda • Intro to RabbitMQ • The Problem • Solution Proposal • Improvements
  6. 6. What is RabbitMQ
  7. 7. RabbitMQ
  8. 8. RabbitMQ
  9. 9. RabbitMQ • Multi Protocol Messaging Server
  10. 10. RabbitMQ • Multi Protocol Messaging Server! • Open Source (MPL)
  11. 11. RabbitMQ • Multi Protocol Messaging Server! • Open Source (MPL)! • Polyglot
  12. 12. RabbitMQ • Multi Protocol Messaging Server! • Open Source (MPL)! • Polyglot! • Written in Erlang/OTP
  13. 13. Multi Protocol http://bit.ly/rmq-protocols
  14. 14. Polyglot
  15. 15. Polyglot
  16. 16. Polyglot • Java
  17. 17. Polyglot • Java! • node.js
  18. 18. Polyglot • Java! • node.js! • Erlang
  19. 19. Polyglot • Java! • node.js! • Erlang! • PHP
  20. 20. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby
  21. 21. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby! • .Net
  22. 22. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby! • .Net! • Haskell
  23. 23. Polyglot Even COBOL!!!11
  24. 24. Some users of RabbitMQ
  25. 25. Some users of RabbitMQ • Instagram
  26. 26. Some users of RabbitMQ • • Instagram! Indeed.com
  27. 27. Some users of RabbitMQ • • • Instagram! Indeed.com! Telefonica
  28. 28. Some users of RabbitMQ • • • • Instagram! Indeed.com! Telefonica! Mercado Libre
  29. 29. Some users of RabbitMQ • • • • • Instagram! Indeed.com! Telefonica! Mercado Libre! NHS
  30. 30. Some users of RabbitMQ • • • • • • Instagram! Indeed.com! Telefonica! Mercado Libre! NHS! Mozilla
  31. 31. http://www.rabbitmq.com/download.html Unix - Mac - Windows
  32. 32. Messaging with RabbitMQ A demo with the RabbitMQ Simulator https://github.com/RabbitMQSimulator/RabbitMQSimulator
  33. 33. http://tryrabbitmq.com
  34. 34. RabbitMQ Simulator
  35. 35. The Problem
  36. 36. Distributed Application App App App App
  37. 37. Distributed Application App App App App
  38. 38. Ad-hoc solution
  39. 39. A process that replicates data to the remote server
  40. 40. Possible issues • Remote server is offline • • • Prevent unbounded local buffers Prevent message loss Prevent unnecessary message replication • No need for those messages on remote server • Messages that became stale
  41. 41. Can we do better?
  42. 42. RabbitMQ Federation
  43. 43. RabbitMQ Federation • Supports replication across different administrative domains • Supports mix of Erlang and RabbitMQ versions • Supports Network Partitions • Specificity - not everything has to be federated
  44. 44. RabbitMQ Federation
  45. 45. RabbitMQ Federation
  46. 46. RabbitMQ Federation
  47. 47. RabbitMQ Federation • It’s a RabbitMQ Plugin
  48. 48. RabbitMQ Federation • It’s a RabbitMQ Plugin • Internally uses Queues and Exchanges Decorators
  49. 49. RabbitMQ Federation • It’s a RabbitMQ Plugin • Internally uses Queues and Exchanges Decorators • Managed using Parameters and Policies
  50. 50. Enabling the Plugin rabbitmq-plugins enable rabbitmq_federation
  51. 51. Enabling the Plugin rabbitmq-plugins enable rabbitmq_federation rabbitmq-plugins enable rabbitmq_federation_management
  52. 52. Federating an Exchange rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}'
  53. 53. Federating an Exchange rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}' ! rabbitmqctl set_policy --apply-to exchanges federate-me "^amq." '{"federation-upstream-set":"all"}'
  54. 54. Federating an Exchange
  55. 55. Configuring Federation
  56. 56. Config Options rabbitmqctl set_parameter federation-upstream name ‘json-object’
  57. 57. Config Options rabbitmqctl set_parameter federation-upstream name ‘json-object’ ! json-object: { ‘uri’: ‘amqp://server-name/’, ‘prefetch-count’: 1000, ‘reconnect-delay’: 1, ‘ack-mode’: on-confirm } http://www.rabbitmq.com/federation-reference.html
  58. 58. Prevent unbound buffers expires: N // ms. message-ttl: N // ms.
  59. 59. Prevent message forwarding max-hops: N
  60. 60. Speed vs No Message Loss ack-mode: on-confirm ack-mode: on-publish ack-mode: no-ack
  61. 61. AMQP URI: amqp://user:pass@host:10000/vhost http://www.rabbitmq.com/uri-spec.html
  62. 62. Config can be applied via • CLI using rabbitmqctl • HTTP API • RabbitMQ Management Interface
  63. 63. RabbitMQ Federation
  64. 64. Scaling the Setup
  65. 65. The Problem
  66. 66. The Problem • Queues contents live in the node where the Queue was declared
  67. 67. The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every connected node
  68. 68. The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every connected node • Queues are an Erlang process (tied to one core)
  69. 69. The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every connected node • Queues are an Erlang process (tied to one core) • Adding more nodes doesn’t really help
  70. 70. Sharded Queues
  71. 71. Pieces of the Puzzle • consistent hash exchange • good ol’ queues
  72. 72. Sharded Queues
  73. 73. Sharded Queues
  74. 74. Sharded Queues
  75. 75. Sharded Queues • Declare Queues with name: nodename.queuename.index
  76. 76. Sharded Queues • Declare Queues with name: nodename.queuename.index • Bind the queues to a consistent hash exchange
  77. 77. Sharded Queues • Declare Queues with name: nodename.queuename.index • Bind the queues to a consistent hash exchange • Get the consumer to randomly build the queue name
  78. 78. We need more scale!
  79. 79. Federated Queues
  80. 80. Federated Queues • Load-balance messages across federated queues • Only moves messages when needed
  81. 81. Federating a Queue rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}'
  82. 82. Federating a Queue rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}' ! rabbitmqctl set_policy --apply-to queues federate-me "^images." '{"federation-upstream-set":"all"}'
  83. 83. With RabbitMQ we can
  84. 84. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP
  85. 85. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation
  86. 86. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation • Scale up using Sharding
  87. 87. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation • Scale up using Sharding • Load balance consumers with Federated Queues
  88. 88. Credits world map: wikipedia.org federation diagrams: rabbitmq.com
  89. 89. Questions?
  90. 90. Thanks Alvaro Videla - @old_sound

×