Your SlideShare is downloading. ×
0
Building a Distributed
Data Ingestion System
with RabbitMQ
Alvaro Videla - RabbitMQ
Alvaro Videla
• Developer Advocate at Pivotal / RabbitMQ!
• Co-Author of RabbitMQ in Action!
• Creator of the RabbitMQ Sim...
About Me
Co-authored!
!
RabbitMQ in Action!
http://bit.ly/rabbitmq
About this Talk
• Exploratory Talk
• A ‘what could be done’ talk instead of ‘this is how you do it’
Agenda
• Intro to RabbitMQ
• The Problem
• Solution Proposal
• Improvements
Intermission
Intermission
History Lessons
The Hungarian Connection
The Hungarian Connection
• I’m from Uruguay
The Hungarian Connection
• I’m from Uruguay	

• I live in Switzerland
The Hungarian Connection
• I’m from Uruguay	

• I live in Switzerland	

• I’m in Hungary right now
WHAT?
The Hungarian Connection
The Hungarian Connection
Switzerland World Cup - 1954
The Hungarian Connection
Switzerland World Cup - 1954
The Hungarian Connection
Switzerland World Cup - 1954
Hungary 4 - Uruguay 2
The Hungarian Connection II
Tivadar Puskás
Telephone Switch Inventor
The Hungarian Connection III
Old Hungarian Alphabet
The Hungarian Connection III
Erlang
The Hungarian Connection III
As Neumann János Lajos said
The Hungarian Connection III
As Neumann János Lajos said
Use Erlang
What is RabbitMQ
RabbitMQ
RabbitMQ
RabbitMQ
• Multi Protocol Messaging Server
• Multi Protocol Messaging Server!
• Open Source (MPL)
RabbitMQ
• Multi Protocol Messaging Server!
• Open Source (MPL)!
• Polyglot
RabbitMQ
• Multi Protocol Messaging Server!
• Open Source (MPL)!
• Polyglot!
• Written in Erlang/OTP
RabbitMQ
Multi Protocol
http://bit.ly/rmq-protocols
http://www.rabbitmq.com/community-plugins.html
Community Plugins
Polyglot
Polyglot
Polyglot
• Java
Polyglot
• Java!
• node.js
Polyglot
• Java!
• node.js!
• Erlang
Polyglot
• Java!
• node.js!
• Erlang!
• PHP
Polyglot
• Java!
• node.js!
• Erlang!
• PHP!
• Ruby
Polyglot
• Java!
• node.js!
• Erlang!
• PHP!
• Ruby!
• .Net
Polyglot
• Java!
• node.js!
• Erlang!
• PHP!
• Ruby!
• .Net!
• Haskell
Polyglot
Even COBOL!!!11
Some users of RabbitMQ
• Instagram
Some users of RabbitMQ
• Instagram!
• Indeed.com
Some users of RabbitMQ
• Instagram!
• Indeed.com!
• Telefonica
Some users of RabbitMQ
• Instagram!
• Indeed.com!
• Telefonica!
• Mercado Libre
Some users of RabbitMQ
• Instagram!
• Indeed.com!
• Telefonica!
• Mercado Libre!
• NHS
Some users of RabbitMQ
• Instagram!
• Indeed.com!
• Telefonica!
• Mercado Libre!
• NHS!
• Mozilla
Some users of RabbitMQ
The NewYork Times on RabbitMQ
This architecture - Fabrik - has dozens of RabbitMQ instances
spread across 6 AWS zones in O...
http://www.rabbitmq.com/download.html
Unix - Mac - Windows
Messaging with RabbitMQ
A demo with the RabbitMQ Simulator
https://github.com/RabbitMQSimulator/RabbitMQSimulator
http://tryrabbitmq.com
RabbitMQ Simulator
The Problem
Distributed Application
App
App
App
App
Distributed Application
App
App
App
App
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
!
Connection connection = factory.newCo...
channel.exchangeDeclare(EXCHANGE_NAME, "direct", true);
Data Producer
Declare an Exchange
String message = "Hello Federation!";
channel.basicPublish(EXCHANGE_NAME, "", null,
message.getBytes());
Data Producer
Pub...
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
!
Connection connection = factory.newCo...
channel.queueDeclare(QUEUE_NAME, true, false, false,
null);
channel.exchangeDeclare(EXCHANGE_NAME, "direct", true);
channe...
QueueingConsumer consumer = new
QueueingConsumer(channel);
channel.basicConsume(QUEUE_NAME, false, consumer);
Data Consume...
while (true) {
QueueingConsumer.Delivery delivery =
consumer.nextDelivery();
String message = new String(delivery.getBody(...
Ad-hoc solution
A process that replicates data
to the remote server
Possible issues
• Remote server is offline
• Prevent unbounded local buffers
• Prevent message loss
• Prevent unnecessary m...
Can we do better?
RabbitMQ Federation
RabbitMQ Federation
• Supports replication across different administrative domains
• Supports mix of Erlang and RabbitMQ v...
RabbitMQ Federation
RabbitMQ Federation
RabbitMQ Federation
RabbitMQ Federation
• It’s a RabbitMQ Plugin
RabbitMQ Federation
• It’s a RabbitMQ Plugin
• Internally uses Queues and Exchanges Decorators
RabbitMQ Federation
• It’s a RabbitMQ Plugin
• Internally uses Queues and Exchanges Decorators
• Managed using Parameters ...
Enabling the Plugin
rabbitmq-plugins enable rabbitmq_federation
Enabling the Plugin
rabbitmq-plugins enable rabbitmq_federation
rabbitmq-plugins enable rabbitmq_federation_management
Federating an Exchange
rabbitmqctl set_parameter federation-upstream my-upstream 
‘{“uri":"amqp://server-name","expires":3...
Federating an Exchange
rabbitmqctl set_parameter federation-upstream my-upstream 
‘{“uri":"amqp://server-name","expires":3...
Federating an Exchange
Configuring Federation
Config Options
rabbitmqctl set_parameter federation-upstream 
name ‘json-object’
Config Options
rabbitmqctl set_parameter federation-upstream 
name ‘json-object’
!
json-object: {
‘uri’: ‘amqp://server-nam...
Prevent unbound buffers
expires: N // ms.
message-ttl: N // ms.
Prevent message forwarding
max-hops: N
Speed vs No Message Loss
ack-mode: on-confirm
ack-mode: on-publish
ack-mode: no-ack
AMQP URI:
amqp://user:pass@host:10000/vhost
http://www.rabbitmq.com/uri-spec.html
Config can be applied via
• CLI using rabbitmqctl
• HTTP API
• RabbitMQ Management Interface
RabbitMQ Federation
Scaling the Setup
The Problem
The Problem
• Queues contents live in the node where the Queue was declared
The Problem
• Queues contents live in the node where the Queue was declared
• A cluster can access the queue from every co...
The Problem
• Queues contents live in the node where the Queue was declared
• A cluster can access the queue from every co...
The Problem
• Queues contents live in the node where the Queue was declared
• A cluster can access the queue from every co...
Enter Sharded Queues
Enter Sharded Queues
Pieces of the Puzzle
• modulo hash exchange (consistent hash works as well)
• good ol’ queues
Sharded Queues
Sharded Queues
Sharded Queues
Sharded Queues
Sharded Queues
• Declare Queues with name: nodename.queuename.index
Sharded Queues
• Declare Queues with name: nodename.queuename.index
• Bind the queues to a consistent hash exchange
Sharded Queues
• Declare Queues with name: nodename.queuename.index
• Bind the queues to a partitioner exchange
• Transpar...
We need more scale!
Federated Queues
Federated Queues
• Load-balance messages across federated queues
• Only moves messages when needed
Federating a Queue
rabbitmqctl set_parameter federation-upstream my-upstream 
‘{“uri":"amqp://server-name","expires":36000...
Federating a Queue
rabbitmqctl set_parameter federation-upstream my-upstream 
‘{“uri":"amqp://server-name","expires":36000...
With RabbitMQ we can
With RabbitMQ we can
• Ingest data using various protocols: AMQP, MQTT and STOMP
With RabbitMQ we can
• Ingest data using various protocols: AMQP, MQTT and STOMP
• Distribute that data globally using Fed...
With RabbitMQ we can
• Ingest data using various protocols: AMQP, MQTT and STOMP
• Distribute that data globally using Fed...
With RabbitMQ we can
• Ingest data using various protocols: AMQP, MQTT and STOMP
• Distribute that data globally using Fed...
Credits
world map: wikipedia.org
federation diagrams: rabbitmq.com
Questions?
Thanks
AlvaroVidela - @old_sound
Upcoming SlideShare
Loading in...5
×

RabbitMQ Data Ingestion at Craft Conf

1,631

Published on

Presentation about RabbitMQ and Federation

Published in: Technology

Transcript of "RabbitMQ Data Ingestion at Craft Conf"

  1. 1. Building a Distributed Data Ingestion System with RabbitMQ Alvaro Videla - RabbitMQ
  2. 2. Alvaro Videla • Developer Advocate at Pivotal / RabbitMQ! • Co-Author of RabbitMQ in Action! • Creator of the RabbitMQ Simulator! • Blogs about RabbitMQ Internals: http://videlalvaro.github.io/internals.html! • @old_sound — alvaro@rabbitmq.com — github.com/videlalvaro

  3. 3. About Me Co-authored! ! RabbitMQ in Action! http://bit.ly/rabbitmq
  4. 4. About this Talk • Exploratory Talk • A ‘what could be done’ talk instead of ‘this is how you do it’
  5. 5. Agenda • Intro to RabbitMQ • The Problem • Solution Proposal • Improvements
  6. 6. Intermission
  7. 7. Intermission History Lessons
  8. 8. The Hungarian Connection
  9. 9. The Hungarian Connection • I’m from Uruguay
  10. 10. The Hungarian Connection • I’m from Uruguay • I live in Switzerland
  11. 11. The Hungarian Connection • I’m from Uruguay • I live in Switzerland • I’m in Hungary right now
  12. 12. WHAT?
  13. 13. The Hungarian Connection
  14. 14. The Hungarian Connection Switzerland World Cup - 1954
  15. 15. The Hungarian Connection Switzerland World Cup - 1954
  16. 16. The Hungarian Connection Switzerland World Cup - 1954 Hungary 4 - Uruguay 2
  17. 17. The Hungarian Connection II Tivadar Puskás Telephone Switch Inventor
  18. 18. The Hungarian Connection III Old Hungarian Alphabet
  19. 19. The Hungarian Connection III Erlang
  20. 20. The Hungarian Connection III As Neumann János Lajos said
  21. 21. The Hungarian Connection III As Neumann János Lajos said Use Erlang
  22. 22. What is RabbitMQ
  23. 23. RabbitMQ
  24. 24. RabbitMQ
  25. 25. RabbitMQ • Multi Protocol Messaging Server
  26. 26. • Multi Protocol Messaging Server! • Open Source (MPL) RabbitMQ
  27. 27. • Multi Protocol Messaging Server! • Open Source (MPL)! • Polyglot RabbitMQ
  28. 28. • Multi Protocol Messaging Server! • Open Source (MPL)! • Polyglot! • Written in Erlang/OTP RabbitMQ
  29. 29. Multi Protocol http://bit.ly/rmq-protocols
  30. 30. http://www.rabbitmq.com/community-plugins.html Community Plugins
  31. 31. Polyglot
  32. 32. Polyglot
  33. 33. Polyglot • Java
  34. 34. Polyglot • Java! • node.js
  35. 35. Polyglot • Java! • node.js! • Erlang
  36. 36. Polyglot • Java! • node.js! • Erlang! • PHP
  37. 37. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby
  38. 38. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby! • .Net
  39. 39. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby! • .Net! • Haskell
  40. 40. Polyglot Even COBOL!!!11
  41. 41. Some users of RabbitMQ
  42. 42. • Instagram Some users of RabbitMQ
  43. 43. • Instagram! • Indeed.com Some users of RabbitMQ
  44. 44. • Instagram! • Indeed.com! • Telefonica Some users of RabbitMQ
  45. 45. • Instagram! • Indeed.com! • Telefonica! • Mercado Libre Some users of RabbitMQ
  46. 46. • Instagram! • Indeed.com! • Telefonica! • Mercado Libre! • NHS Some users of RabbitMQ
  47. 47. • Instagram! • Indeed.com! • Telefonica! • Mercado Libre! • NHS! • Mozilla Some users of RabbitMQ
  48. 48. The NewYork Times on RabbitMQ This architecture - Fabrik - has dozens of RabbitMQ instances spread across 6 AWS zones in Oregon and Dublin. Upon launch today, the system autoscaled to ~500,000 users. Connection times remained flat at ~200ms. http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2014-January/032943.html
  49. 49. http://www.rabbitmq.com/download.html Unix - Mac - Windows
  50. 50. Messaging with RabbitMQ A demo with the RabbitMQ Simulator https://github.com/RabbitMQSimulator/RabbitMQSimulator
  51. 51. http://tryrabbitmq.com
  52. 52. RabbitMQ Simulator
  53. 53. The Problem
  54. 54. Distributed Application App App App App
  55. 55. Distributed Application App App App App
  56. 56. ConnectionFactory factory = new ConnectionFactory(); factory.setHost("localhost"); ! Connection connection = factory.newConnection(); ! Channel channel = connection.createChannel(); Data Producer Obtain a Channel
  57. 57. channel.exchangeDeclare(EXCHANGE_NAME, "direct", true); Data Producer Declare an Exchange
  58. 58. String message = "Hello Federation!"; channel.basicPublish(EXCHANGE_NAME, "", null, message.getBytes()); Data Producer Publish a message
  59. 59. ConnectionFactory factory = new ConnectionFactory(); factory.setHost("localhost"); ! Connection connection = factory.newConnection(); ! Channel channel = connection.createChannel(); Data Consumer Obtain a Channel
  60. 60. channel.queueDeclare(QUEUE_NAME, true, false, false, null); channel.exchangeDeclare(EXCHANGE_NAME, "direct", true); channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, ""); Data Consumer Declare Queue and bind it
  61. 61. QueueingConsumer consumer = new QueueingConsumer(channel); channel.basicConsume(QUEUE_NAME, false, consumer); Data Consumer Start a consumer
  62. 62. while (true) { QueueingConsumer.Delivery delivery = consumer.nextDelivery(); String message = new String(delivery.getBody()); System.out.println("Received '" + message + "'"); ! channel.basicAck( delivery.getEnvelope(). getDeliveryTag(), false); } Data Consumer Process messages
  63. 63. Ad-hoc solution
  64. 64. A process that replicates data to the remote server
  65. 65. Possible issues • Remote server is offline • Prevent unbounded local buffers • Prevent message loss • Prevent unnecessary message replication • No need for those messages on remote server • Messages that became stale
  66. 66. Can we do better?
  67. 67. RabbitMQ Federation
  68. 68. RabbitMQ Federation • Supports replication across different administrative domains • Supports mix of Erlang and RabbitMQ versions • Supports Network Partitions • Specificity - not everything has to be federated
  69. 69. RabbitMQ Federation
  70. 70. RabbitMQ Federation
  71. 71. RabbitMQ Federation
  72. 72. RabbitMQ Federation • It’s a RabbitMQ Plugin
  73. 73. RabbitMQ Federation • It’s a RabbitMQ Plugin • Internally uses Queues and Exchanges Decorators
  74. 74. RabbitMQ Federation • It’s a RabbitMQ Plugin • Internally uses Queues and Exchanges Decorators • Managed using Parameters and Policies
  75. 75. Enabling the Plugin rabbitmq-plugins enable rabbitmq_federation
  76. 76. Enabling the Plugin rabbitmq-plugins enable rabbitmq_federation rabbitmq-plugins enable rabbitmq_federation_management
  77. 77. Federating an Exchange rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}'
  78. 78. Federating an Exchange rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}' ! rabbitmqctl set_policy --apply-to exchanges federate-me "^amq." '{"federation-upstream-set":"all"}'
  79. 79. Federating an Exchange
  80. 80. Configuring Federation
  81. 81. Config Options rabbitmqctl set_parameter federation-upstream name ‘json-object’
  82. 82. Config Options rabbitmqctl set_parameter federation-upstream name ‘json-object’ ! json-object: { ‘uri’: ‘amqp://server-name/’, ‘prefetch-count’: 1000, ‘reconnect-delay’: 1, ‘ack-mode’: on-confirm } http://www.rabbitmq.com/federation-reference.html
  83. 83. Prevent unbound buffers expires: N // ms. message-ttl: N // ms.
  84. 84. Prevent message forwarding max-hops: N
  85. 85. Speed vs No Message Loss ack-mode: on-confirm ack-mode: on-publish ack-mode: no-ack
  86. 86. AMQP URI: amqp://user:pass@host:10000/vhost http://www.rabbitmq.com/uri-spec.html
  87. 87. Config can be applied via • CLI using rabbitmqctl • HTTP API • RabbitMQ Management Interface
  88. 88. RabbitMQ Federation
  89. 89. Scaling the Setup
  90. 90. The Problem
  91. 91. The Problem • Queues contents live in the node where the Queue was declared
  92. 92. The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every connected node
  93. 93. The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every connected node • Queues are an Erlang process (tied to one core)
  94. 94. The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every connected node • Queues are an Erlang process (tied to one core) • Adding more nodes doesn’t really help
  95. 95. Enter Sharded Queues
  96. 96. Enter Sharded Queues
  97. 97. Pieces of the Puzzle • modulo hash exchange (consistent hash works as well) • good ol’ queues
  98. 98. Sharded Queues
  99. 99. Sharded Queues
  100. 100. Sharded Queues
  101. 101. Sharded Queues
  102. 102. Sharded Queues • Declare Queues with name: nodename.queuename.index
  103. 103. Sharded Queues • Declare Queues with name: nodename.queuename.index • Bind the queues to a consistent hash exchange
  104. 104. Sharded Queues • Declare Queues with name: nodename.queuename.index • Bind the queues to a partitioner exchange • Transparent to the consumer (virtual queue name)
  105. 105. We need more scale!
  106. 106. Federated Queues
  107. 107. Federated Queues • Load-balance messages across federated queues • Only moves messages when needed
  108. 108. Federating a Queue rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}'
  109. 109. Federating a Queue rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}' ! rabbitmqctl set_policy --apply-to queues federate-me "^images." '{"federation-upstream-set":"all"}'
  110. 110. With RabbitMQ we can
  111. 111. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP
  112. 112. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation
  113. 113. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation • Scale up using Sharding
  114. 114. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation • Scale up using Sharding • Load balance consumers with Federated Queues
  115. 115. Credits world map: wikipedia.org federation diagrams: rabbitmq.com
  116. 116. Questions?
  117. 117. Thanks AlvaroVidela - @old_sound
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×