RabbitMQ Data Ingestion at Craft Conf

2,867 views

Published on

Presentation about RabbitMQ and Federation

Published in: Technology

RabbitMQ Data Ingestion at Craft Conf

  1. 1. Building a Distributed Data Ingestion System with RabbitMQ Alvaro Videla - RabbitMQ
  2. 2. Alvaro Videla • Developer Advocate at Pivotal / RabbitMQ! • Co-Author of RabbitMQ in Action! • Creator of the RabbitMQ Simulator! • Blogs about RabbitMQ Internals: http://videlalvaro.github.io/internals.html! • @old_sound — alvaro@rabbitmq.com — github.com/videlalvaro

  3. 3. About Me Co-authored! ! RabbitMQ in Action! http://bit.ly/rabbitmq
  4. 4. About this Talk • Exploratory Talk • A ‘what could be done’ talk instead of ‘this is how you do it’
  5. 5. Agenda • Intro to RabbitMQ • The Problem • Solution Proposal • Improvements
  6. 6. Intermission
  7. 7. Intermission History Lessons
  8. 8. The Hungarian Connection
  9. 9. The Hungarian Connection • I’m from Uruguay
  10. 10. The Hungarian Connection • I’m from Uruguay • I live in Switzerland
  11. 11. The Hungarian Connection • I’m from Uruguay • I live in Switzerland • I’m in Hungary right now
  12. 12. WHAT?
  13. 13. The Hungarian Connection
  14. 14. The Hungarian Connection Switzerland World Cup - 1954
  15. 15. The Hungarian Connection Switzerland World Cup - 1954
  16. 16. The Hungarian Connection Switzerland World Cup - 1954 Hungary 4 - Uruguay 2
  17. 17. The Hungarian Connection II Tivadar Puskás Telephone Switch Inventor
  18. 18. The Hungarian Connection III Old Hungarian Alphabet
  19. 19. The Hungarian Connection III Erlang
  20. 20. The Hungarian Connection III As Neumann János Lajos said
  21. 21. The Hungarian Connection III As Neumann János Lajos said Use Erlang
  22. 22. What is RabbitMQ
  23. 23. RabbitMQ
  24. 24. RabbitMQ
  25. 25. RabbitMQ • Multi Protocol Messaging Server
  26. 26. • Multi Protocol Messaging Server! • Open Source (MPL) RabbitMQ
  27. 27. • Multi Protocol Messaging Server! • Open Source (MPL)! • Polyglot RabbitMQ
  28. 28. • Multi Protocol Messaging Server! • Open Source (MPL)! • Polyglot! • Written in Erlang/OTP RabbitMQ
  29. 29. Multi Protocol http://bit.ly/rmq-protocols
  30. 30. http://www.rabbitmq.com/community-plugins.html Community Plugins
  31. 31. Polyglot
  32. 32. Polyglot
  33. 33. Polyglot • Java
  34. 34. Polyglot • Java! • node.js
  35. 35. Polyglot • Java! • node.js! • Erlang
  36. 36. Polyglot • Java! • node.js! • Erlang! • PHP
  37. 37. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby
  38. 38. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby! • .Net
  39. 39. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby! • .Net! • Haskell
  40. 40. Polyglot Even COBOL!!!11
  41. 41. Some users of RabbitMQ
  42. 42. • Instagram Some users of RabbitMQ
  43. 43. • Instagram! • Indeed.com Some users of RabbitMQ
  44. 44. • Instagram! • Indeed.com! • Telefonica Some users of RabbitMQ
  45. 45. • Instagram! • Indeed.com! • Telefonica! • Mercado Libre Some users of RabbitMQ
  46. 46. • Instagram! • Indeed.com! • Telefonica! • Mercado Libre! • NHS Some users of RabbitMQ
  47. 47. • Instagram! • Indeed.com! • Telefonica! • Mercado Libre! • NHS! • Mozilla Some users of RabbitMQ
  48. 48. The NewYork Times on RabbitMQ This architecture - Fabrik - has dozens of RabbitMQ instances spread across 6 AWS zones in Oregon and Dublin. Upon launch today, the system autoscaled to ~500,000 users. Connection times remained flat at ~200ms. http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2014-January/032943.html
  49. 49. http://www.rabbitmq.com/download.html Unix - Mac - Windows
  50. 50. Messaging with RabbitMQ A demo with the RabbitMQ Simulator https://github.com/RabbitMQSimulator/RabbitMQSimulator
  51. 51. http://tryrabbitmq.com
  52. 52. RabbitMQ Simulator
  53. 53. The Problem
  54. 54. Distributed Application App App App App
  55. 55. Distributed Application App App App App
  56. 56. ConnectionFactory factory = new ConnectionFactory(); factory.setHost("localhost"); ! Connection connection = factory.newConnection(); ! Channel channel = connection.createChannel(); Data Producer Obtain a Channel
  57. 57. channel.exchangeDeclare(EXCHANGE_NAME, "direct", true); Data Producer Declare an Exchange
  58. 58. String message = "Hello Federation!"; channel.basicPublish(EXCHANGE_NAME, "", null, message.getBytes()); Data Producer Publish a message
  59. 59. ConnectionFactory factory = new ConnectionFactory(); factory.setHost("localhost"); ! Connection connection = factory.newConnection(); ! Channel channel = connection.createChannel(); Data Consumer Obtain a Channel
  60. 60. channel.queueDeclare(QUEUE_NAME, true, false, false, null); channel.exchangeDeclare(EXCHANGE_NAME, "direct", true); channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, ""); Data Consumer Declare Queue and bind it
  61. 61. QueueingConsumer consumer = new QueueingConsumer(channel); channel.basicConsume(QUEUE_NAME, false, consumer); Data Consumer Start a consumer
  62. 62. while (true) { QueueingConsumer.Delivery delivery = consumer.nextDelivery(); String message = new String(delivery.getBody()); System.out.println("Received '" + message + "'"); ! channel.basicAck( delivery.getEnvelope(). getDeliveryTag(), false); } Data Consumer Process messages
  63. 63. Ad-hoc solution
  64. 64. A process that replicates data to the remote server
  65. 65. Possible issues • Remote server is offline • Prevent unbounded local buffers • Prevent message loss • Prevent unnecessary message replication • No need for those messages on remote server • Messages that became stale
  66. 66. Can we do better?
  67. 67. RabbitMQ Federation
  68. 68. RabbitMQ Federation • Supports replication across different administrative domains • Supports mix of Erlang and RabbitMQ versions • Supports Network Partitions • Specificity - not everything has to be federated
  69. 69. RabbitMQ Federation
  70. 70. RabbitMQ Federation
  71. 71. RabbitMQ Federation
  72. 72. RabbitMQ Federation • It’s a RabbitMQ Plugin
  73. 73. RabbitMQ Federation • It’s a RabbitMQ Plugin • Internally uses Queues and Exchanges Decorators
  74. 74. RabbitMQ Federation • It’s a RabbitMQ Plugin • Internally uses Queues and Exchanges Decorators • Managed using Parameters and Policies
  75. 75. Enabling the Plugin rabbitmq-plugins enable rabbitmq_federation
  76. 76. Enabling the Plugin rabbitmq-plugins enable rabbitmq_federation rabbitmq-plugins enable rabbitmq_federation_management
  77. 77. Federating an Exchange rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}'
  78. 78. Federating an Exchange rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}' ! rabbitmqctl set_policy --apply-to exchanges federate-me "^amq." '{"federation-upstream-set":"all"}'
  79. 79. Federating an Exchange
  80. 80. Configuring Federation
  81. 81. Config Options rabbitmqctl set_parameter federation-upstream name ‘json-object’
  82. 82. Config Options rabbitmqctl set_parameter federation-upstream name ‘json-object’ ! json-object: { ‘uri’: ‘amqp://server-name/’, ‘prefetch-count’: 1000, ‘reconnect-delay’: 1, ‘ack-mode’: on-confirm } http://www.rabbitmq.com/federation-reference.html
  83. 83. Prevent unbound buffers expires: N // ms. message-ttl: N // ms.
  84. 84. Prevent message forwarding max-hops: N
  85. 85. Speed vs No Message Loss ack-mode: on-confirm ack-mode: on-publish ack-mode: no-ack
  86. 86. AMQP URI: amqp://user:pass@host:10000/vhost http://www.rabbitmq.com/uri-spec.html
  87. 87. Config can be applied via • CLI using rabbitmqctl • HTTP API • RabbitMQ Management Interface
  88. 88. RabbitMQ Federation
  89. 89. Scaling the Setup
  90. 90. The Problem
  91. 91. The Problem • Queues contents live in the node where the Queue was declared
  92. 92. The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every connected node
  93. 93. The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every connected node • Queues are an Erlang process (tied to one core)
  94. 94. The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every connected node • Queues are an Erlang process (tied to one core) • Adding more nodes doesn’t really help
  95. 95. Enter Sharded Queues
  96. 96. Enter Sharded Queues
  97. 97. Pieces of the Puzzle • modulo hash exchange (consistent hash works as well) • good ol’ queues
  98. 98. Sharded Queues
  99. 99. Sharded Queues
  100. 100. Sharded Queues
  101. 101. Sharded Queues
  102. 102. Sharded Queues • Declare Queues with name: nodename.queuename.index
  103. 103. Sharded Queues • Declare Queues with name: nodename.queuename.index • Bind the queues to a consistent hash exchange
  104. 104. Sharded Queues • Declare Queues with name: nodename.queuename.index • Bind the queues to a partitioner exchange • Transparent to the consumer (virtual queue name)
  105. 105. We need more scale!
  106. 106. Federated Queues
  107. 107. Federated Queues • Load-balance messages across federated queues • Only moves messages when needed
  108. 108. Federating a Queue rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}'
  109. 109. Federating a Queue rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}' ! rabbitmqctl set_policy --apply-to queues federate-me "^images." '{"federation-upstream-set":"all"}'
  110. 110. With RabbitMQ we can
  111. 111. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP
  112. 112. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation
  113. 113. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation • Scale up using Sharding
  114. 114. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation • Scale up using Sharding • Load balance consumers with Federated Queues
  115. 115. Credits world map: wikipedia.org federation diagrams: rabbitmq.com
  116. 116. Questions?
  117. 117. Thanks AlvaroVidela - @old_sound

×