This document discusses building a distributed data ingestion system using RabbitMQ. It begins with an introduction to RabbitMQ and its key features like being multi-protocol, open source, polyglot and written in Erlang. It then discusses the problem of distributing data across multiple servers. RabbitMQ federation is proposed as a solution, allowing replication across servers. Federation uses queues and exchanges to replicate data and can be configured using parameters and policies. The document also discusses scaling the system using sharded queues and federated queues to load balance consumers.

1. Building a Distributed
Data Ingestion System
with RabbitMQ
Alvaro Videla - RabbitMQ

2. Alvaro Videla
• Developer Advocate at Pivotal / RabbitMQ!
• Co-Author of RabbitMQ in Action!
• Creator of the RabbitMQ Simulator!
• Blogs about RabbitMQ Internals: http://videlalvaro.github.io/internals.html!
• @old_sound — alvaro@rabbitmq.com — github.com/videlalvaro

3. About Me
Co-authored!
!
RabbitMQ in Action!
http://bit.ly/rabbitmq

4. About this Talk
• Exploratory Talk
• A ‘what could be done’ talk instead of ‘this is how you do it’

5. Agenda
• Intro to RabbitMQ
• The Problem
• Solution Proposal
• Improvements

6. Intermission

7. Intermission
History Lessons

8. The Hungarian Connection

9. The Hungarian Connection
• I’m from Uruguay

10. The Hungarian Connection
• I’m from Uruguay
• I live in Switzerland

11. The Hungarian Connection
• I’m from Uruguay
• I live in Switzerland
• I’m in Hungary right now

12. WHAT?

13. The Hungarian Connection

14. The Hungarian Connection
Switzerland World Cup - 1954

15. The Hungarian Connection
Switzerland World Cup - 1954

16. The Hungarian Connection
Switzerland World Cup - 1954
Hungary 4 - Uruguay 2

17. The Hungarian Connection II
Tivadar Puskás
Telephone Switch Inventor

18. The Hungarian Connection III
Old Hungarian Alphabet

19. The Hungarian Connection III
Erlang

20. The Hungarian Connection III
As Neumann János Lajos said

21. The Hungarian Connection III
As Neumann János Lajos said
Use Erlang

22. What is RabbitMQ

23. RabbitMQ

24. RabbitMQ

25. RabbitMQ
• Multi Protocol Messaging Server

26. • Multi Protocol Messaging Server!
• Open Source (MPL)
RabbitMQ

27. • Multi Protocol Messaging Server!
• Open Source (MPL)!
• Polyglot
RabbitMQ

28. • Multi Protocol Messaging Server!
• Open Source (MPL)!
• Polyglot!
• Written in Erlang/OTP
RabbitMQ

29. Multi Protocol
http://bit.ly/rmq-protocols

30. http://www.rabbitmq.com/community-plugins.html
Community Plugins

31. Polyglot

32. Polyglot

33. Polyglot
• Java

34. Polyglot
• Java!
• node.js

35. Polyglot
• Java!
• node.js!
• Erlang

36. Polyglot
• Java!
• node.js!
• Erlang!
• PHP

37. Polyglot
• Java!
• node.js!
• Erlang!
• PHP!
• Ruby

38. Polyglot
• Java!
• node.js!
• Erlang!
• PHP!
• Ruby!
• .Net

39. Polyglot
• Java!
• node.js!
• Erlang!
• PHP!
• Ruby!
• .Net!
• Haskell

40. Polyglot
Even COBOL!!!11

41. Some users of RabbitMQ

42. • Instagram
Some users of RabbitMQ

43. • Instagram!
• Indeed.com
Some users of RabbitMQ

44. • Instagram!
• Indeed.com!
• Telefonica
Some users of RabbitMQ

45. • Instagram!
• Indeed.com!
• Telefonica!
• Mercado Libre
Some users of RabbitMQ

46. • Instagram!
• Indeed.com!
• Telefonica!
• Mercado Libre!
• NHS
Some users of RabbitMQ

47. • Instagram!
• Indeed.com!
• Telefonica!
• Mercado Libre!
• NHS!
• Mozilla
Some users of RabbitMQ

48. The NewYork Times on RabbitMQ
This architecture - Fabrik - has dozens of RabbitMQ instances
spread across 6 AWS zones in Oregon and Dublin.
Upon launch today, the system autoscaled to ~500,000 users.
Connection times remained flat at ~200ms.
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2014-January/032943.html

49. http://www.rabbitmq.com/download.html
Unix - Mac - Windows

50. Messaging with RabbitMQ
A demo with the RabbitMQ Simulator
https://github.com/RabbitMQSimulator/RabbitMQSimulator

51. http://tryrabbitmq.com

52. RabbitMQ Simulator

53. The Problem

54. Distributed Application
App
App
App
App

55. Distributed Application
App
App
App
App

56. ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
!
Connection connection = factory.newConnection();
!
Channel channel = connection.createChannel();
Data Producer
Obtain a Channel

57. channel.exchangeDeclare(EXCHANGE_NAME, "direct", true);
Data Producer
Declare an Exchange

58. String message = "Hello Federation!";
channel.basicPublish(EXCHANGE_NAME, "", null,
message.getBytes());
Data Producer
Publish a message

59. ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
!
Connection connection = factory.newConnection();
!
Channel channel = connection.createChannel();
Data Consumer
Obtain a Channel

60. channel.queueDeclare(QUEUE_NAME, true, false, false,
null);
channel.exchangeDeclare(EXCHANGE_NAME, "direct", true);
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, "");
Data Consumer
Declare Queue and bind it

61. QueueingConsumer consumer = new
QueueingConsumer(channel);
channel.basicConsume(QUEUE_NAME, false, consumer);
Data Consumer
Start a consumer

62. while (true) {
QueueingConsumer.Delivery delivery =
consumer.nextDelivery();
String message = new String(delivery.getBody());
System.out.println("Received '" + message + "'");
!
channel.basicAck(
delivery.getEnvelope().
getDeliveryTag(), false);
}
Data Consumer
Process messages

63. Ad-hoc solution

64. A process that replicates data
to the remote server

65. Possible issues
• Remote server is offline
• Prevent unbounded local buffers
• Prevent message loss
• Prevent unnecessary message replication
• No need for those messages on remote server
• Messages that became stale

66. Can we do better?

67. RabbitMQ Federation

68. RabbitMQ Federation
• Supports replication across different administrative domains
• Supports mix of Erlang and RabbitMQ versions
• Supports Network Partitions
• Specificity - not everything has to be federated

69. RabbitMQ Federation

70. RabbitMQ Federation

71. RabbitMQ Federation

72. RabbitMQ Federation
• It’s a RabbitMQ Plugin

73. RabbitMQ Federation
• It’s a RabbitMQ Plugin
• Internally uses Queues and Exchanges Decorators

74. RabbitMQ Federation
• It’s a RabbitMQ Plugin
• Internally uses Queues and Exchanges Decorators
• Managed using Parameters and Policies

75. Enabling the Plugin
rabbitmq-plugins enable rabbitmq_federation

76. Enabling the Plugin
rabbitmq-plugins enable rabbitmq_federation
rabbitmq-plugins enable rabbitmq_federation_management

77. Federating an Exchange
rabbitmqctl set_parameter federation-upstream my-upstream
‘{“uri":"amqp://server-name","expires":3600000}'

78. Federating an Exchange
rabbitmqctl set_parameter federation-upstream my-upstream
‘{“uri":"amqp://server-name","expires":3600000}'
!
rabbitmqctl set_policy --apply-to exchanges federate-me "^amq."
'{"federation-upstream-set":"all"}'

79. Federating an Exchange

80. Configuring Federation

81. Config Options
rabbitmqctl set_parameter federation-upstream
name ‘json-object’

82. Config Options
rabbitmqctl set_parameter federation-upstream
name ‘json-object’
!
json-object: {
‘uri’: ‘amqp://server-name/’,
‘prefetch-count’: 1000,
‘reconnect-delay’: 1,
‘ack-mode’: on-confirm
}
http://www.rabbitmq.com/federation-reference.html

83. Prevent unbound buffers
expires: N // ms.
message-ttl: N // ms.

84. Prevent message forwarding
max-hops: N

85. Speed vs No Message Loss
ack-mode: on-confirm
ack-mode: on-publish
ack-mode: no-ack

86. AMQP URI:
amqp://user:pass@host:10000/vhost
http://www.rabbitmq.com/uri-spec.html

87. Config can be applied via
• CLI using rabbitmqctl
• HTTP API
• RabbitMQ Management Interface

88. RabbitMQ Federation

89. Scaling the Setup

90. The Problem

91. The Problem
• Queues contents live in the node where the Queue was declared

92. The Problem
• Queues contents live in the node where the Queue was declared
• A cluster can access the queue from every connected node

93. The Problem
• Queues contents live in the node where the Queue was declared
• A cluster can access the queue from every connected node
• Queues are an Erlang process (tied to one core)

94. The Problem
• Queues contents live in the node where the Queue was declared
• A cluster can access the queue from every connected node
• Queues are an Erlang process (tied to one core)
• Adding more nodes doesn’t really help

95. Enter Sharded Queues

96. Enter Sharded Queues

97. Pieces of the Puzzle
• modulo hash exchange (consistent hash works as well)
• good ol’ queues

98. Sharded Queues

99. Sharded Queues

100. Sharded Queues

101. Sharded Queues

102. Sharded Queues
• Declare Queues with name: nodename.queuename.index

103. Sharded Queues
• Declare Queues with name: nodename.queuename.index
• Bind the queues to a consistent hash exchange

104. Sharded Queues
• Declare Queues with name: nodename.queuename.index
• Bind the queues to a partitioner exchange
• Transparent to the consumer (virtual queue name)

105. We need more scale!

106. Federated Queues

107. Federated Queues
• Load-balance messages across federated queues
• Only moves messages when needed

108. Federating a Queue
rabbitmqctl set_parameter federation-upstream my-upstream
‘{“uri":"amqp://server-name","expires":3600000}'

109. Federating a Queue
rabbitmqctl set_parameter federation-upstream my-upstream
‘{“uri":"amqp://server-name","expires":3600000}'
!
rabbitmqctl set_policy --apply-to queues federate-me "^images."
'{"federation-upstream-set":"all"}'

110. With RabbitMQ we can

111. With RabbitMQ we can
• Ingest data using various protocols: AMQP, MQTT and STOMP

112. With RabbitMQ we can
• Ingest data using various protocols: AMQP, MQTT and STOMP
• Distribute that data globally using Federation

113. With RabbitMQ we can
• Ingest data using various protocols: AMQP, MQTT and STOMP
• Distribute that data globally using Federation
• Scale up using Sharding

114. With RabbitMQ we can
• Ingest data using various protocols: AMQP, MQTT and STOMP
• Distribute that data globally using Federation
• Scale up using Sharding
• Load balance consumers with Federated Queues

115. Credits
world map: wikipedia.org
federation diagrams: rabbitmq.com

116. Questions?

117. Thanks
AlvaroVidela - @old_sound