1.
Building a Distributed
Data Ingestion System
with RabbitMQ
Alvaro Videla - RabbitMQ

2.
Alvaro Videla
•
•
•
•
•
Developer Advocate at Pivotal / RabbitMQ!
Co-Author of RabbitMQ in Action!
Creator of the RabbitMQ Simulator!
Blogs about RabbitMQ Internals: http://videlalvaro.github.io/internals.html!
@old_sound — alvaro@rabbitmq.com — github.com/videlalvaro

3.
About Me
Co-authored!
!
RabbitMQ in Action!
http://bit.ly/rabbitmq

4.
About this Talk
•
Exploratory Talk
•
A ‘what could be done’ talk instead of ‘this is how you do it’

5.
Agenda
•
Intro to RabbitMQ
•
The Problem
•
Solution Proposal
•
Improvements

6.
What is RabbitMQ

7.
RabbitMQ

8.
RabbitMQ

9.
RabbitMQ
• Multi Protocol Messaging Server

10.
RabbitMQ
• Multi Protocol Messaging Server!
• Open Source (MPL)

11.
RabbitMQ
• Multi Protocol Messaging Server!
• Open Source (MPL)!
• Polyglot

12.
RabbitMQ
• Multi Protocol Messaging Server!
• Open Source (MPL)!
• Polyglot!
• Written in Erlang/OTP

13.
Multi Protocol
http://bit.ly/rmq-protocols

14.
Polyglot

15.
Polyglot

16.
Polyglot
• Java

17.
Polyglot
• Java!
• node.js

18.
Polyglot
• Java!
• node.js!
• Erlang

19.
Polyglot
• Java!
• node.js!
• Erlang!
• PHP

20.
Polyglot
• Java!
• node.js!
• Erlang!
• PHP!
• Ruby

21.
Polyglot
• Java!
• node.js!
• Erlang!
• PHP!
• Ruby!
• .Net

22.
Polyglot
• Java!
• node.js!
• Erlang!
• PHP!
• Ruby!
• .Net!
• Haskell

23.
Polyglot
Even COBOL!!!11

24.
Some users of RabbitMQ

25.
Some users of RabbitMQ
•
Instagram

26.
Some users of RabbitMQ
•
•
Instagram!
Indeed.com

27.
Some users of RabbitMQ
•
•
•
Instagram!
Indeed.com!
Telefonica

28.
Some users of RabbitMQ
•
•
•
•
Instagram!
Indeed.com!
Telefonica!
Mercado Libre

29.
Some users of RabbitMQ
•
•
•
•
•
Instagram!
Indeed.com!
Telefonica!
Mercado Libre!
NHS

30.
Some users of RabbitMQ
•
•
•
•
•
•
Instagram!
Indeed.com!
Telefonica!
Mercado Libre!
NHS!
Mozilla

31.
http://www.rabbitmq.com/download.html
Unix - Mac - Windows

32.
Messaging with RabbitMQ
A demo with the RabbitMQ Simulator
https://github.com/RabbitMQSimulator/RabbitMQSimulator

33.
http://tryrabbitmq.com

34.
RabbitMQ Simulator

35.
The Problem

36.
Distributed Application
App
App
App
App

37.
Distributed Application
App
App
App
App

38.
Ad-hoc solution

39.
A process that replicates data
to the remote server

40.
Possible issues
•
Remote server is offline
•
•
•
Prevent unbounded local buffers
Prevent message loss
Prevent unnecessary message replication
•
No need for those messages on remote server
•
Messages that became stale

41.
Can we do better?

42.
RabbitMQ Federation

43.
RabbitMQ Federation
•
Supports replication across different administrative domains
•
Supports mix of Erlang and RabbitMQ versions
•
Supports Network Partitions
•
Specificity - not everything has to be federated

44.
RabbitMQ Federation

45.
RabbitMQ Federation

46.
RabbitMQ Federation

47.
RabbitMQ Federation
•
It’s a RabbitMQ Plugin

48.
RabbitMQ Federation
•
It’s a RabbitMQ Plugin
•
Internally uses Queues and Exchanges Decorators

49.
RabbitMQ Federation
•
It’s a RabbitMQ Plugin
•
Internally uses Queues and Exchanges Decorators
•
Managed using Parameters and Policies

50.
Enabling the Plugin
rabbitmq-plugins enable rabbitmq_federation

51.
Enabling the Plugin
rabbitmq-plugins enable rabbitmq_federation
rabbitmq-plugins enable rabbitmq_federation_management

52.
Federating an Exchange
rabbitmqctl set_parameter federation-upstream my-upstream
‘{“uri":"amqp://server-name","expires":3600000}'

53.
Federating an Exchange
rabbitmqctl set_parameter federation-upstream my-upstream
‘{“uri":"amqp://server-name","expires":3600000}'
!
rabbitmqctl set_policy --apply-to exchanges federate-me "^amq."
'{"federation-upstream-set":"all"}'

54.
Federating an Exchange

55.
Configuring Federation

56.
Config Options
rabbitmqctl set_parameter federation-upstream
name ‘json-object’

57.
Config Options
rabbitmqctl set_parameter federation-upstream
name ‘json-object’
!
json-object: {
‘uri’: ‘amqp://server-name/’,
‘prefetch-count’: 1000,
‘reconnect-delay’: 1,
‘ack-mode’: on-confirm
}
http://www.rabbitmq.com/federation-reference.html

58.
Prevent unbound buffers
expires: N // ms.
message-ttl: N // ms.

59.
Prevent message forwarding
max-hops: N

60.
Speed vs No Message Loss
ack-mode: on-confirm
ack-mode: on-publish
ack-mode: no-ack

61.
AMQP URI:
amqp://user:pass@host:10000/vhost
http://www.rabbitmq.com/uri-spec.html

62.
Config can be applied via
•
CLI using rabbitmqctl
•
HTTP API
•
RabbitMQ Management Interface

63.
RabbitMQ Federation

64.
Scaling the Setup

65.
The Problem

66.
The Problem
•
Queues contents live in the node where the Queue was declared

67.
The Problem
•
Queues contents live in the node where the Queue was declared
•
A cluster can access the queue from every connected node

68.
The Problem
•
Queues contents live in the node where the Queue was declared
•
A cluster can access the queue from every connected node
•
Queues are an Erlang process (tied to one core)

69.
The Problem
•
Queues contents live in the node where the Queue was declared
•
A cluster can access the queue from every connected node
•
Queues are an Erlang process (tied to one core)
•
Adding more nodes doesn’t really help

70.
Sharded Queues

71.
Pieces of the Puzzle
•
consistent hash exchange
•
good ol’ queues

72.
Sharded Queues

73.
Sharded Queues

74.
Sharded Queues

75.
Sharded Queues
•
Declare Queues with name: nodename.queuename.index

76.
Sharded Queues
•
Declare Queues with name: nodename.queuename.index
•
Bind the queues to a consistent hash exchange

77.
Sharded Queues
•
Declare Queues with name: nodename.queuename.index
•
Bind the queues to a consistent hash exchange
•
Get the consumer to randomly build the queue name

78.
We need more scale!

79.
Federated Queues

80.
Federated Queues
•
Load-balance messages across federated queues
•
Only moves messages when needed

81.
Federating a Queue
rabbitmqctl set_parameter federation-upstream my-upstream
‘{“uri":"amqp://server-name","expires":3600000}'

82.
Federating a Queue
rabbitmqctl set_parameter federation-upstream my-upstream
‘{“uri":"amqp://server-name","expires":3600000}'
!
rabbitmqctl set_policy --apply-to queues federate-me "^images."
'{"federation-upstream-set":"all"}'

83.
With RabbitMQ we can

84.
With RabbitMQ we can
•
Ingest data using various protocols: AMQP, MQTT and STOMP

85.
With RabbitMQ we can
•
Ingest data using various protocols: AMQP, MQTT and STOMP
•
Distribute that data globally using Federation

86.
With RabbitMQ we can
•
Ingest data using various protocols: AMQP, MQTT and STOMP
•
Distribute that data globally using Federation
•
Scale up using Sharding

87.
With RabbitMQ we can
•
Ingest data using various protocols: AMQP, MQTT and STOMP
•
Distribute that data globally using Federation
•
Scale up using Sharding
•
Load balance consumers with Federated Queues

88.
Credits
world map: wikipedia.org
federation diagrams: rabbitmq.com

89.
Questions?

90.
Thanks
Alvaro Videla - @old_sound