1. Building a Distributed
Data Ingestion System
with RabbitMQ
Alvaro Videla - RabbitMQ

2. Alvaro Videla
•
•
•
•
•
Developer Advocate at Pivotal / RabbitMQ!
Co-Author of RabbitMQ in Action!
Creator of the RabbitMQ Simulator!
Blogs about RabbitMQ Internals: http://videlalvaro.github.io/internals.html!
@old_sound — alvaro@rabbitmq.com — github.com/videlalvaro

3. About Me
Co-authored!
!
RabbitMQ in Action!
http://bit.ly/rabbitmq

4. About this Talk
•
Exploratory Talk
•
A ‘what could be done’ talk instead of ‘this is how you do it’

5. Agenda
•
Intro to RabbitMQ
•
The Problem
•
Solution Proposal
•
Improvements

6. What is RabbitMQ

7. RabbitMQ

8. RabbitMQ

9. RabbitMQ
• Multi Protocol Messaging Server

10. RabbitMQ
• Multi Protocol Messaging Server!
• Open Source (MPL)

11. RabbitMQ
• Multi Protocol Messaging Server!
• Open Source (MPL)!
• Polyglot

12. RabbitMQ
• Multi Protocol Messaging Server!
• Open Source (MPL)!
• Polyglot!
• Written in Erlang/OTP

13. Multi Protocol
http://bit.ly/rmq-protocols

14. Polyglot

15. Polyglot

16. Polyglot
• Java

17. Polyglot
• Java!
• node.js

18. Polyglot
• Java!
• node.js!
• Erlang

19. Polyglot
• Java!
• node.js!
• Erlang!
• PHP

20. Polyglot
• Java!
• node.js!
• Erlang!
• PHP!
• Ruby

21. Polyglot
• Java!
• node.js!
• Erlang!
• PHP!
• Ruby!
• .Net

22. Polyglot
• Java!
• node.js!
• Erlang!
• PHP!
• Ruby!
• .Net!
• Haskell

23. Polyglot
Even COBOL!!!11

24. Some users of RabbitMQ

25. Some users of RabbitMQ
•
Instagram

26. Some users of RabbitMQ
•
•
Instagram!
Indeed.com

27. Some users of RabbitMQ
•
•
•
Instagram!
Indeed.com!
Telefonica

28. Some users of RabbitMQ
•
•
•
•
Instagram!
Indeed.com!
Telefonica!
Mercado Libre

29. Some users of RabbitMQ
•
•
•
•
•
Instagram!
Indeed.com!
Telefonica!
Mercado Libre!
NHS

30. Some users of RabbitMQ
•
•
•
•
•
•
Instagram!
Indeed.com!
Telefonica!
Mercado Libre!
NHS!
Mozilla

31. http://www.rabbitmq.com/download.html
Unix - Mac - Windows

32. Messaging with RabbitMQ
A demo with the RabbitMQ Simulator
https://github.com/RabbitMQSimulator/RabbitMQSimulator

33. http://tryrabbitmq.com

34. RabbitMQ Simulator

35. The Problem

36. Distributed Application
App
App
App
App

37. Distributed Application
App
App
App
App

38. Ad-hoc solution

39. A process that replicates data
to the remote server

40. Possible issues
•
Remote server is offline
•
•
•
Prevent unbounded local buffers
Prevent message loss
Prevent unnecessary message replication
•
No need for those messages on remote server
•
Messages that became stale

41. Can we do better?

42. RabbitMQ Federation

43. RabbitMQ Federation
•
Supports replication across different administrative domains
•
Supports mix of Erlang and RabbitMQ versions
•
Supports Network Partitions
•
Specificity - not everything has to be federated

44. RabbitMQ Federation

45. RabbitMQ Federation

46. RabbitMQ Federation

47. RabbitMQ Federation
•
It’s a RabbitMQ Plugin

48. RabbitMQ Federation
•
It’s a RabbitMQ Plugin
•
Internally uses Queues and Exchanges Decorators

49. RabbitMQ Federation
•
It’s a RabbitMQ Plugin
•
Internally uses Queues and Exchanges Decorators
•
Managed using Parameters and Policies

50. Enabling the Plugin
rabbitmq-plugins enable rabbitmq_federation

51. Enabling the Plugin
rabbitmq-plugins enable rabbitmq_federation
rabbitmq-plugins enable rabbitmq_federation_management

52. Federating an Exchange
rabbitmqctl set_parameter federation-upstream my-upstream
‘{“uri":"amqp://server-name","expires":3600000}'

53. Federating an Exchange
rabbitmqctl set_parameter federation-upstream my-upstream
‘{“uri":"amqp://server-name","expires":3600000}'
!
rabbitmqctl set_policy --apply-to exchanges federate-me "^amq."
'{"federation-upstream-set":"all"}'

54. Federating an Exchange

55. Configuring Federation

56. Config Options
rabbitmqctl set_parameter federation-upstream
name ‘json-object’

57. Config Options
rabbitmqctl set_parameter federation-upstream
name ‘json-object’
!
json-object: {
‘uri’: ‘amqp://server-name/’,
‘prefetch-count’: 1000,
‘reconnect-delay’: 1,
‘ack-mode’: on-confirm
}
http://www.rabbitmq.com/federation-reference.html

58. Prevent unbound buffers
expires: N // ms.
message-ttl: N // ms.

59. Prevent message forwarding
max-hops: N

60. Speed vs No Message Loss
ack-mode: on-confirm
ack-mode: on-publish
ack-mode: no-ack

61. AMQP URI:
amqp://user:pass@host:10000/vhost
http://www.rabbitmq.com/uri-spec.html

62. Config can be applied via
•
CLI using rabbitmqctl
•
HTTP API
•
RabbitMQ Management Interface

63. RabbitMQ Federation

64. Scaling the Setup

65. The Problem

66. The Problem
•
Queues contents live in the node where the Queue was declared

67. The Problem
•
Queues contents live in the node where the Queue was declared
•
A cluster can access the queue from every connected node

68. The Problem
•
Queues contents live in the node where the Queue was declared
•
A cluster can access the queue from every connected node
•
Queues are an Erlang process (tied to one core)

69. The Problem
•
Queues contents live in the node where the Queue was declared
•
A cluster can access the queue from every connected node
•
Queues are an Erlang process (tied to one core)
•
Adding more nodes doesn’t really help

70. Sharded Queues

71. Pieces of the Puzzle
•
consistent hash exchange
•
good ol’ queues

72. Sharded Queues

73. Sharded Queues

74. Sharded Queues

75. Sharded Queues
•
Declare Queues with name: nodename.queuename.index

76. Sharded Queues
•
Declare Queues with name: nodename.queuename.index
•
Bind the queues to a consistent hash exchange

77. Sharded Queues
•
Declare Queues with name: nodename.queuename.index
•
Bind the queues to a consistent hash exchange
•
Get the consumer to randomly build the queue name

78. We need more scale!

79. Federated Queues

80. Federated Queues
•
Load-balance messages across federated queues
•
Only moves messages when needed

81. Federating a Queue
rabbitmqctl set_parameter federation-upstream my-upstream
‘{“uri":"amqp://server-name","expires":3600000}'

82. Federating a Queue
rabbitmqctl set_parameter federation-upstream my-upstream
‘{“uri":"amqp://server-name","expires":3600000}'
!
rabbitmqctl set_policy --apply-to queues federate-me "^images."
'{"federation-upstream-set":"all"}'

83. With RabbitMQ we can

84. With RabbitMQ we can
•
Ingest data using various protocols: AMQP, MQTT and STOMP

85. With RabbitMQ we can
•
Ingest data using various protocols: AMQP, MQTT and STOMP
•
Distribute that data globally using Federation

86. With RabbitMQ we can
•
Ingest data using various protocols: AMQP, MQTT and STOMP
•
Distribute that data globally using Federation
•
Scale up using Sharding

87. With RabbitMQ we can
•
Ingest data using various protocols: AMQP, MQTT and STOMP
•
Distribute that data globally using Federation
•
Scale up using Sharding
•
Load balance consumers with Federated Queues

88. Credits
world map: wikipedia.org
federation diagrams: rabbitmq.com

89. Questions?

90. Thanks
Alvaro Videla - @old_sound