Scaling application
with RabbitMQ
Nahidul Kibria
Co-Leader, OWASP Bangladesh,
Principal Software Engineer,
Orbitax Bangladesh Ltd.
@nahidupa
What is RabbitMQ?
We See Code
Clustering and High Availability
Docker+RabbitMQ
I will stop on some checkpoint and take question.
Birds eye view.
Checkpoint sign
Demo sign
What is RabbitMQ ?
http://goo.gl/gEDUUo
Polyglot
Java and JVM
Ruby
Python
.NET
PHP
Perl
C / C++
Node.js
Go
•Erlang
•Haskell
•Provisioning (Chef, Puppet, etc)
and Operations
•Mobile
•Database Integration
•Web Messaging
•CLI
•3rd party
plugins
•Ocaml
•Common Lisp
•COBOL
Scaling
Scaling UP vs S c a leO UT
Why do we need messaging?
We are about to get a new project give me a prototype
will be simple
Product owner
come…Notify friends
We need to post in
twitter
First delivery
It take too much
bandwidth…lablabla..
I actually need that yesterday
We are going to third-
party integration
PHP
Python
Java
UX guys, Customer
feedback
we don’t want to wait
We need to build instagram!
Poor you (:
You play boss killer some time and get back to work
All of this the simple solution is
messaging
Simple message queue
Who are using RabbitMQ?
It is used in the Ocean Observatories Initiative—an
architecture that collects 8 terabytes of data per day.
Who are using RabbitMQ?
Instagram
Indeed.com
Telefonica
Mercado Libre
NHS
Mozilla
Install RabbitMQ
Install in *nix/debian
After Install
Install Management portal for
RabbitMQ
Demo time
Hello word
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare("hello", false, false, false, null);
string message = "Hello World!";
var body = Encoding.UTF8.GetBytes(message);
channel.BasicPublish("", "hello", null, body);
Console.WriteLine(" [x] Sent {0}", message);
}
}
Queue Declare
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare("hello", false, false, false, null);
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume("hello", true, consumer);
Console.WriteLine(" [*] Waiting for messages." +
"To exit press CTRL+C");
while (true)
{
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
}
}
}
Exchange
Coding
Work Queues
Exchanges
Routing
Direct exchange
Multiple bindings
Log
Topic exchange
Remote procedure call
(RPC)
easynetq
https://www.rabbitmq.com/devtools.html
Easynetq save your time
//many simplification
var bus = RabbitHutch.CreateBus("host=ubuntu:5672,ubuntu:5673");
//also give freedom
var advancedBus = RabbitHutch.CreateBus("host=localhost").Advanced
RabbitMQ Features
RabbitMQ Features
Large Message
Messaging in general should be small
AMPQ support very large messages
RabbitMQ you can use large messages but you need to be careful
MSMQ-Message Queuing is limited
in the size of documents that can
be processed. The maximum size
document that Message Queuing
supports is 4 MB if the document
is in ASCII. The message queue
can handle a maximum of 2 MB if
the document is in Unicode.
Buffered and chunked
http://goo.gl/SgnBYf
Virtual Hosts
Error Handling
//no retry
var ea = (BasicDeliverEventArgs) consumer.Queue.Dequeue();
channel.BasicReject(ea.DeliveryTag,false);
//retry
var ea = (BasicDeliverEventArgs) consumer.Queue.Dequeue();
channel.BasicReject(ea.DeliveryTag,true);
We cannot retry forever!
channel.BasicAck(ea.DeliveryTag,false);
ea.BasicProperties.Headers.Add("RetryHeader",1);
channel.BasicNack(ea.DeliveryTag, false,true);
Dead letter queue
feelsleepy
Clustering and High Availability
Demo Environment
Sync Erlang cookie
10.10.52.130 debian
If erlang version mismatch
Join cluster
Demo in management
Details http://goo.gl/BqQHjp
???
Breaking up a cluster
rabbit3$ rabbitmqctl stop_app
Stopping node rabbit@rabbit3 ...done.
rabbit3$ rabbitmqctl reset Resetting node
rabbit@rabbit3 ...done.
rabbit3$ rabbitmqctl start_app
Starting node rabbit@rabbit3 ...done.
Distributed Application
Shovel plugin
Loose coupling
they may have different users and virtual hosts;
they may run on different versions of RabbitMQ and
Erlang.
A process that replicates data
to the remote server
Cluster/LAN Cluster/LAN
Shovel
WAN friendly
rabbitmq-plugins enable rabbitmq_shovel
rabbitmq-plugins enable rabbitmq_shovel_management
Demo time
RabbitMQ Federation
Federation Plugin
Loose coupling
WAN-friendly
Specificity
Scalability
Federation Plugin
RabbitMQ Federation
•Supports replication across different administrative
domains
•Supports mix of Erlang and RabbitMQ versions
•Supports Network Partitions
•Specificity - not everything has to be federated
Federated Exchanges
Topologies
Topologies
Small complete graph
Topologies
Fan-out
Topologies
Ring
Federated Queues
rabbitmqctl set_parameter federation-upstream my-upstream
'{"uri":"amqp://admin:admin@192.168.52.133","expires":36000
00}‘
rabbitmqctl set_policy federate-me '^DigitalWorld.'
'{"federation-upstream-set":"all"}' --priority 0 --apply-to
exchanges
Enterprise Integration Patterns for RabbitMQ
Connecting with SSL
Authentication and Authorization
Security
Monitoring RabbitMQ
https://github.com/jamesc/nagios-plugins-rabbitmq
http://newrelic.com/plugins/pivotal/95
https://www.youtube.com/watch?v=CAak2ayFc
https://github.com/zenoss/ZenPacks.zenoss.RabbitMQ
http://looselycoupledlabs.com/2014/08/monitoring-
rabbitmq/
http://www.rabbitmq.com/how.html have a section
Memory Alarms
Disk Alarms
https://www.rabbitmq.com/alarms.html
Docker+RabbitMQ
http://www.fig.sh/install.html
Load Test jMeter
Apache JMeter and the AMQP plugin to run load tests:
https://github.com/jlavallee/JMeter-Rabbit-AMQP
Real Time
Socket.IO/SignalR
http://goo.gl/ObvblZ
SignalR
http://goo.gl/xumPOG
A client that is
connected to one
server will not
receive messages
sent from another
server.
http://goo.gl/79T7aZ
RabbitMQ
RabbitMQ is extremely efficient,
widely deployed & tested
message broker but can hit
performance, if tuned for more
durability and reliability. It’s
more targeted towards serving
enterprise messaging with
advanced routing requirements.
Redis is quick to start,
lightweight & fast broker but
does Not support reliable
delivery Hence can be chosen
for applications where in case
system terminates, losing the
information about tasks for a
few minutes is Not critical.
http://goo.gl/J5ak0j
References
http://jmcle.github.io/rabbitmq-visualizer/
https://github.com/RabbitMQSimulator/RabbitMQSimulator
http://java.dzone.com/articles/docker-rabbitmq-cluster
Questions
@nahidupa

Scaling application with RabbitMQ

Editor's Notes

  • #5 Ask audience Any bus lover
  • #8 Advanced Message Queuing Protocol Message Queue Telemetry Transport—IBM-- binary packet payload--low bandwidth, high latency networks such as dial up lines and satellite links Simple/Streaming Text Oriented Messaging Protocol--browser, mobile app, or machine in real-time Publish from a protocol consume from another protocol
  • #10 Single point of failure
  • #24 Do not start showing other pattern
  • #28 sysv-rc-conf
  • #29 nano /etc/apt/sources.list deb http://www.rabbitmq.com/debian/ testing main wget https://www.rabbitmq.com/rabbitmq-signing-key-public.asc sudo apt-key add rabbitmq-signing-key-public.asc apt-get update sudo apt-get install rabbitmq-server
  • #30 update-rc.d -f rabbitmq-server remove update-rc.d -f rabbitmq-server defaults http://localhost:15672/ Sudo invoke-rc.d rabbitmq-server stop/start/etc apt-get install chkconfig chkconfig –list chkconfig rabbitmq-server on
  • #31 C:\Program Files (x86)\RabbitMQ Server\rabbitmq_server-3.4.3\sbin rabbitmq-plugins enable rabbitmq_management "C:\Program Files (x86)\RabbitMQ Server\rabbitmq_server-3.4.3\sbin" "rabbitmq-plugins enable rabbitmq_management“
  • #32 Show management console
  • #39 Again go to simulator
  • #45 routekey
  • #53 mnesia
  • #55 Message Queuing is limited in the size of documents that can be processed. The maximum size document that Message Queuing supports is 4 MB if the document is in ASCII. The message queue can handle a maximum of 2 MB if the document is in Unicode.
  • #57 /debug /production demo
  • #64  C:\Program Files (x86)\RabbitMQ Server\rabbitmq_server-3.4.3\sbin>rabbitmqctl jo in_cluster --ram rabbit@exdev Clustering node rabbit@JINOM with rabbit@exdev ... Error: {inconsistent_cluster,"OTP version mismatch: local node is 17.4, remote n ode R15B01"} Check erlang verssion erl -eval 'erlang:display(erlang:system_info(otp_release)), halt().' –noshell in_cluster --ram rabbit@exdev Clustering node rabbit@JINOM with rabbit@exdev ...
  • #66 On Unix systems, the cookie will be typically located in /var/lib/rabbitmq/.erlang.cookie or$HOME/.erlang.cookie. On Windows, the locations are C:\Users\Current User\.erlang.cookie(%HOMEDRIVE% + %HOMEPATH%\.erlang.cookie) orC:\Documents and Settings\Current User\.erlang.cookie, and C:\Windows\.erlang.cookie for RabbitMQ Windows service. If Windows service is used, the cookie should be placed in both places. As an alternative, you can insert the option "-setcookie cookie" in the erl call in the rabbitmq-serverand rabbitmqctl scripts.
  • #77 demo
  • #80 https://www.rabbitmq.com/federation.html rabbitmq-plugins enable rabbitmq_federation rabbitmq-plugins enable rabbitmq_federation_management
  • #89 demo
  • #92 https://www.youtube.com/watch?v=CAak2ayFcV0 https://github.com/zenoss/ZenPacks.zenoss.RabbitMQ http://looselycoupledlabs.com/2014/08/monitoring-rabbitmq/ http://www.rabbitmq.com/how.html have a section
  • #94 curl -L https://github.com/docker/fig/releases/download/1.0.1/fig-`uname -s`-`uname -m` > /usr/local/bin/fig; chmod +x /usr/local/bin/fig http://java.dzone.com/articles/docker-rabbitmq-cluster
  • #96 https://github.com/rajaraodv/rabbitpubsub