RabbitMQ is used in several example applications and real case studies. Three example codes demonstrate: 1) the RabbitMQ tutorial using request/reply queues, 2) RPC using correlated messages, and 3) Celery's distributed task queue using polling. Case studies show RabbitMQ for: 1) message passing between Zenoss daemons, 2) Nova controller RPC, 3) Instagram's task queue with Celery, 4) MercadoLibre's service bus architecture, and 5) Indeed.com's distributed crawling. RabbitMQ provides reliability, scalability, and asynchronous messaging between distributed applications and microservices.

1. RabbitMQ Model and Some
Example Applications
houcheng@gmail.com
CCMA ITRI

2. Outline
•
•
•
•
RabbitMQ model and standard
Simplified message queue model
3 example code
5 real case studies

3. RabbitMQ Model
s1
C1
tmp1: private
(Distribution)
s2
direct
exchange
C2
q1: ! private
key: q1
reply-to: tmp1
cor-id: 1
s3
key1
q3:
fan-out
key1
q4:
to: fan-out
key:key1
topic
exchange
C3
STK.US.*
STK.IND.*
STK.US.* AND
type == pdf
to: topic
key: STK.US.C
filter
exchange
q5:
!durable
q6:
s4
(Fan-out)

4. Notations in RabbitMQ Model
s4
application
thread
virtual
host
TCP
connection
exchange
channel
binding
message
queue
consuming

5. RabbitMQ Model
• Virtual Host
• Exchange
– direct
– fan-out
– topic
• Binding
– topic
– cascading
– message select
• queue
– flags: private, durable
• Connection
– channel: every thread work
with one channel
• Message
– content header
• Properties: Reply-To, Cor-Id,
Message-Id, Key
• queue server may add
properties , wont remove/
modify
– content body (wont modify)
• binary/ file/ stream
• Application
– client/ server

6. Other Items in AMQP 0.8,
not included in Model
– AMQP transaction – deliver ACK to sender after
message received
– AMQP distributed transaction – deliver ACK to a
3rd party software after message received
– Access – a realm based access control model
we think these are irrelevant to ADDM
transaction detection

7. Simplified Message Queue Model
C1
s1
send rpc:q1.1
rpc
q1.*
s2
reply-to: tmp1
cor-id: (2)
q2.*
tmp1
C2
rpcresult
send update:q34
s3
update
q34
cor-id: 2
(Distribution)
s4
s5
rpc
q5
(Fan out)
s6
q5
queue server

8. Model Notation
C1
queue server
application thread
$name
exchange
($destination: key)
message
$key
$properties
binding
queue
consuming

9. 3 Example Code
1. RabbitMQ tutorial code (synchronous)
2. RabbitMQ RPC (synchronous)
3. Celery framework (distributed task queue /
asynchronous)

10. Classical Scenario
in Example Code
– one request queue per service; one reply queue per client
– properties used on message
• correlate-id
• message-id
• reply-to
– steps
1.
2.
3.
4.
client thread send REQUEST and wait on reply queue
server thread receive REQUEST, process it and send REPLY to reply queue
client thread get reply from reply queue
if asynchronous mode, client may polls processing status by another status
update REQUEST message, and server reply another status update REPLY
message

11. Example Code 1:
RabbitMQ tutorial
C1
(q1.1)
s1
tmp1
reply-to: (tmp1)
(q1)
C1
q1
tmp1
1
(tmp1)
q1
3
reply-to: (tmp1)
2
C1
tmp1
q1
s1
s1
C1
(tmp1)
s1
tmp1
q1
4
• coordinate REQUEST-REPLY into transaction by reply queue id

12. Example Code 2:
RabbitMQ RPC
(rpcq1)
C1
s1
tmp1
reply-to: (tmp1)
cor-id: (1)
(rpcq1)
C1
rpcq1
tmp1
q1
1
tmp1
rpcq1
reply-to: (tmp1)
cor-id: (1)
2
(tmp1)
C1
cor-id: (1)
s1
C1
(tmp1)
cor-id: (1)
s1
rpcq1
tmp1
3
•
s1
Coordinate REQUEST-REPLY into transaction by correlate-id
4

13. Example Code 3:
Celery (Distributed Task Queue)
(celery:celery)
(celery:celery)
C1
s1
add
(1,1)
celery
result:
1
celery:
celery
cor-id: (1)
C1
s1
celery
result:
1
step 1
(
add
(1,1)
cor-id: (1)
step 2
(celery-result:1)
step 3: polling
status of result
queue many
times:
celery:
celery
C1
(celery-result:1)
C1
poll
step a
empty
step b
)*

14. Example Code 3:
Celery (Distributed Task Queue)
(celery-result:1)
C1
(celery-result:1)
worker
celeryresult:
1
celery:
celery
step 4
cor-id: (1)
C1
worker
cor-id: (1)
celeryresult:
1
celery:
celery
step 5
• Coordinate REQUEST-REPLY into transaction by (correlate-id)

15. Celery: one client application with two worker
w1
service
C1
w1
2+2
1+1
cor-id: (2)
cor-id: (1)
worker thread pool
1. client make two calls via the same connection
w1
C1
2
cor-id: (1)
w2
4
cor-id: (2)
2. worker send REPLY via its own connection
3. client receive REPLYs via the same connection

16. 5 Real Cases Study
• Applications use RabbitMQ
1. Zenoss4‘s message bus between daemons (non
transaction app)
2. Nova controller’s RPC call between hosts (transaction app)
• Enterprise use RabbitMQ
3. Instagram : Celery + RabbitMQ
4. MercadoLibre: RabbitMQ
5. Indeed.com
MailboxApp

17. Case 1: Zenoss 4
• A message bus to pass event and heartbeat between daemons
– RabbitMQ as message queue server
– Protobufs to serialization
• Non-transaction application
zenhub
zenoss
event server
heart
beat
event
zenhubworker
zeventd
raw
event
RabbitMQ

18. Heartbeat
zenhub
zeneventd
zenhubwork
er
zenhubworker.py---(1)-->[RabbitMQ]
zenhubworker.py---(1)-->[RabbitMQ]
zenhubworker.py---(1)-->[RabbitMQ]
[RabbitMQ]---(3)-->java
[RabbitMQ]---(3)-->java
[RabbitMQ]---(3)-->java
zenoss
event server
HB
HB
HB
heart
beat
HB
message bus
publish exchange: zenoss.heartbeats key: zenoss.heartbeat.localhost
content header: org.zenoss.protobufs.zep.DaemonHeartbeat
content with len: 33
deliver
content header: org.zenoss.protobufs.zep.DaemonHeartbeat
content with len: 33

19. Ping Failure
zenoss
event server
zenping
1
raw
ping
event
event
2
ping
event
zeventd
raw
event
RabbitMQ

20. Ping Failure Messages
zenping---(1)-->[RabbitMQ]
zenping---(1)-->[RabbitMQ]
zenping---(1)-->[RabbitMQ]
[RabbitMQ]---(1)-->zeneventd.py
[RabbitMQ]---(1)-->zeneventd.py
[RabbitMQ]---(1)-->zeneventd.py
zeneventd.py---(1)-->[RabbitMQ]
zeneventd.py---(1)-->[RabbitMQ]
zeneventd.py---(1)-->[RabbitMQ]
[RabbitMQ]---(1)-->java
[RabbitMQ]---(1)-->java
[RabbitMQ]---(1)-->java
java---(6)-->[RabbitMQ]
java---(6)-->[RabbitMQ]
java---(6)-->[RabbitMQ]
publish exchange: zenoss.zenevents.raw key: zenoss.zenevent.status.ping
content header: org.zenoss.protobufs.zep.Event
content with len: 235
deliver
content header: org.zenoss.protobufs.zep.Event
content with len: 235
publish exchange: zenoss.zenevents.zep key: zenoss.zenevent.status.ping
content header: org.zenoss.protobufs.zep.ZepRawEvent
content with len: 432
deliver
content header: org.zenoss.protobufs.zep.ZepRawEvent
content with len: 432
publish exchange: zenoss.zenevents key: zenoss.zenevent.status.ping
content header: org.zenoss.protobufs.zep.EventSummary
content with len: 433

21. Case 2: Nova Controller
• A message queue server for RPC call between hosts
• Each RPC call has its own unique message-id generated by UUID library

22. Case 3: Instagram
• Web server uses nginx
• Messaging system use
RabbitMQ
– Scalability: self-made message
broker, dispatch message in RR
– Reliability: RabbitMQ cluster
• Worker system use Celery
– Celery is distributed task queue
– worker concurrency model
• slow task: pre-fork pool
• fast task: gevent
• Overall throughput
– 25k application threads
publishing task
– 4K tasks per second
– Also monitor queue length
http://lanyrd.com/2013/pycon/scdyrp/

23. Nginx Server with Celery
nginx
w1
client1
1
cor-id: (c1’s request)
w2
client
client
client
10k
10k
10k
2
cor-id: (c2’s request)
queue server
Nginx
• single thread
• event driven
• asynchronous (non-blocking)
• Task complete and
return REPLY messages

24. Case 4: MercadoLibre
• MercadoLibre is 8th largest ecommerce company in
Latin American
• original architecture of their ecommerce system:
requests
requests
requests
Redis
nodeJS
nginx
proxy
to route
requests
20 million request per
minutes, 4GB per seconds
Grails
Groovy
7000+ vm insance
(by openstack)
MongoDb

25. Case 4: MercadoLibre
• Changes from LB + web
app to enterprise
service bus architecture
• Benefit
–
–
–
–
error handling
routing
scalability
event driven for higher
performance
web
server
app1
req
app2
item
feed
req
enterprise
service bus
app3
persist

26. MercadoLibre’s
News Update System
• The items-app is not transaction based

27. MercadoLibre’s
Selectively Receive Message
• The items-app is not transaction based

28. Case 5:
indeed.COM
crawler
engine
Rabbit
MQ
crawler
engine
Rabbit
MQ
crawler
engine
Rabbit
MQ
Asian DC
Job write
service
Job write
service
DB
Job write
service
Rabbit
Rabbit
MQ
MQ
HA
DB
DB
pacific
US DC
ocean
• crawler engine app is not transaction based

29. Reference
•
•
•
•
•
•
•
RabbitMQ specification 0.8
RabbitMQ Tutorial, example 1/6 and its packet dump
Celery: Distributed Task Queue and its packet dump
Messaging at Scale at Instagram
How is nginx different from apache
Zenoss source code
nova controller source code