one system to rule
them all

the challenge

• write a backend that scales well
• handle various frontends (web, iPhone, ...)
• runs “somewhere in the cloud”
• easy to maintain and extend

the solution

RabbitMQ
+CouchDB
=Awesome

Part I

RabbitMQ
• AMQP based Message Queue
• lots of additional transports (XMPP, HTTP,
STOMP, ...)
• erlang based

frontend frontend frontend
daemon RabbitMQ
daemon daemon daemon

• every functionality gets a backed daemon
• every daemon has a queue associated
• communication via a JSON message that
gets passed around

the JSON message

{
"response": {
"expiry": 710,
"ts": "1234425918",
"name": "Lenz Gschwendtner",
"last4": 0,
"cc_type": "master"
},
"data": {
"options": {
"billing_id": "E16003D4-B312-BEA23D9AB789"
},
"command": "billing_info"
},
"meta": {
"lang": "en",
"reply_to": "B312-E16003D4-BEA23D9AB789",
"platform": "iwmn"
}
}

works great for ...
• RPC style communication
• if the flow through the system is static
• fast transactions

gets fiddly when ...
• there are asynchronous callbacks from
outside
• message flow can change based on various
factors
• tasks can take days to complete

Part II

workflows

... again ...

what we have ...
• various frontends
• backend daemons
• JSON messages

what we need
• a way to describe message flow
• a easy flexible way to write and update it

CouchDB

... to the rescue

workflow
• really only a path definition through
backends
• some sort of condition management
• a backend definition

{
"_id": "f7a4408898e5...a05b4181045",
"_rev": "3-633060011",
"name": "billing_info",
"type": "workflow",
"user": "iwmn",
"path": [
"logger",
"billing",
"billing_iwmn_cc"
]
}

... and the backend
definition

{
"_id": "33adf2e3efc3...f423929f057ac",
"_rev": "3-3507820969",
"conn": "rpc",
"type": "worker",
"name": "billing_iwmn_cc",
"location": "billing.cc",
"description": "the CC interface",
"command": "billing_verify_cc"
}

now we need a
workflow daemon

frontend frontend frontend
daemon RabbitMQ workflow
daemon daemon daemon

where are we now?
• we can call backends as before
• we can call workflows via the workflow
queue
• we can manage workflows in CouchDB

Part III

asynchronous callbacks

problem
• we start a workflow
• we call some external API
• the external API somehow notifies us
(Mail, Callback URL, ...) about progress
• we need to continue in the workflow

we need a persistent
message store

{
"_id": "22cd37afd06c82129adb665d3150f570",
"_rev": "3-2669449653",
"status": "done",
"last_update": 1252296819,
"name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789",
"message": {
"response": {
"expiry": 710,
"name": "Lenz Gschwendtner",
"last4": 0,
"cc_type": "master"
},
"data": {
"options": {
"billing_id": "E16003D4-F8DB-11DD-B312-BEA23D9AB789"
},
"command": "billing_info"
},
"meta": {
"lang": "de",
"name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789",
"orig_reply_to": "D234ADE8-9B64-11DE-A92C-E5EF11C6FBDB",
"reply_to": "workflow",
"user": "iwmn",
"log": [
"logger",
"billing",
"billing.iwmn.info"
],
"platform": "iwmn"
}
},
"created": 1252296815,
"type": "bill"
}

one more problem
how do we correlate messages

... with some CouchDB
awesome ...

• the backend daemon for the callback polls a
view in CouchDB
• we can search for any value in the request
we sent
• all we need is a view that searches for the
very value this external API uses as
correlation ID

external frontend frontend frontend
callback
daemon RabbitMQ workflow
daemon daemon daemon

me
Lenz Gschwendtner
CTO iWantMyName
@norbu09

credits
• RabbitMQ
• CouchDB
• Erlang and Perl

more credits
• http://www.flickr.com/photos/djpd
• http://www.flickr.com/photos/lilyapp
• http://www.flickr.com/photos/b-tal
• http://www.flickr.com/photos/mikebaird
• http://www.flickr.com/photos/eyetwist