RabbitMQ + CouchDB = Awesome

one system to rule them all

the challenge

• write a backend that scales well • handle various frontends (web, iPhone, ...) • runs “somewhere in the cloud” • easy to maintain and extend

the solution

RabbitMQ +CouchDB =Awesome

Part I

RabbitMQ • AMQP based Message Queue • lots of additional transports (XMPP, HTTP, STOMP, ...) • erlang based

frontend frontend frontend daemon RabbitMQ daemon daemon daemon

• every functionality gets a backed daemon • every daemon has a queue associated • communication via a JSON message that gets passed around

the JSON message

{ "response": { "expiry": 710, "ts": "1234425918", "name": "Lenz Gschwendtner", "last4": 0, "cc_type": "master" }, "data": { "options": { "billing_id": "E16003D4-B312-BEA23D9AB789" }, "command": "billing_info" }, "meta": { "lang": "en", "reply_to": "B312-E16003D4-BEA23D9AB789", "platform": "iwmn" } }

works great for ... • RPC style communication • if the ﬂow through the system is static • fast transactions

gets ﬁddly when ... • there are asynchronous callbacks from outside • message ﬂow can change based on various factors • tasks can take days to complete

Part II

workﬂows

... again ...

what we have ... • various frontends • backend daemons • JSON messages

what we need • a way to describe message ﬂow • a easy ﬂexible way to write and update it

CouchDB

... to the rescue

workflow • really only a path definition through backends • some sort of condition management • a backend definition

{ "_id": "f7a4408898e5...a05b4181045", "_rev": "3-633060011", "name": "billing_info", "type": "workflow", "user": "iwmn", "path": [ "logger", "billing", "billing_iwmn_cc" ] }

... and the backend deﬁnition

{ "_id": "33adf2e3efc3...f423929f057ac", "_rev": "3-3507820969", "conn": "rpc", "type": "worker", "name": "billing_iwmn_cc", "location": "billing.cc", "description": "the CC interface", "command": "billing_verify_cc" }

now we need a workﬂow daemon

frontend frontend frontend daemon RabbitMQ workﬂow daemon daemon daemon

where are we now? • we can call backends as before • we can call workflows via the workflow queue • we can manage workflows in CouchDB

Part III

asynchronous callbacks

problem • we start a workflow • we call some external API • the external API somehow notifies us (Mail, Callback URL, ...) about progress • we need to continue in the workflow

we need a persistent message store

{ "_id": "22cd37afd06c82129adb665d3150f570", "_rev": "3-2669449653", "status": "done", "last_update": 1252296819, "name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789", "message": { "response": { "expiry": 710, "name": "Lenz Gschwendtner", "last4": 0, "cc_type": "master" }, "data": { "options": { "billing_id": "E16003D4-F8DB-11DD-B312-BEA23D9AB789" }, "command": "billing_info" }, "meta": { "lang": "de", "name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789", "orig_reply_to": "D234ADE8-9B64-11DE-A92C-E5EF11C6FBDB", "reply_to": "workflow", "user": "iwmn", "log": [ "logger", "billing", "billing.iwmn.info" ], "platform": "iwmn" } }, "created": 1252296815, "type": "bill" }

one more problem how do we correlate messages

... with some CouchDB awesome ...

• the backend daemon for the callback polls a view in CouchDB • we can search for any value in the request we sent • all we need is a view that searches for the very value this external API uses as correlation ID

external frontend frontend frontend callback daemon RabbitMQ workﬂow daemon daemon daemon

me Lenz Gschwendtner CTO iWantMyName @norbu09

credits • RabbitMQ • CouchDB • Erlang and Perl

more credits • http://www.flickr.com/photos/djpd • http://www.flickr.com/photos/lilyapp • http://www.flickr.com/photos/b-tal • http://www.flickr.com/photos/mikebaird • http://www.flickr.com/photos/eyetwist

http://abrdev.com	894
http://nosql.mypopescu.com	468
http://www.slideshare.net	119
http://norbu09.org	103
https://twimg0-a.akamaihd.net	10
url_unknown	8
http://www.bookandman.com	5
http://www.mefeedia.com	2
http://www.risingcode.com	1
http://www.techgig.com	1
http://webcache.googleusercontent.com	1
http://trunk.ly	1

RabbitMQ + CouchDB = Awesome

by Lenz Gschwendtner on Oct 14, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

12 Embeds 1,613

Statistics

RabbitMQ + CouchDB = Awesome — Presentation Transcript