RabbitMQ + CouchDB = Awesome
Upcoming SlideShare
Loading in...5
×
 

RabbitMQ + CouchDB = Awesome

on

  • 21,045 views

Slides from a talk I gave at erlounge Wellington about a workflow system using RabbitMQ and CouchDB.

Slides from a talk I gave at erlounge Wellington about a workflow system using RabbitMQ and CouchDB.

Statistics

Views

Total Views
21,045
Views on SlideShare
17,720
Embed Views
3,325

Actions

Likes
18
Downloads
234
Comments
2

16 Embeds 3,325

http://abrdev.com 2388
http://nosql.mypopescu.com 667
http://www.slideshare.net 119
http://norbu09.org 115
https://twimg0-a.akamaihd.net 10
url_unknown 8
http://www.bookandman.com 7
http://www.mefeedia.com 2
http://www.linkedin.com 2
http://hghltd.yandex.net 1
http://flavors.me 1
http://www.risingcode.com 1
http://trunk.ly 1
http://webcache.googleusercontent.com 1
http://www.techgig.com 1
http://cc.bingj.com 1
More...

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • sure you can, you only have to write all the message passing on your own. i did that before using rabbit and feel sad about all the time i spent to get the message passing right. if you write in erlang that is done for you in the VM and you can do it right away. if you write in anything else, a message queue is a huge benefit :-)
    Are you sure you want to
    Your message goes here
    Processing…
  • I don't get it, why can't you do all of this without RabbitMQ?
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

RabbitMQ + CouchDB = Awesome RabbitMQ + CouchDB = Awesome Presentation Transcript

  • one system to rule them all
  • the challenge
  • • write a backend that scales well • handle various frontends (web, iPhone, ...) • runs “somewhere in the cloud” • easy to maintain and extend
  • the solution
  • RabbitMQ +CouchDB =Awesome
  • Part I
  • RabbitMQ • AMQP based Message Queue • lots of additional transports (XMPP, HTTP, STOMP, ...) • erlang based
  • frontend frontend frontend daemon RabbitMQ daemon daemon daemon
  • • every functionality gets a backed daemon • every daemon has a queue associated • communication via a JSON message that gets passed around
  • the JSON message
  • { "response": { "expiry": 710, "ts": "1234425918", "name": "Lenz Gschwendtner", "last4": 0, "cc_type": "master" }, "data": { "options": { "billing_id": "E16003D4-B312-BEA23D9AB789" }, "command": "billing_info" }, "meta": { "lang": "en", "reply_to": "B312-E16003D4-BEA23D9AB789", "platform": "iwmn" } }
  • works great for ... • RPC style communication • if the flow through the system is static • fast transactions
  • gets fiddly when ... • there are asynchronous callbacks from outside • message flow can change based on various factors • tasks can take days to complete
  • Part II
  • workflows
  • ... again ...
  • what we have ... • various frontends • backend daemons • JSON messages
  • what we need • a way to describe message flow • a easy flexible way to write and update it
  • CouchDB
  • ... to the rescue
  • workflow • really only a path definition through backends • some sort of condition management • a backend definition
  • { "_id": "f7a4408898e5...a05b4181045", "_rev": "3-633060011", "name": "billing_info", "type": "workflow", "user": "iwmn", "path": [ "logger", "billing", "billing_iwmn_cc" ] }
  • ... and the backend definition
  • { "_id": "33adf2e3efc3...f423929f057ac", "_rev": "3-3507820969", "conn": "rpc", "type": "worker", "name": "billing_iwmn_cc", "location": "billing.cc", "description": "the CC interface", "command": "billing_verify_cc" }
  • now we need a workflow daemon
  • frontend frontend frontend daemon RabbitMQ workflow daemon daemon daemon
  • where are we now? • we can call backends as before • we can call workflows via the workflow queue • we can manage workflows in CouchDB
  • Part III
  • asynchronous callbacks
  • problem • we start a workflow • we call some external API • the external API somehow notifies us (Mail, Callback URL, ...) about progress • we need to continue in the workflow
  • we need a persistent message store
  • { "_id": "22cd37afd06c82129adb665d3150f570", "_rev": "3-2669449653", "status": "done", "last_update": 1252296819, "name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789", "message": { "response": { "expiry": 710, "name": "Lenz Gschwendtner", "last4": 0, "cc_type": "master" }, "data": { "options": { "billing_id": "E16003D4-F8DB-11DD-B312-BEA23D9AB789" }, "command": "billing_info" }, "meta": { "lang": "de", "name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789", "orig_reply_to": "D234ADE8-9B64-11DE-A92C-E5EF11C6FBDB", "reply_to": "workflow", "user": "iwmn", "log": [ "logger", "billing", "billing.iwmn.info" ], "platform": "iwmn" } }, "created": 1252296815, "type": "bill" }
  • one more problem how do we correlate messages
  • ... with some CouchDB awesome ...
  • • the backend daemon for the callback polls a view in CouchDB • we can search for any value in the request we sent • all we need is a view that searches for the very value this external API uses as correlation ID
  • external frontend frontend frontend callback daemon RabbitMQ workflow daemon daemon daemon
  • me Lenz Gschwendtner CTO iWantMyName @norbu09
  • credits • RabbitMQ • CouchDB • Erlang and Perl
  • more credits • http://www.flickr.com/photos/djpd • http://www.flickr.com/photos/lilyapp • http://www.flickr.com/photos/b-tal • http://www.flickr.com/photos/mikebaird • http://www.flickr.com/photos/eyetwist