one system to rule
     them all
the challenge
• write a backend that scales well
• handle various frontends (web, iPhone, ...)
• runs “somewhere in the cloud”
• easy to...
the solution
RabbitMQ
+CouchDB
=Awesome
Part I
RabbitMQ

• AMQP based Message Queue
• lots of additional transports (XMPP, HTTP,
  STOMP, ...)
• erlang based
frontend    frontend   frontend




daemon       RabbitMQ




daemon        daemon         daemon
• every functionality gets a backed daemon
• every daemon has a queue associated
• communication via a JSON message that
 ...
the JSON message
{
    "response": {
        "expiry": 710,
        "ts": "1234425918",
        "name": "Lenz Gschwendtner",
        "last4...
works great for ...

• RPC style communication
• if the flow through the system is static
• fast transactions
gets fiddly when ...

• there are asynchronous callbacks from
  outside
• message flow can change based on various
  factors...
Part II
workflows
... again ...
what we have ...

• various frontends
• backend daemons
• JSON messages
what we need


• a way to describe message flow
• a easy flexible way to write and update it
CouchDB
... to the rescue
workflow

• really only a path definition through
  backends
• some sort of condition management
• a backend definition
{
    "_id": "f7a4408898e5...a05b4181045",
    "_rev": "3-633060011",
    "name": "billing_info",
    "type": "workflow",
...
... and the backend
      definition
{
    "_id": "33adf2e3efc3...f423929f057ac",
    "_rev": "3-3507820969",
    "conn": "rpc",
    "type": "worker",
    "nam...
now we need a
workflow daemon
frontend    frontend   frontend




daemon       RabbitMQ         workflow




daemon        daemon         daemon
where are we now?

• we can call backends as before
• we can call workflows via the workflow
  queue
• we can manage workflow...
Part III
asynchronous callbacks
problem

• we start a workflow
• we call some external API
• the external API somehow notifies us
  (Mail, Callback URL, ......
we need a persistent
   message store
{
    "_id": "22cd37afd06c82129adb665d3150f570",
    "_rev": "3-2669449653",
    "status": "done",
    "last_update": 1252...
one more problem
  how do we correlate messages
... with some CouchDB
       awesome ...
• the backend daemon for the callback polls a
  view in CouchDB

• we can search for any value in the request
  we sent

•...
external   frontend    frontend   frontend
callback



       daemon         RabbitMQ         workflow




       daemon   ...
me
Lenz Gschwendtner
CTO iWantMyName
   @norbu09
credits

• RabbitMQ
• CouchDB
• Erlang and Perl
more credits

• http://www.flickr.com/photos/djpd
• http://www.flickr.com/photos/lilyapp
• http://www.flickr.com/photos/b-tal...
RabbitMQ + CouchDB = Awesome
RabbitMQ + CouchDB = Awesome
RabbitMQ + CouchDB = Awesome
RabbitMQ + CouchDB = Awesome
RabbitMQ + CouchDB = Awesome
Upcoming SlideShare
Loading in...5
×

RabbitMQ + CouchDB = Awesome

17,280

Published on

Slides from a talk I gave at erlounge Wellington about a workflow system using RabbitMQ and CouchDB.

Published in: Technology, Economy & Finance
2 Comments
23 Likes
Statistics
Notes
  • sure you can, you only have to write all the message passing on your own. i did that before using rabbit and feel sad about all the time i spent to get the message passing right. if you write in erlang that is done for you in the VM and you can do it right away. if you write in anything else, a message queue is a huge benefit :-)
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I don't get it, why can't you do all of this without RabbitMQ?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
17,280
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
249
Comments
2
Likes
23
Embeds 0
No embeds

No notes for slide

RabbitMQ + CouchDB = Awesome

  1. 1. one system to rule them all
  2. 2. the challenge
  3. 3. • write a backend that scales well • handle various frontends (web, iPhone, ...) • runs “somewhere in the cloud” • easy to maintain and extend
  4. 4. the solution
  5. 5. RabbitMQ +CouchDB =Awesome
  6. 6. Part I
  7. 7. RabbitMQ • AMQP based Message Queue • lots of additional transports (XMPP, HTTP, STOMP, ...) • erlang based
  8. 8. frontend frontend frontend daemon RabbitMQ daemon daemon daemon
  9. 9. • every functionality gets a backed daemon • every daemon has a queue associated • communication via a JSON message that gets passed around
  10. 10. the JSON message
  11. 11. { "response": { "expiry": 710, "ts": "1234425918", "name": "Lenz Gschwendtner", "last4": 0, "cc_type": "master" }, "data": { "options": { "billing_id": "E16003D4-B312-BEA23D9AB789" }, "command": "billing_info" }, "meta": { "lang": "en", "reply_to": "B312-E16003D4-BEA23D9AB789", "platform": "iwmn" } }
  12. 12. works great for ... • RPC style communication • if the flow through the system is static • fast transactions
  13. 13. gets fiddly when ... • there are asynchronous callbacks from outside • message flow can change based on various factors • tasks can take days to complete
  14. 14. Part II
  15. 15. workflows
  16. 16. ... again ...
  17. 17. what we have ... • various frontends • backend daemons • JSON messages
  18. 18. what we need • a way to describe message flow • a easy flexible way to write and update it
  19. 19. CouchDB
  20. 20. ... to the rescue
  21. 21. workflow • really only a path definition through backends • some sort of condition management • a backend definition
  22. 22. { "_id": "f7a4408898e5...a05b4181045", "_rev": "3-633060011", "name": "billing_info", "type": "workflow", "user": "iwmn", "path": [ "logger", "billing", "billing_iwmn_cc" ] }
  23. 23. ... and the backend definition
  24. 24. { "_id": "33adf2e3efc3...f423929f057ac", "_rev": "3-3507820969", "conn": "rpc", "type": "worker", "name": "billing_iwmn_cc", "location": "billing.cc", "description": "the CC interface", "command": "billing_verify_cc" }
  25. 25. now we need a workflow daemon
  26. 26. frontend frontend frontend daemon RabbitMQ workflow daemon daemon daemon
  27. 27. where are we now? • we can call backends as before • we can call workflows via the workflow queue • we can manage workflows in CouchDB
  28. 28. Part III
  29. 29. asynchronous callbacks
  30. 30. problem • we start a workflow • we call some external API • the external API somehow notifies us (Mail, Callback URL, ...) about progress • we need to continue in the workflow
  31. 31. we need a persistent message store
  32. 32. { "_id": "22cd37afd06c82129adb665d3150f570", "_rev": "3-2669449653", "status": "done", "last_update": 1252296819, "name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789", "message": { "response": { "expiry": 710, "name": "Lenz Gschwendtner", "last4": 0, "cc_type": "master" }, "data": { "options": { "billing_id": "E16003D4-F8DB-11DD-B312-BEA23D9AB789" }, "command": "billing_info" }, "meta": { "lang": "de", "name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789", "orig_reply_to": "D234ADE8-9B64-11DE-A92C-E5EF11C6FBDB", "reply_to": "workflow", "user": "iwmn", "log": [ "logger", "billing", "billing.iwmn.info" ], "platform": "iwmn" } }, "created": 1252296815, "type": "bill" }
  33. 33. one more problem how do we correlate messages
  34. 34. ... with some CouchDB awesome ...
  35. 35. • the backend daemon for the callback polls a view in CouchDB • we can search for any value in the request we sent • all we need is a view that searches for the very value this external API uses as correlation ID
  36. 36. external frontend frontend frontend callback daemon RabbitMQ workflow daemon daemon daemon
  37. 37. me Lenz Gschwendtner CTO iWantMyName @norbu09
  38. 38. credits • RabbitMQ • CouchDB • Erlang and Perl
  39. 39. more credits • http://www.flickr.com/photos/djpd • http://www.flickr.com/photos/lilyapp • http://www.flickr.com/photos/b-tal • http://www.flickr.com/photos/mikebaird • http://www.flickr.com/photos/eyetwist
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×