RestMQ

Redis based Message Queue
WHAT !?

RestMQ - Message Queue based on Redis

Data manipulation:
HTTP GET/POST/DELETE, COMET and WebSockets

No signalin...
Key/Value storage

Think memcached, GET and SET operations

[key] = value

>> SET key meh
>> GET key
meh

>> SET fucounter...
Redis

   Key-Value database
   Atomic operations
   Semi-persistent or persistent storage
   Publish/Subscription channel...
Redis

Data types

  Strings
  Lists
  Sets and Ordered Sets
  Hashes
  Publish/Subscribe channels
Redis

Things you can do:

   Cache
   Inverted indexes
   Fast counters
   Throttle control
   Cooler stuff at Redis Cook...
Redis

Things you can't do:

- Search inside all values for a given key
- Search inside all values for a given key
- Searc...
Message Queues

 Does everything really needs to be tightly coupled ?
 Order: first come, first served
 Transactions: ther...
Message Queues


 SMTP: Oldest Message Queue around
 Avoid hitting your DB real time with MQ
 Real time: Ratings, Voting a...
RestMQ - Endpoints Diagram
RestMQ - Messages

    /q/<queue> : Queue handling (GET/POST/DELETE)
   $ curl -X POST -d "value=foobar" http://localhost:...
RestMQ - Messages

   /c/<queue> : Comet Endpoint

$ curl http://localhost:8888/c/testq
     (hangs until there is a messa...
RestMQ - Messages
JSON Protocol inspired by Amazon SQS
First prototype at http://jsonqueue.appspot.com
      add          ...
RestMQ - Status

 /stats/<queue> : Queue status
 $ curl http://localhost:8888/stats/test

 {
     "queue" : "test",
      ...
RestMQ - Controls

 /p/<queue> : Distribution Policy for COMET and WS

 Round Robin a message for each consumer
 Broadcast...
RestMQ - Data structure Diagram
RestMQ - Algorithm

Algorithm for pushing messages into a queue named 'testq'

   SADD testq into a SET called queueset
  ...
RestMQ - Algorithm

Algorithm for reading messages from a queue named 'testq'

   RPOP a key from a LIST named testq:queue...
RestMQ - Implementation
(the core functionality can be implemented on any language)


Main branch 'Enterprise Edition'

  ...
RestMQ - Sinatra Implementation
http://gist.github.com/524240

GET /q - List all queues
RestMQ - Sinatra Implementation
http://gist.github.com/524240

GET /q/<queue> - Get a message from <queue>
RestMQ - Sinatra Implementation
http://gist.github.com/524240

POST /q/<queue> - Insert a message in <queue>
RestMQ - Map/Reduce
Questions ?
Thanks
Upcoming SlideShare
Loading in...5
×

RestMQ - HTTP/Redis based Message Queue

89,106

Published on

Presentation from TDC 2010 - http://www.thedevelopersconference.com.br/tdc/2010/sp/trilha-nosql

Published in: Technology

RestMQ - HTTP/Redis based Message Queue

  1. 1. RestMQ Redis based Message Queue
  2. 2. WHAT !? RestMQ - Message Queue based on Redis Data manipulation: HTTP GET/POST/DELETE, COMET and WebSockets No signaling or specific protocol (optional JSON protocol) http://www.restmq.com/ Gleicon Moraes http://github.com/gleicon http://zenmachine.wordpress.com http://www.7co.cc
  3. 3. Key/Value storage Think memcached, GET and SET operations [key] = value >> SET key meh >> GET key meh >> SET fucounter 0 >> INCR fucounter 1 >> GET fucounter 1
  4. 4. Redis Key-Value database Atomic operations Semi-persistent or persistent storage Publish/Subscription channels Different Data types: Sets, Sorted Sets, Lists, Hashes http://code.google.com/p/redis/ http://github.com/antirez/redis http://rediscookbook.org/
  5. 5. Redis Data types Strings Lists Sets and Ordered Sets Hashes Publish/Subscribe channels
  6. 6. Redis Things you can do: Cache Inverted indexes Fast counters Throttle control Cooler stuff at Redis Cookbook Load Balancing Things you can't do: Search inside all values for a given string
  7. 7. Redis Things you can't do: - Search inside all values for a given key - Search inside all values for a given key - Search inside all values for a given key
  8. 8. Message Queues Does everything really needs to be tightly coupled ? Order: first come, first served Transactions: there is no transaction, just tracking Job Scheduling Publish/Subscribe Map/Reduce
  9. 9. Message Queues SMTP: Oldest Message Queue around Avoid hitting your DB real time with MQ Real time: Ratings, Voting and Comments Twitter Do I need NoSQL or I really need to clean my mess ?
  10. 10. RestMQ - Endpoints Diagram
  11. 11. RestMQ - Messages /q/<queue> : Queue handling (GET/POST/DELETE) $ curl -X POST -d "value=foobar" http://localhost:8888/q/testq testq:1 $ curl -X POST -d "value=foobar" http://localhost:8888/q/testq testq:2 $ curl http://localhost:8888/q/testq { "count": 0, "value": "foobar", "key": "testq:1" }
  12. 12. RestMQ - Messages /c/<queue> : Comet Endpoint $ curl http://localhost:8888/c/testq (hangs until there is a message to be received) /ws/<queue> : WebSocket Endpoint Needs the proper javascript code
  13. 13. RestMQ - Messages JSON Protocol inspired by Amazon SQS First prototype at http://jsonqueue.appspot.com add get { { "cmd" : "add", "cmd" : "get", "queue" : "testq", "queue" : "testq", "value" : "abacab" } } del take { { "cmd" : "del", "cmd" : "take", "queue" : "testq", "queue" : "testq", "key" : "testq:31" } }
  14. 14. RestMQ - Status /stats/<queue> : Queue status $ curl http://localhost:8888/stats/test { "queue" : "test", "redis": "127.0.0.1:6379 - 10 connection(s)", "len": {"len": 1} } /stats : Server status $ curl http://localhost:8888/stats/ { "count": 4, "queues": [ "devops:queue", "test3:queue", "test:queue", "test2:queue"], "redis": "127.0.0.1:6379 - 10 connection(s)" }
  15. 15. RestMQ - Controls /p/<queue> : Distribution Policy for COMET and WS Round Robin a message for each consumer Broadcast a message to all consumers /control/<queue> : Queue start/stop Pause all COMET consumers (useful for job schedulers)
  16. 16. RestMQ - Data structure Diagram
  17. 17. RestMQ - Algorithm Algorithm for pushing messages into a queue named 'testq' SADD testq into a SET called queueset INCR a per-queue UUID generator called testq:uuid SET testq:<<id>>, message LPUSH testq:<<id>> in to a LIST named testq:queue Works for new and existing queues. On-the-fly queue creation.
  18. 18. RestMQ - Algorithm Algorithm for reading messages from a queue named 'testq' RPOP a key from a LIST named testq:queue GET the value associated to this key For non-destructive GET operations: LINDEX -1 to get a key in testq:queue LIST INCR key:refcount +1 GET the value stored for key
  19. 19. RestMQ - Implementation (the core functionality can be implemented on any language) Main branch 'Enterprise Edition' Python Twisted Cyclone Redis async client Embedded version Sinatra + Ruby Reduced endpoints (GET/POST)
  20. 20. RestMQ - Sinatra Implementation http://gist.github.com/524240 GET /q - List all queues
  21. 21. RestMQ - Sinatra Implementation http://gist.github.com/524240 GET /q/<queue> - Get a message from <queue>
  22. 22. RestMQ - Sinatra Implementation http://gist.github.com/524240 POST /q/<queue> - Insert a message in <queue>
  23. 23. RestMQ - Map/Reduce
  24. 24. Questions ?
  25. 25. Thanks
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×