Presentation from TDC 2010 - http://www.thedevelopersconference.com.br/tdc/2010/sp/trilha-nosql

1. RestMQ
Redis based Message Queue

2. WHAT !?
RestMQ - Message Queue based on Redis
Data manipulation:
HTTP GET/POST/DELETE, COMET and WebSockets
No signaling or specific protocol (optional JSON protocol)
http://www.restmq.com/
Gleicon Moraes
http://github.com/gleicon
http://zenmachine.wordpress.com
http://www.7co.cc

3. Key/Value storage
Think memcached, GET and SET operations
[key] = value
>> SET key meh
>> GET key
meh
>> SET fucounter 0
>> INCR fucounter
1
>> GET fucounter
1

4. Redis
Key-Value database
Atomic operations
Semi-persistent or persistent storage
Publish/Subscription channels
Different Data types: Sets, Sorted Sets, Lists, Hashes
http://code.google.com/p/redis/
http://github.com/antirez/redis
http://rediscookbook.org/

5. Redis
Data types
Strings
Lists
Sets and Ordered Sets
Hashes
Publish/Subscribe channels

6. Redis
Things you can do:
Cache
Inverted indexes
Fast counters
Throttle control
Cooler stuff at Redis Cookbook
Load Balancing
Things you can't do:
Search inside all values for a given string

7. Redis
Things you can't do:
- Search inside all values for a given key
- Search inside all values for a given key
- Search inside all values for a given key

8. Message Queues
Does everything really needs to be tightly coupled ?
Order: first come, first served
Transactions: there is no transaction, just tracking
Job Scheduling
Publish/Subscribe
Map/Reduce

9. Message Queues
SMTP: Oldest Message Queue around
Avoid hitting your DB real time with MQ
Real time: Ratings, Voting and Comments
Twitter
Do I need NoSQL or I really need to clean my mess ?

10. RestMQ - Endpoints Diagram

11. RestMQ - Messages
/q/<queue> : Queue handling (GET/POST/DELETE)
$ curl -X POST -d "value=foobar" http://localhost:8888/q/testq
testq:1
$ curl -X POST -d "value=foobar" http://localhost:8888/q/testq
testq:2
$ curl http://localhost:8888/q/testq
{
"count": 0,
"value": "foobar",
"key": "testq:1"
}

12. RestMQ - Messages
/c/<queue> : Comet Endpoint
$ curl http://localhost:8888/c/testq
(hangs until there is a message to be received)
/ws/<queue> : WebSocket Endpoint
Needs the proper javascript code

13. RestMQ - Messages
JSON Protocol inspired by Amazon SQS
First prototype at http://jsonqueue.appspot.com
add get
{ {
"cmd" : "add", "cmd" : "get",
"queue" : "testq", "queue" : "testq",
"value" : "abacab" }
}
del take
{ {
"cmd" : "del", "cmd" : "take",
"queue" : "testq", "queue" : "testq",
"key" : "testq:31" }
}

14. RestMQ - Status
/stats/<queue> : Queue status
$ curl http://localhost:8888/stats/test
{
"queue" : "test",
"redis": "127.0.0.1:6379 - 10 connection(s)",
"len": {"len": 1}
}
/stats : Server status
$ curl http://localhost:8888/stats/
{
"count": 4,
"queues": [ "devops:queue", "test3:queue", "test:queue", "test2:queue"],
"redis": "127.0.0.1:6379 - 10 connection(s)"
}

15. RestMQ - Controls
/p/<queue> : Distribution Policy for COMET and WS
Round Robin a message for each consumer
Broadcast a message to all consumers
/control/<queue> : Queue start/stop
Pause all COMET consumers
(useful for job schedulers)

16. RestMQ - Data structure Diagram

17. RestMQ - Algorithm
Algorithm for pushing messages into a queue named 'testq'
SADD testq into a SET called queueset
INCR a per-queue UUID generator called testq:uuid
SET testq:<<id>>, message
LPUSH testq:<<id>> in to a LIST named testq:queue
Works for new and existing queues. On-the-fly queue creation.

18. RestMQ - Algorithm
Algorithm for reading messages from a queue named 'testq'
RPOP a key from a LIST named testq:queue
GET the value associated to this key
For non-destructive GET operations:
LINDEX -1 to get a key in testq:queue LIST
INCR key:refcount +1
GET the value stored for key

19. RestMQ - Implementation
(the core functionality can be implemented on any language)
Main branch 'Enterprise Edition'
Python
Twisted
Cyclone
Redis async client
Embedded version
Sinatra + Ruby
Reduced endpoints (GET/POST)

20. RestMQ - Sinatra Implementation
http://gist.github.com/524240
GET /q - List all queues

21. RestMQ - Sinatra Implementation
http://gist.github.com/524240
GET /q/<queue> - Get a message from <queue>

22. RestMQ - Sinatra Implementation
http://gist.github.com/524240
POST /q/<queue> - Insert a message in <queue>

23. RestMQ - Map/Reduce

24. Questions ?

25. Thanks