Successfully reported this slideshow.
Your SlideShare is downloading. ×

RestMQ - HTTP/Redis based Message Queue

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Upcoming SlideShare
Redis as a message queue
Redis as a message queue
Loading in …3
×

Check these out next

1 of 25 Ad

More Related Content

Slideshows for you (20)

Similar to RestMQ - HTTP/Redis based Message Queue (20)

Advertisement

More from Gleicon Moraes (20)

Recently uploaded (20)

Advertisement

RestMQ - HTTP/Redis based Message Queue

  1. 1. RestMQ Redis based Message Queue
  2. 2. WHAT !? RestMQ - Message Queue based on Redis Data manipulation: HTTP GET/POST/DELETE, COMET and WebSockets No signaling or specific protocol (optional JSON protocol) http://www.restmq.com/ Gleicon Moraes http://github.com/gleicon http://zenmachine.wordpress.com http://www.7co.cc
  3. 3. Key/Value storage Think memcached, GET and SET operations [key] = value >> SET key meh >> GET key meh >> SET fucounter 0 >> INCR fucounter 1 >> GET fucounter 1
  4. 4. Redis Key-Value database Atomic operations Semi-persistent or persistent storage Publish/Subscription channels Different Data types: Sets, Sorted Sets, Lists, Hashes http://code.google.com/p/redis/ http://github.com/antirez/redis http://rediscookbook.org/
  5. 5. Redis Data types Strings Lists Sets and Ordered Sets Hashes Publish/Subscribe channels
  6. 6. Redis Things you can do: Cache Inverted indexes Fast counters Throttle control Cooler stuff at Redis Cookbook Load Balancing Things you can't do: Search inside all values for a given string
  7. 7. Redis Things you can't do: - Search inside all values for a given key - Search inside all values for a given key - Search inside all values for a given key
  8. 8. Message Queues Does everything really needs to be tightly coupled ? Order: first come, first served Transactions: there is no transaction, just tracking Job Scheduling Publish/Subscribe Map/Reduce
  9. 9. Message Queues SMTP: Oldest Message Queue around Avoid hitting your DB real time with MQ Real time: Ratings, Voting and Comments Twitter Do I need NoSQL or I really need to clean my mess ?
  10. 10. RestMQ - Endpoints Diagram
  11. 11. RestMQ - Messages /q/<queue> : Queue handling (GET/POST/DELETE) $ curl -X POST -d "value=foobar" http://localhost:8888/q/testq testq:1 $ curl -X POST -d "value=foobar" http://localhost:8888/q/testq testq:2 $ curl http://localhost:8888/q/testq { "count": 0, "value": "foobar", "key": "testq:1" }
  12. 12. RestMQ - Messages /c/<queue> : Comet Endpoint $ curl http://localhost:8888/c/testq (hangs until there is a message to be received) /ws/<queue> : WebSocket Endpoint Needs the proper javascript code
  13. 13. RestMQ - Messages JSON Protocol inspired by Amazon SQS First prototype at http://jsonqueue.appspot.com add get { { "cmd" : "add", "cmd" : "get", "queue" : "testq", "queue" : "testq", "value" : "abacab" } } del take { { "cmd" : "del", "cmd" : "take", "queue" : "testq", "queue" : "testq", "key" : "testq:31" } }
  14. 14. RestMQ - Status /stats/<queue> : Queue status $ curl http://localhost:8888/stats/test { "queue" : "test", "redis": "127.0.0.1:6379 - 10 connection(s)", "len": {"len": 1} } /stats : Server status $ curl http://localhost:8888/stats/ { "count": 4, "queues": [ "devops:queue", "test3:queue", "test:queue", "test2:queue"], "redis": "127.0.0.1:6379 - 10 connection(s)" }
  15. 15. RestMQ - Controls /p/<queue> : Distribution Policy for COMET and WS Round Robin a message for each consumer Broadcast a message to all consumers /control/<queue> : Queue start/stop Pause all COMET consumers (useful for job schedulers)
  16. 16. RestMQ - Data structure Diagram
  17. 17. RestMQ - Algorithm Algorithm for pushing messages into a queue named 'testq' SADD testq into a SET called queueset INCR a per-queue UUID generator called testq:uuid SET testq:<<id>>, message LPUSH testq:<<id>> in to a LIST named testq:queue Works for new and existing queues. On-the-fly queue creation.
  18. 18. RestMQ - Algorithm Algorithm for reading messages from a queue named 'testq' RPOP a key from a LIST named testq:queue GET the value associated to this key For non-destructive GET operations: LINDEX -1 to get a key in testq:queue LIST INCR key:refcount +1 GET the value stored for key
  19. 19. RestMQ - Implementation (the core functionality can be implemented on any language) Main branch 'Enterprise Edition' Python Twisted Cyclone Redis async client Embedded version Sinatra + Ruby Reduced endpoints (GET/POST)
  20. 20. RestMQ - Sinatra Implementation http://gist.github.com/524240 GET /q - List all queues
  21. 21. RestMQ - Sinatra Implementation http://gist.github.com/524240 GET /q/<queue> - Get a message from <queue>
  22. 22. RestMQ - Sinatra Implementation http://gist.github.com/524240 POST /q/<queue> - Insert a message in <queue>
  23. 23. RestMQ - Map/Reduce
  24. 24. Questions ?
  25. 25. Thanks

×