RestMQ - HTTP/Redis based Message Queue

  • 71,848 views
Uploaded on

Presentation from TDC 2010 - http://www.thedevelopersconference.com.br/tdc/2010/sp/trilha-nosql …

Presentation from TDC 2010 - http://www.thedevelopersconference.com.br/tdc/2010/sp/trilha-nosql

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
71,848
On Slideshare
0
From Embeds
0
Number of Embeds
33

Actions

Shares
Downloads
274
Comments
1
Likes
41

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. RestMQ Redis based Message Queue
  • 2. WHAT !? RestMQ - Message Queue based on Redis Data manipulation: HTTP GET/POST/DELETE, COMET and WebSockets No signaling or specific protocol (optional JSON protocol) http://www.restmq.com/ Gleicon Moraes http://github.com/gleicon http://zenmachine.wordpress.com http://www.7co.cc
  • 3. Key/Value storage Think memcached, GET and SET operations [key] = value >> SET key meh >> GET key meh >> SET fucounter 0 >> INCR fucounter 1 >> GET fucounter 1
  • 4. Redis Key-Value database Atomic operations Semi-persistent or persistent storage Publish/Subscription channels Different Data types: Sets, Sorted Sets, Lists, Hashes http://code.google.com/p/redis/ http://github.com/antirez/redis http://rediscookbook.org/
  • 5. Redis Data types Strings Lists Sets and Ordered Sets Hashes Publish/Subscribe channels
  • 6. Redis Things you can do: Cache Inverted indexes Fast counters Throttle control Cooler stuff at Redis Cookbook Load Balancing Things you can't do: Search inside all values for a given string
  • 7. Redis Things you can't do: - Search inside all values for a given key - Search inside all values for a given key - Search inside all values for a given key
  • 8. Message Queues Does everything really needs to be tightly coupled ? Order: first come, first served Transactions: there is no transaction, just tracking Job Scheduling Publish/Subscribe Map/Reduce
  • 9. Message Queues SMTP: Oldest Message Queue around Avoid hitting your DB real time with MQ Real time: Ratings, Voting and Comments Twitter Do I need NoSQL or I really need to clean my mess ?
  • 10. RestMQ - Endpoints Diagram
  • 11. RestMQ - Messages /q/<queue> : Queue handling (GET/POST/DELETE) $ curl -X POST -d "value=foobar" http://localhost:8888/q/testq testq:1 $ curl -X POST -d "value=foobar" http://localhost:8888/q/testq testq:2 $ curl http://localhost:8888/q/testq { "count": 0, "value": "foobar", "key": "testq:1" }
  • 12. RestMQ - Messages /c/<queue> : Comet Endpoint $ curl http://localhost:8888/c/testq (hangs until there is a message to be received) /ws/<queue> : WebSocket Endpoint Needs the proper javascript code
  • 13. RestMQ - Messages JSON Protocol inspired by Amazon SQS First prototype at http://jsonqueue.appspot.com add get { { "cmd" : "add", "cmd" : "get", "queue" : "testq", "queue" : "testq", "value" : "abacab" } } del take { { "cmd" : "del", "cmd" : "take", "queue" : "testq", "queue" : "testq", "key" : "testq:31" } }
  • 14. RestMQ - Status /stats/<queue> : Queue status $ curl http://localhost:8888/stats/test { "queue" : "test", "redis": "127.0.0.1:6379 - 10 connection(s)", "len": {"len": 1} } /stats : Server status $ curl http://localhost:8888/stats/ { "count": 4, "queues": [ "devops:queue", "test3:queue", "test:queue", "test2:queue"], "redis": "127.0.0.1:6379 - 10 connection(s)" }
  • 15. RestMQ - Controls /p/<queue> : Distribution Policy for COMET and WS Round Robin a message for each consumer Broadcast a message to all consumers /control/<queue> : Queue start/stop Pause all COMET consumers (useful for job schedulers)
  • 16. RestMQ - Data structure Diagram
  • 17. RestMQ - Algorithm Algorithm for pushing messages into a queue named 'testq' SADD testq into a SET called queueset INCR a per-queue UUID generator called testq:uuid SET testq:<<id>>, message LPUSH testq:<<id>> in to a LIST named testq:queue Works for new and existing queues. On-the-fly queue creation.
  • 18. RestMQ - Algorithm Algorithm for reading messages from a queue named 'testq' RPOP a key from a LIST named testq:queue GET the value associated to this key For non-destructive GET operations: LINDEX -1 to get a key in testq:queue LIST INCR key:refcount +1 GET the value stored for key
  • 19. RestMQ - Implementation (the core functionality can be implemented on any language) Main branch 'Enterprise Edition' Python Twisted Cyclone Redis async client Embedded version Sinatra + Ruby Reduced endpoints (GET/POST)
  • 20. RestMQ - Sinatra Implementation http://gist.github.com/524240 GET /q - List all queues
  • 21. RestMQ - Sinatra Implementation http://gist.github.com/524240 GET /q/<queue> - Get a message from <queue>
  • 22. RestMQ - Sinatra Implementation http://gist.github.com/524240 POST /q/<queue> - Insert a message in <queue>
  • 23. RestMQ - Map/Reduce
  • 24. Questions ?
  • 25. Thanks