RestMQ - HTTP/Redis based Message Queue
Upcoming SlideShare
Loading in...5
×
 

RestMQ - HTTP/Redis based Message Queue

on

  • 67,343 views

Presentation from TDC 2010 - http://www.thedevelopersconference.com.br/tdc/2010/sp/trilha-nosql

Presentation from TDC 2010 - http://www.thedevelopersconference.com.br/tdc/2010/sp/trilha-nosql

Statistics

Views

Total Views
67,343
Views on SlideShare
17,078
Embed Views
50,265

Actions

Likes
39
Downloads
270
Comments
1

42 Embeds 50,265

http://restmq.com 34776
http://www.restmq.com 9844
http://blog.nosqlfan.com 2539
http://abrdev.com 2373
http://nosql.mypopescu.com 606
url_unknown 19
http://translate.googleusercontent.com 15
http://irr.posterous.com 9
http://www.iredis.com 8
http://taotao2345.hatenablog.com 7
http://www.google.com 6
http://stage.clpbrd.com 6
http://thinkery.me 5
http://cc.bingj.com 4
https://www.google.com 4
http://www.google.co.uk 4
http://webcache.googleusercontent.com 4
http://cache.baidu.com 3
http://www.16kan.com 3
http://www.pega.vn 3
http://www.slideshare.net 2
https://www.google.co.uk 2
http://207.46.192.232 2
http://www.clipboard.com 2
http://clipboard.com 2
http://www.google.it 1
http://restmq.com.netzcheck.com 1
http://www.google.co.kr 1
http://www.google.co.in 1
http://www.google.bg 1
http://ranksit.com 1
http://www.200please.com 1
http://www.google.co.za 1
http://www.google.fr 1
http://xianguo.com 1
http://1.1.103.1 1
https://twitter.com 1
https://www.google.dk 1
http://www.yahoo.com&_=1371701865223 HTTP 1
http://172.20.0.9 1
http://hghltd.yandex.net 1
http://www.google.ee 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

RestMQ - HTTP/Redis based Message Queue Presentation Transcript

  • 1. RestMQ Redis based Message Queue
  • 2. WHAT !? RestMQ - Message Queue based on Redis Data manipulation: HTTP GET/POST/DELETE, COMET and WebSockets No signaling or specific protocol (optional JSON protocol) http://www.restmq.com/ Gleicon Moraes http://github.com/gleicon http://zenmachine.wordpress.com http://www.7co.cc
  • 3. Key/Value storage Think memcached, GET and SET operations [key] = value >> SET key meh >> GET key meh >> SET fucounter 0 >> INCR fucounter 1 >> GET fucounter 1
  • 4. Redis Key-Value database Atomic operations Semi-persistent or persistent storage Publish/Subscription channels Different Data types: Sets, Sorted Sets, Lists, Hashes http://code.google.com/p/redis/ http://github.com/antirez/redis http://rediscookbook.org/
  • 5. Redis Data types Strings Lists Sets and Ordered Sets Hashes Publish/Subscribe channels
  • 6. Redis Things you can do: Cache Inverted indexes Fast counters Throttle control Cooler stuff at Redis Cookbook Load Balancing Things you can't do: Search inside all values for a given string
  • 7. Redis Things you can't do: - Search inside all values for a given key - Search inside all values for a given key - Search inside all values for a given key
  • 8. Message Queues Does everything really needs to be tightly coupled ? Order: first come, first served Transactions: there is no transaction, just tracking Job Scheduling Publish/Subscribe Map/Reduce
  • 9. Message Queues SMTP: Oldest Message Queue around Avoid hitting your DB real time with MQ Real time: Ratings, Voting and Comments Twitter Do I need NoSQL or I really need to clean my mess ?
  • 10. RestMQ - Endpoints Diagram
  • 11. RestMQ - Messages /q/<queue> : Queue handling (GET/POST/DELETE) $ curl -X POST -d "value=foobar" http://localhost:8888/q/testq testq:1 $ curl -X POST -d "value=foobar" http://localhost:8888/q/testq testq:2 $ curl http://localhost:8888/q/testq { "count": 0, "value": "foobar", "key": "testq:1" }
  • 12. RestMQ - Messages /c/<queue> : Comet Endpoint $ curl http://localhost:8888/c/testq (hangs until there is a message to be received) /ws/<queue> : WebSocket Endpoint Needs the proper javascript code
  • 13. RestMQ - Messages JSON Protocol inspired by Amazon SQS First prototype at http://jsonqueue.appspot.com add get { { "cmd" : "add", "cmd" : "get", "queue" : "testq", "queue" : "testq", "value" : "abacab" } } del take { { "cmd" : "del", "cmd" : "take", "queue" : "testq", "queue" : "testq", "key" : "testq:31" } }
  • 14. RestMQ - Status /stats/<queue> : Queue status $ curl http://localhost:8888/stats/test { "queue" : "test", "redis": "127.0.0.1:6379 - 10 connection(s)", "len": {"len": 1} } /stats : Server status $ curl http://localhost:8888/stats/ { "count": 4, "queues": [ "devops:queue", "test3:queue", "test:queue", "test2:queue"], "redis": "127.0.0.1:6379 - 10 connection(s)" }
  • 15. RestMQ - Controls /p/<queue> : Distribution Policy for COMET and WS Round Robin a message for each consumer Broadcast a message to all consumers /control/<queue> : Queue start/stop Pause all COMET consumers (useful for job schedulers)
  • 16. RestMQ - Data structure Diagram
  • 17. RestMQ - Algorithm Algorithm for pushing messages into a queue named 'testq' SADD testq into a SET called queueset INCR a per-queue UUID generator called testq:uuid SET testq:<<id>>, message LPUSH testq:<<id>> in to a LIST named testq:queue Works for new and existing queues. On-the-fly queue creation.
  • 18. RestMQ - Algorithm Algorithm for reading messages from a queue named 'testq' RPOP a key from a LIST named testq:queue GET the value associated to this key For non-destructive GET operations: LINDEX -1 to get a key in testq:queue LIST INCR key:refcount +1 GET the value stored for key
  • 19. RestMQ - Implementation (the core functionality can be implemented on any language) Main branch 'Enterprise Edition' Python Twisted Cyclone Redis async client Embedded version Sinatra + Ruby Reduced endpoints (GET/POST)
  • 20. RestMQ - Sinatra Implementation http://gist.github.com/524240 GET /q - List all queues
  • 21. RestMQ - Sinatra Implementation http://gist.github.com/524240 GET /q/<queue> - Get a message from <queue>
  • 22. RestMQ - Sinatra Implementation http://gist.github.com/524240 POST /q/<queue> - Insert a message in <queue>
  • 23. RestMQ - Map/Reduce
  • 24. Questions ?
  • 25. Thanks