Your SlideShare is downloading. ×
0
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

RestMQ - HTTP/Redis based Message Queue

85,200

Published on

Presentation from TDC 2010 - http://www.thedevelopersconference.com.br/tdc/2010/sp/trilha-nosql …

Presentation from TDC 2010 - http://www.thedevelopersconference.com.br/tdc/2010/sp/trilha-nosql

Published in: Technology
1 Comment
51 Likes
Statistics
Notes
No Downloads
Views
Total Views
85,200
On Slideshare
0
From Embeds
0
Number of Embeds
38
Actions
Shares
0
Downloads
290
Comments
1
Likes
51
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. RestMQ Redis based Message Queue
  • 2. WHAT !? RestMQ - Message Queue based on Redis Data manipulation: HTTP GET/POST/DELETE, COMET and WebSockets No signaling or specific protocol (optional JSON protocol) http://www.restmq.com/ Gleicon Moraes http://github.com/gleicon http://zenmachine.wordpress.com http://www.7co.cc
  • 3. Key/Value storage Think memcached, GET and SET operations [key] = value >> SET key meh >> GET key meh >> SET fucounter 0 >> INCR fucounter 1 >> GET fucounter 1
  • 4. Redis Key-Value database Atomic operations Semi-persistent or persistent storage Publish/Subscription channels Different Data types: Sets, Sorted Sets, Lists, Hashes http://code.google.com/p/redis/ http://github.com/antirez/redis http://rediscookbook.org/
  • 5. Redis Data types Strings Lists Sets and Ordered Sets Hashes Publish/Subscribe channels
  • 6. Redis Things you can do: Cache Inverted indexes Fast counters Throttle control Cooler stuff at Redis Cookbook Load Balancing Things you can't do: Search inside all values for a given string
  • 7. Redis Things you can't do: - Search inside all values for a given key - Search inside all values for a given key - Search inside all values for a given key
  • 8. Message Queues Does everything really needs to be tightly coupled ? Order: first come, first served Transactions: there is no transaction, just tracking Job Scheduling Publish/Subscribe Map/Reduce
  • 9. Message Queues SMTP: Oldest Message Queue around Avoid hitting your DB real time with MQ Real time: Ratings, Voting and Comments Twitter Do I need NoSQL or I really need to clean my mess ?
  • 10. RestMQ - Endpoints Diagram
  • 11. RestMQ - Messages /q/<queue> : Queue handling (GET/POST/DELETE) $ curl -X POST -d "value=foobar" http://localhost:8888/q/testq testq:1 $ curl -X POST -d "value=foobar" http://localhost:8888/q/testq testq:2 $ curl http://localhost:8888/q/testq { "count": 0, "value": "foobar", "key": "testq:1" }
  • 12. RestMQ - Messages /c/<queue> : Comet Endpoint $ curl http://localhost:8888/c/testq (hangs until there is a message to be received) /ws/<queue> : WebSocket Endpoint Needs the proper javascript code
  • 13. RestMQ - Messages JSON Protocol inspired by Amazon SQS First prototype at http://jsonqueue.appspot.com add get { { "cmd" : "add", "cmd" : "get", "queue" : "testq", "queue" : "testq", "value" : "abacab" } } del take { { "cmd" : "del", "cmd" : "take", "queue" : "testq", "queue" : "testq", "key" : "testq:31" } }
  • 14. RestMQ - Status /stats/<queue> : Queue status $ curl http://localhost:8888/stats/test { "queue" : "test", "redis": "127.0.0.1:6379 - 10 connection(s)", "len": {"len": 1} } /stats : Server status $ curl http://localhost:8888/stats/ { "count": 4, "queues": [ "devops:queue", "test3:queue", "test:queue", "test2:queue"], "redis": "127.0.0.1:6379 - 10 connection(s)" }
  • 15. RestMQ - Controls /p/<queue> : Distribution Policy for COMET and WS Round Robin a message for each consumer Broadcast a message to all consumers /control/<queue> : Queue start/stop Pause all COMET consumers (useful for job schedulers)
  • 16. RestMQ - Data structure Diagram
  • 17. RestMQ - Algorithm Algorithm for pushing messages into a queue named 'testq' SADD testq into a SET called queueset INCR a per-queue UUID generator called testq:uuid SET testq:<<id>>, message LPUSH testq:<<id>> in to a LIST named testq:queue Works for new and existing queues. On-the-fly queue creation.
  • 18. RestMQ - Algorithm Algorithm for reading messages from a queue named 'testq' RPOP a key from a LIST named testq:queue GET the value associated to this key For non-destructive GET operations: LINDEX -1 to get a key in testq:queue LIST INCR key:refcount +1 GET the value stored for key
  • 19. RestMQ - Implementation (the core functionality can be implemented on any language) Main branch 'Enterprise Edition' Python Twisted Cyclone Redis async client Embedded version Sinatra + Ruby Reduced endpoints (GET/POST)
  • 20. RestMQ - Sinatra Implementation http://gist.github.com/524240 GET /q - List all queues
  • 21. RestMQ - Sinatra Implementation http://gist.github.com/524240 GET /q/<queue> - Get a message from <queue>
  • 22. RestMQ - Sinatra Implementation http://gist.github.com/524240 POST /q/<queue> - Insert a message in <queue>
  • 23. RestMQ - Map/Reduce
  • 24. Questions ?
  • 25. Thanks

×