Redis

Case studies
Redis

 Key/Value
 Async I/O
 Very fast (most ops take O(1))
 Active development (VM, APPEND Only datafile, HASH
 type)
 V...
Redis

 Why ?
 RestMQ
 Brief MongoDB/Redis recap
 Information Retrieval: using SETs to search books
Why ? Another Key/Value ?

REDIS is a key value storage, but it presents different data
types.

These datatypes are the bu...
RestMQ

RestMQ is a HTTP/REST/JSON based message queue.

   HTTP as transport protocol
   REST as a way to organize resour...
RestMQ and Redis

For each queue <<q>> in Redis:

q:uuid - The queue unique id counter
q:queue - The queue LIST (fifo)
q:c...
An async/sharding Redis client

Original python clients:

redis.py: Synchronous
txredis: Incomplete

Needed:

Async client...
Web app framework

Original RestMQ ways twisted.web based. Cool, but too much
work.

http://github.com/fiorix/cyclone

A t...
RestMQ

   COMET consumer
   REST producer/consumer
   JSON Based producer/consumer
   COMET is pausable (start/stop contr...
Brief MongoDB/Redis recap - Books
MongoDB                                           Redis
{
 'id': 1,
                    ...
Brief MongoDB/Redis recap - Books
MongoDB                                     Redis
Search tags for erlang or haskell:
   ...
DOCDB


http://github.com/gleicon/docdb

Almost a document database.

eBook indexing - Basic IR procedure

   tokenize(spl...
DOCDB

Simulation about how many wordSETs would be created by book:


$ python doc_to_sets.py books/10702.txt
5965

$ pyth...
DOCDB

Simulation about how many wordSETs would be created by book, accumulating the result:


$ python doc_to_sets.py boo...
The End


- Check the project's website: http://code.google.com/p/redis/

- Python/Twisted driver: http://github.com/fiori...
Upcoming SlideShare
Loading in...5
×

Redis

8,278

Published on

Presentation about Redis and use cases

Published in: Technology
0 Comments
16 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
8,278
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
153
Comments
0
Likes
16
Embeds 0
No embeds

No notes for slide

Redis

  1. 1. Redis Case studies
  2. 2. Redis Key/Value Async I/O Very fast (most ops take O(1)) Active development (VM, APPEND Only datafile, HASH type) Values can be data types: LISTS, SETS, ORDERED SETS (http://code.google. com/p/redis/wiki/IntroductionToRedisDataTypes) One step further than memcached, same intuitive applications and patterns
  3. 3. Redis Why ? RestMQ Brief MongoDB/Redis recap Information Retrieval: using SETs to search books
  4. 4. Why ? Another Key/Value ? REDIS is a key value storage, but it presents different data types. These datatypes are the building blocks of more complex stuff you use already. Think LISTs, SETs, Ordered SETs, and methods to deal with them as you would do with a good standard library. Also, different persistence strategies, replication, locks, increments...
  5. 5. RestMQ RestMQ is a HTTP/REST/JSON based message queue. HTTP as transport protocol REST as a way to organize resources JSON as data exchange format - Built initially to mimic Amazon's SQS functionality at GAE (http://jsonqueue.appspot.com) - Standalone server, uses Python, Cyclone, Twisted and Redis. - COMET consumer (bind your http client and get objects)
  6. 6. RestMQ and Redis For each queue <<q>> in Redis: q:uuid - The queue unique id counter q:queue - The queue LIST (fifo) q:control - Queue pause control q:<<id>> - objects in queue. Global: QUEUESET: SET containing all queues Also: Persistence, statistics
  7. 7. An async/sharding Redis client Original python clients: redis.py: Synchronous txredis: Incomplete Needed: Async client, with connection pool and sharding (well sharding is a plus). http://github.com/fiorix/txredisapi
  8. 8. Web app framework Original RestMQ ways twisted.web based. Cool, but too much work. http://github.com/fiorix/cyclone A twisted based tornado clone. COMET is a breeze, lots of web framework stuff, json encode/decode support built in. Integrates easily with txredis-api. The core queue protocol was ported and extended form the GAE version.
  9. 9. RestMQ COMET consumer REST producer/consumer JSON Based producer/consumer COMET is pausable (start/stop control) HTTP based. Even CURL can operate a MQ now. Asynchronous I/O Map/Reduce and Actors are a given (easy to implement, example shipped) http://github.com/gleicon/restmq
  10. 10. Brief MongoDB/Redis recap - Books MongoDB Redis { 'id': 1, SET book:1 {'title' : 'Diving into Python', 'title' : 'Diving into Python', 'author': 'Mark Pilgrim'} 'author': 'Mark Pilgrim', SET book:2 { 'title' : 'Programing Erlang', 'tags': ['python','programming', 'computing'] 'author': 'Joe Armstrong'} } SET book:3 { 'title' : 'Programing in Haskell', 'author': 'Graham Hutton'} { 'id':2, 'title' : 'Programing Erlang', SADD tag:python 1 'author': 'Joe Armstrong', SADD tag:erlang 2 'tags': ['erlang','programming', 'computing', SADD tag:haskell 3 'distributedcomputing', 'FP'] SADD tag:programming 1 2 3 } SADD tag computing 1 2 3 SADD tag:distributedcomputing 2 { SADD tag:FP 2 3 'id':3, 'title' : 'Programing in Haskell', 'author': 'Graham Hutton', 'tags': ['haskell','programming', 'computing', 'FP'] }
  11. 11. Brief MongoDB/Redis recap - Books MongoDB Redis Search tags for erlang or haskell: SINTER 'tag:erlang' 'tag:haskell' db.books.find({"tags": 0 results { $in: ['erlang', 'haskell'] } SINTER 'tag:programming' 'tag:computing' }) 3 results: 1, 2, 3 Search tags for erlang AND haskell (no results) SUNION 'tag:erlang' 'tag:haskell' 2 results: 2 and 3 db.books.find({"tags": { $all: ['erlang', 'haskell'] SDIFF 'tag:programming' 'tag:haskell' } 2 results: 1 and 2 (haskell is excluded) }) This search yields 3 results db.books.find({"tags": { $all: ['programming', 'computing'] } })
  12. 12. DOCDB http://github.com/gleicon/docdb Almost a document database. eBook indexing - Basic IR procedure tokenize(split) each word take the stop words out stemming group words to make composed searches possible Lots of wordSETs, but as documents are stored, the growing rate slows.
  13. 13. DOCDB Simulation about how many wordSETs would be created by book: $ python doc_to_sets.py books/10702.txt 5965 $ python doc_to_sets.py books/13437-8.txt 6125 $ python doc_to_sets.py books/2346.txt 1920 $ python doc_to_sets.py books/24022.txt 3470 $ python doc_to_sets.py books/advsh12.txt 5576
  14. 14. DOCDB Simulation about how many wordSETs would be created by book, accumulating the result: $ python doc_to_sets.py books/10702.txt books/13437-8.txt books/2346.txt books/24022. txt books/advsh12.txt 5965 9183 9426 10030 11400 That would mean 11400 SETs in Redis, named for the STEM of the word, each one containing the IDs of books with this word. The growing rate starts with 5965 (no sets) and goes to 1370 sets between the last two documents. The search would be like using SINTER, SUNION and SDIFF as shown before, to find book by words.
  15. 15. The End - Check the project's website: http://code.google.com/p/redis/ - Python/Twisted driver: http://github.com/fiorix/txredisapi (connection pool, consistent hashing) - No silver bullet - Plan ahead, use IR techniques - Own your data - SETs and LISTs are building blocks for most operations regarding indexes. Use them. - http://code.google.com/p/redis/wiki/IntroductionToRedisDataTypes - Intro do Redis DataTypes - More about its features: http://code.google.com/p/redis/wiki/Features - http://code.google.com/p/redis/wiki/TwitterAlikeExample - Twitter clone using Redis
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×