Redis — The AK-47 of Post-relational Databases

Redis — The AK-47 Of Post-relational DatabasesKarel Minařík

Karel Minařík→ Independent web designer and developer→ Ruby, Rails, Git, CouchDB propagandista in .cz→ Previously: Flash Developer; Art Director; Information Architect;… (see LinkedIn)→ @karmiq at Twitter→ karmi.cz Redis

WEAPON DESIGNAK-47Designed at the end of WWII by Mikhail KalashnikovAssault Rifle, not a “submachine gun”Simple designDesigned for mass production & low-quality manufacturingExtreme reliability, at the cost of accuracy Redis

http://www.youtube.com/watch?v=J6c3DLlM9KA#t=8m32s Redis

Reliability Redis

SIMPLICITYThe Redis ManifestoWere against complexity.We believe designing systems is a fight against complexity.Most of the time the best way to fight complexity is by notcreating it at all.http://antirez.com/post/redis-manifesto.html Redis

SIMPLICITYRedis and NoSQLWhats wrong with RDBMS when used for (many) tasksthat dont need all this complexity? The data model: nonscalable, time complexity hard to predict, and cant modelmany common problems well enough.http://antirez.com/post/MongoDB-and-Redis.html Redis

Assault rifles as the crucial invention for new rules of warfare: fighting in shorter distances and shooting while on the run.http://www.youtube.com/watch?v=a1KBsqvKpXk#t=7m50s Redis

Memory is the new disk.Disk is the new tape.— Jim Grayhttp://www.infoq.com/news/2008/06/ram-is-diskhttp://research.microsoft.com/en-us/um/people/gray/talks/Flash_is_Good.ppt Redis

Redis: Main Features Redis

Simplicity SpeedPredictability VersatilityReliability Low Footprint

SIMPLICITYInstallationgit clone http://github.com/antirez/rediscd redismake./src/redis-‐server./src/redis-‐cli Redis

SIMPLICITYThe Zen of Redis(...) what Redis provides are data structures (...)http://antirez.com/post/MongoDB-and-Redis.html#c1537 Redis

REDISData StructuresStringsListsSetsSorted SetsHashes Redis

REDIS DATA STRUCTURESStringsSET key "value" ~ 4 billion of keysGET key=> "value"DEL key Redis

REDIS DATA STRUCTURESFetch multiple keys at onceSET key1 "value1"SET key2 "value2"MGET key1 key2=> "value1"=> "value2" Redis

REDIS DATA STRUCTURESExpirationEXPIRE key 5GET key=> "value"TTL key=> 1GET key=> (nil) Redis

UsageCachehttp://antirez.com/post/redis-as-LRU-cache.htmlSessionshttps://github.com/mattmatt/redis-session-store Redis

REDIS DATA STRUCTURESAtomic IncrementsGET key=> nilINCR key=> 1INCR key=> 2GET key=> 2 Redis

UsageCounters (downloads, hits, votes, …)$ curl http://example.com/downloads/file1.mpgINCR downloads:totalINCR downloads:total:todayINCR downloads:total:2011-‐05-‐10INCR downloads:/downloads/file1.mpg:totalINCR downloads:/downloads/file1.mpg:todayINCR downloads:/downloads/file1.mpg:2011-‐05-‐10 Redis

UsageCounters (downloads, hits, votes, …)# Total downloads for server, all timeGET downloads:total# Total downloads for server, todayGET downloads:total:today# Total downloads for fileGET downloads:/downloads/file1.mpg:total# Total downloads for file todayINCR downloads:/downloads/file1.mpg:today... Redis

UsageCounters (downloads, hits, votes, …)# Nightly maintenance at 23:59RENAME downloads:total:today downloads:total:yesterdayAll this runs at super-sonic speed, with minimal overhead and resource consumption.See implementation for Rubygems.org: https://gist.github.com/296921However, youll hit denormalization bloat once you start adding metrics (eg. downloads percountry, per category, …) Redis

UsageVariations: Rate limiting$ curl http://api.example.com/list.jsonINCR api:<TOKEN>:hits=> 1 if INCR(api:abc123:hits) > LIMIT return 420 Enhance Your Calm end# Every hour...DEL api:<TOKEN>:hits Redis

UsageGenerating unique IDsINCR global:users_ids=> 1SET users:1:username "john"INCR global:users_ids=> 2SET users:2:username "mary" Redis

REDIS DATA STRUCTURESListsLPUSH key 1 RPOP key=> 1 => "1"LPUSH key 2 LRANGE key 0 -‐1=> 2 => "3"LPUSH key 3 => "2"=> 3 LLEN keyLRANGE key 0 -‐1 => 2=> "3" LTRIM key 0 1=> "2" => OK=> "1" Redis

UsageIndexes (list of comments, ...)LPUSH article:comments <ID>Timelines (of all sorts: messages, logs, …)LPUSH user:<ID>:inbox "message from Alice"LPUSH user:<ID>:inbox "message from Bob"# Limit the messages to 100LTRIM user:<ID>:inbox 0 99# Get last 10 messagesLRANGE user:<ID>:inbox 0 9# Get next 10 messagesLRANGE user:<ID>:inbox 10 19 Redis

UsageQueues# PublisherRPUSH queue "task-‐1"RPUSH queue "task-‐2"# Worker (blocks and waits for tasks)BLPOP queue 0 Redis

UsageQueues# publisher.shfor i in {1..10}; do redis-cli RPUSH "queue" "task-$i"done# worker.shwhile true; do redis-cli BLPOP "queue" 0done Demo Redis

REDIS DATA STRUCTURESResque: Background Processing from Github RPUSH } LPOP O(1) … tens or millions of items … https://github.com/defunkt/resque/blob/v1.13.0/lib/resque.rb#L133-138 Redis

MESSAGE QUEUESRestMQ Redis

TALK ON ASYNCHRONOUS PROCESSINGThe Code of the Forking PathsSlideshttp://www.slideshare.net/karmi/the-code-of-the-forking-paths-asynchronous-processing-with-resque-and-amqpVideo (in Czech)http://multimedia.vse.cz/media/Viewer/?peid=51c06c512f4645289c4e9c749dc85acc1d Redis

REDIS DATA STRUCTURESSetsSADD key 1 SISMEMBER key 1=> 1 => "1"SADD key 2 SISMEMBER key 5=> 2 => "0"SADD key 3 SRANDMEMBER key=> 3 => "<RAND>"SMEMBERS key SREM key 3=> "3" => 1=> "1"=> "2" Redis

REDIS DATA STRUCTURESSet OperationsSADD A 1 SADD B 1SADD A 2 SADD B 3SMEMBERS A SMEMBERS B=> "1" => "1"=> "2" => "3" Redis

REDIS DATA STRUCTURESSet Operations[1,2] [1,3] Union SUNION A B => 1 => 2[1,2] [1,3] => 3 Intersection SINTER A B => 1[1,2] [1,3] Difference SDIFF A B => 2http://en.wikipedia.org/wiki/Set_(mathematics)#Basic_operations Redis

UsageAd servingSADD ads:cars "Check out Toyota!"SADD ads:cars "Check out Ford!"...SADD ads:movies "Check out Winters Bone!"...SRANDMEMBER ads:carsSRANDMEMBER ads:moviesNote: Time complexity is O(1). “Check out ODER BY RAND()!” Redis

UsageRelations (Friends/followers)SADD users:A:follows BSADD users:B:follows CSADD users:B:follows DSADD users:C:follows ASADD users:C:follows D Redis

UsageRelations (Friends/followers)# Joint network of A and BSUNION users:A:follows users:B:follows1) "C"2) "D"3) "B" Redis

UsageRelations (Friends/followers)# Common for A and BSINTER users:A:follows users:B:follows[]# Common for B and CSINTER users:B:follows users:C:follows1) "D"# Unique to B compared to CSDIFF users:B:follows users:C:follows1) "C" Redis

UsageRelations (Friends/followers)# Whom I follow...SADD friends A SMEMBERS friendsSADD friends B 1) "A" 2) "B"# Who follows me...SADD followers B SMEMBERS followersSADD followers C 1) "C" 2) "B" Redis

UsageRelations (Friends/followers)# Mutual relationshipsSINTER friends followers1) "B"# Who does not follow me back?SDIFF friends followers1) "A"# Who am I not following back?SDIFF followers friends1) "C" Redis

Mining the Social Web Analyzing Data from Facebook, Twitter, LinkedIn, and Other Social Media Sites Matthew A. Russell http://books.google.com/books?id=SYM1lrQdrdsC&lpg=PP1&pg=PA94http://oreilly.com/catalog/0636920010203 Redis

UsageRelations (Article tags/categories)SADD tags:ruby article-‐1SADD tags:java article-‐2SADD tags:web article-‐1SADD tags:web article-‐2 Redis

UsageRelations (Article tags/categories)# ruby OR javaSUNION tags:ruby tags:java1) "article-‐2"2) "article-‐1"# ruby AND javaSINTER tags:ruby tags:java[]# web AND NOT rubySDIFF tags:web tags:ruby1) "article-‐2" Redis

UsageFriends Online# My friendsSADD friends ASADD friends BSADD friends C# Friend A connectsSADD online:fresh ASUNIONSTORE online online:fresh online:stale# Whos online now?SINTER friends online[A] Redis

UsageFriends Online# Every minute, rename the "fresh" to "stale" ...RENAME online:fresh online:stale# ... and update the "online" setSUNIONSTORE online online:fresh online:stale# Friend B connectsSADD online:fresh BSUNIONSTORE online online:fresh online:stale# Whos online now?SINTER friends online[A,B] Redis

UsageFriends Online# Time passes ...# Rename the "fresh" to "stale", every minute ...RENAME online:fresh online:stale# ... and update the "online" setSUNIONSTORE online online:fresh online:stale# Whos online now?SINTER friends online[B] Redis

REDIS DATA STRUCTURESSorted SetsZADD key 100 A ZREVRANGE key 0 -‐1ZADD key 10 C 1) "A"ZADD key 80 B 2) "B" 3) "C"ZRANGE key 0 -‐1 ZINCRBY key 10 C1) "C" "20"2) "B"3) "A" Redis

REDIS DATA STRUCTURESSorted SetsZREVRANGE key 0 -‐1 ZREVRANGEBYSCORE WITHSCORES key 100 501) "A" 1) "A"2) "100" 2) "B"3) "B"4) "80"5) "C"6) "20" Redis

UsageLeaderboards# User A got 10 pointsZINCRBY scores 10 A# User B got 15 pointsZINCRBY scores 15 B# User A got another 10 pointsZINCRBY scores 10 A# Display scoresZREVRANGE scores 0 -‐1 WITHSCORES1) "A"2) "20"3) "B"4) "15" Redis

UsageInverted Index# Index document AZINCRBY index:foo 1 document-‐AZINCRBY index:foo 1 document-‐AZINCRBY index:bar 1 document-‐A# Index document BZINCRBY index:foo 1 document-‐BZINCRBY index:baz 1 document-‐B# Search for token foo, sort by occurencesZREVRANGE index:foo 0 -‐1 WITHSCORES1) "document-‐A"2) "2"3) "document-‐B"4) "1" Redis

REDIS SORTED SETS AS AN INDEXInverted Indexhttps://gist.github.com/928605 Redis

REDIS DATA STRUCTURESHashesHMSET users:1 username j name JohnHMSET users:2 username m name MaryHGETALL users:11) "username"2) "j"...HKEYS users:11) "username"2) "name"HSET users:1 score 100HGET users:1 score1) "100" Redis

UsageStructured data (Articles, users, …)HMSET articles:1 title "Redis is cool!" content "I recently ..." published "2011-‐05-‐10"HGETALL articles:11) "title"2) "Redis is cool!"3) "content"4) "I recently ..."5) "published"6) "2011-‐05-‐10"HSET articles:1 title "Redis is very cool!"HGET articles:1 title"Redis is very cool!" Redis

UsageUser Preferences (No login)# Save preferences from <FORM>HMSET prefs:<COOKIE HASH> background #ccc color #333# Keep it for one yearEXPIRE prefs:<COOKIE HASH> 31556926# Retrieve preferencesHGETALL prefs:<COOKIE HASH> Redis

REDIS FEATURESPublish/SubscribeSUBSCRIBE log.errorPUBLISH log.error "ERROR"PSUBSCRIBE log.* “AMQP”PUBLISH log.error "ERROR"=> "ERROR"=> "ERROR"PUBLISH log.info "INFO"=> "INFO" Demo Redis

REDIS FEATURESDurabilityBGSAVE/usr/local/var/db/redis/dump.rdbBGREWRITEAOF# redis.confappendonly yeshttp://redis.io/topics/persistence Redis

REDIS FEATURESVirtual Memory# redis.confvm-‐enabled yesAllows to work with data sets bigger then available RAM.Swaps less often used values to disk.Keys must still fit into RAM.# Make Redis disk-‐bound databasevm-‐max-‐memory 0The future? Redis Diskstore (disk-bound by default, cache in between server and disk)http://redis.io/topics/virtual-memory Redis

REDIS FEATURESReplication# slave.confslaveof 127.0.0.1 6379$ redis-‐server slave.confThe future? Redis Cluster (Dynamo-like, Distributed hash table)http://redis.io/topics/replication Redis

REDIS FEATURESScripting (experimental)$ git clone https://github.com/antirez/redis -‐b scripting redis-‐scripting$ cd redis-‐scripting$ make$ echo port 7397 | ./src/redis-‐server -‐$ ./src/redis-‐cliSET myscript "return HELLO"EVAL "return loadstring(redis(get, KEYS[1]))()" 1 myscript=> "HELLO"http://antirez.com/post/scripting-branch-released.html Redis

REDIS RESOURCESThe Interactive Documentation PROMPThttp://redis.io/commands Redis

REDIS RESOURCESSimon Willison Tutorialhttp://simonwillison.net/static/2010/redis-tutorial/ Redis

REDIS RESOURCESRedis Implementation Detailshttp://pauladamsmith.com/articles/redis-under-the-hood.html Redis

REDIS RESOURCESRedis Use Caseshttp://highscalability.com/blog/2010/12/6/what-the-heck-are-you-actually-using-nosql-for.html Redis

REDIS RESOURCESThe Original Metaphore…http://flazz.me/redis-the-ak-47-of-databases Redis

Thanks! d

Redis — The AK-47 of Post-relational Databases

Karel Minarik

Redis — The AK-47 of Post-relational Databases