• Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,742
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
78
Comments
0
Likes
10

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. memcached on steroids
  • 2. “blazingly fast” – Kirk Haines, Engineyard
  • 3. Back in the day…Salvatore Sanfilippo
  • 4. Back in the day…Salvatore Sanfilippo
  • 5. Back in the day…LLOOGG Google Analytics + Realtime
  • 6. Back in the day…LLOOGG Google Analytics + Realtime
  • 7. Back in the day…LLOOGG Google Analytics + Realtime
  • 8. Back in the day… “ LLOOGG is working very well… RedisWe never experienced any stability problem – Salvatore Sanfilippo Google Analytics + Realtime
  • 9. Back in the day…community growth @simonw Simon Willison @antirez just posted a redis feature request http://groups.google.com/group/redis-db/.../cd7e7c65dba53e27 - a way of randomly fetching an item from a set without also deleting it 20 Oct 09 via web
  • 10. Back in the day…community growth @simonw Simon Willison @antirez just posted a redis feature request http://groups.google.com/group/redis-db/.../cd7e7c65dba53e27 - a way of randomly fetching an item from a set without also deleting it 20 Oct 09 via web
  • 11. Back in the day…VMWare acquisition
  • 12. A little server of awesome man, that’s totally stolen from Simon Willisonhttp://www.flickr.com/photos/ajc1/4663140532
  • 13. Redis Manifesto1. A DSL for Abstract Data Types2. Memory storage is #13. Fundamental data structures for a fundamental API4. Code is like a poem5. We’re against complexity6. Two levels of API7. We optimize for joy
  • 14. A DSL For Abstract Data Types a key/value store? redis> SET key hello OK redis> GET key “hello”
  • 15. A DSL For Abstract Data Types a key/value store? request redis> SET key hello OK redis> GET key “hello”
  • 16. A DSL For Abstract Data Types a key/value store? request redis> SET key hello OK awesome, a REPL! redis-cli redis> GET key “hello”
  • 17. A DSL For Abstract Data Types on par with memcache redis> APPEND key world (integer) 10 redis> SETNX key bye (integer) 1 redis> GET key “helloworld” redis> DEL key (integer) 1
  • 18. A DSL For Abstract Data Types on par with memcache: atomic counters redis> SET participants 10 OK redis> INCR participants (integer) 11 “It gives Memcached a serious run for its money.” – Mathias Meyer, Peritor GmbH
  • 19. A DSL For Abstract Data Types on par with memcache: cache invalidation redis> EXPIRE key 60 OK redis> ...
  • 20. A DSL For Abstract Data Types on par with memcache: cache invalidation redis> EXPIRE key 60 OK redis> TTL key (integer) 53 redis>
  • 21. A DSL For Abstract Data Types on par with memcache: cache invalidation redis> EXPIRE key 60 OK redis> TTL key (integer) 53 redis> GET key “helloworld” redis>
  • 22. A DSL For Abstract Data Types on par with memcache: cache invalidation redis> EXPIRE key 60 OK redis> TTL key (integer) 53 redis> GET key “helloworld” redis> ...
  • 23. A DSL For Abstract Data Types on par with memcache: cache invalidation redis> EXPIRE key 60 OK redis> TTL key (integer) 53 redis> GET key “helloworld” redis> GET key (nil)
  • 24. “Memory isthe new disk. –– Jim Gray, Turing Award laureate
  • 25. Memory storage is #1 Redis key spacehttp://spotfireblog.tibco.com/wp-content/uploads/in-memory-analytics.jpg
  • 26. Memory storage is #1 snapshotting modehttp://www.flickr.com/photos/bionicteaching/3212235059/
  • 27. Memory storage is #1 snapshotting mode every x seconds every y operations on SAVE / BGSAVEhttp://www.flickr.com/photos/bionicteaching/3212235059/
  • 28. Memory storage is #1 snapshotting mode fork(); // copy on write! dump(); move(); every x seconds every y operations on SAVE / BGSAVEhttp://www.flickr.com/photos/bionicteaching/3212235059/
  • 29. Memory storage is #1 Append-Only File – Write-Ahead Logginghttp://www.flickr.com/photos/generationbass/4827013488/
  • 30. Memory storage is #1 Append-Only File – Write-Ahead Logging fsync() on every commandhttp://www.flickr.com/photos/generationbass/4827013488/
  • 31. Memory storage is #1 Append-Only File – Write-Ahead Logging fsync() on every command fsync() every secondhttp://www.flickr.com/photos/generationbass/4827013488/
  • 32. Memory storage is #1 Append-Only File – Write-Ahead Logging fsync() on every command fsync() every second fsync() per OShttp://www.flickr.com/photos/generationbass/4827013488/
  • 33. Memory storage is #1 Append-Only File – Write-Ahead Logging fsync() on every command fsync() every second fsync() per OS BGREWRITEAOF trims AOFhttp://www.flickr.com/photos/generationbass/4827013488/
  • 34. Memory storage is #1 meh, persistence memory overheadhttp://www.flickr.com/photos/generationbass/4827013488/
  • 35. Memory storage is #1 memory contention ?
  • 36. Memory storage is #1 memory contention ?
  • 37. Memory storage is #1 memory contention Virtual Memory to the rescue ?
  • 38. Memory storage is #1 memory contention Virtual Memory to the rescue ? because Salvatore is much brighter than all of Microsoft Research
  • 39. Memory storage is #1 memory contention Virtual Memory to the rescue ? because Salvatore is much brighter than all of Microsoft Research …and Redis types do not map to OS pages
  • 40. Memory storage is #1 memory contentionstay in memory all keys need to values must be swapped in their entirety swapped by age and size Virtual Memory to the rescue ? because Salvatore is much brighter than all of Microsoft Research …and Redis types do not map to OS pages
  • 41. Fundamental data structuresfor a fundamental API
  • 42. Fundamental data structuresfor a fundamental API lists, we can redis> LPUSH databases mysql (integer) 1 redis> LPUSH databases mongodb (integer) 2
  • 43. Fundamental data structuresfor a fundamental API lists, we can redis> LPUSH databases mysql (integer) 1 redis> LPUSH databases mongodb (integer) 2 redis> LRANGE databases 0 -1 1) “mongodb” 2) “mysql”
  • 44. Fundamental data structuresfor a fundamental API lists, we can redis> LPUSH databases mysql (integer) 1 redis> LPUSH databases mongodb (integer) 2 redis> LRANGE databases 0 -1 1) “mongodb” Bulk reply 2) “mysql”
  • 45. Fundamental data structuresfor a fundamental API lists, we can redis> LPUSH databases mysql implemented as Linked Lists (integer) 1 redis> LPUSH databases mongodb (integer) 2 redis> LRANGE databases 0 -1 1) “mongodb” Bulk reply 2) “mysql”
  • 46. Fundamental data structuresfor a fundamental API lists, we can redis> LPUSH databases mysql implemented as Linked Lists (integer) 1 redis> LPUSH databases mongodb (integer) 2 PUSH redis> POP O(1) LRANGE databases 0 -1 1) “mongodb” Bulk reply 2) “mysql”
  • 47. Fundamental data structuresfor a fundamental API a data structure server lists sets sorted hash sets tables
  • 48. Fundamental data structuresfor a fundamental API a data structure server lists sets sorted hash sets tables
  • 49. Fundamental data structuresfor a fundamental API a data structure server lists sets sorted hash sets tables
  • 50. APPEND key value RENAME key newkey BLPOP key [key ...] RENAMENX key newkey BRPOP key [key ...] timeout RPOP keyFundamental data structures BRPOPLPUSH source destination timeout DECR key DECRBY key decrement RPOPLPUSH source destination RPUSH key valuefor a fundamental API DEL key [key ...] RPUSHX key value EXISTS key SADD key member EXPIRE key seconds SCARD key EXPIREAT key timestamp SDIFF key [key ...] a data processing server FLUSHALL R FLUSHDB GET key SDIFFSTORE destination key [key ...] SELECT index SET key value GETBIT key offset SETBIT key offset value GETRANGE key start end SETEX key seconds value GETSET key value SETNX key value HDEL key field SETRANGE key offset value HEXISTS key field SINTER key [key ...] HGET key field SINTERSTORE destination key [key ...] HGETALL key SISMEMBER key member HINCRBY key field increment SMEMBERS key HKEYS key SMOVE source destination member HLEN key SORT key [BY pattern] [LIMIT offset count] [GET pattern [GET pattern ...]] [ASC|DESC] [ALPHA] HMGET key field [field ...] SPOP key HMSET key field value [field value ...] SRANDMEMBER key HSET key field value SREM key member HSETNX key field value STRLEN key HVALS key SUNION key [key ...] INCR key SUNIONSTORE destination key [key ...] INCRBY key increment TTL key KEYS pattern TYPE key LINDEX key index ZADD key score member LINSERT key BEFORE|AFTER pivot value ZCARD key LLEN key ZCOUNT key min max LPOP key ZINCRBY key increment member LPUSH key value ZINTERSTORE destination numkeys key [key ...] LPUSHX key value ZRANGE key start stop [WITHSCORES] LRANGE key start stop ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count] LREM key count value ZRANK key member LSET key index value ZREM key member LTRIM key start stop ZREMRANGEBYRANK key start stop MGET key [key ...] ZREMRANGEBYSCORE key min max MOVE key db ZREVRANGE key start stop [WITHSCORES] MSET key value [key value ...] ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count] MSETNX key value [key value ...] ZREVRANK key member PERSIST key ZSCORE key member RANDOMKEY ZUNIONSTORE destination numkeys key [key ...]
  • 51. APPEND key value RENAME key newkey BLPOP key [key ...] RENAMENX key newkey BRPOP key [key ...] timeout RPOP keyFundamental data structures BRPOPLPUSH source destination timeout DECR key DECRBY key decrement RPOPLPUSH source destination RPUSH key valuefor a fundamental API DEL key [key ...] RPUSHX key value EXISTS key SADD key member EXPIRE key seconds SCARD key EXPIREAT key timestamp SDIFF key [key ...] a data processing server FLUSHALL R FLUSHDB GET key SDIFFSTORE destination key [key ...] SELECT index SET key value GETBIT key offset SETBIT key offset value GETRANGE key start end SETEX key seconds value GETSET key value SETNX key value HDEL key field SETRANGE key offset value HEXISTS key field SINTER key [key ...] HGET key field SINTERSTORE destination key [key ...] HGETALL key SISMEMBER key member HINCRBY key field increment SMEMBERS key HKEYS key SMOVE source destination member HLEN key SORT key [BY pattern] [LIMIT offset count] [GET pattern [GET pattern ...]] [ASC|DESC] [ALPHA] HMGET key field [field ...] SPOP key HMSET key field value [field value ...] SRANDMEMBER key HSET key field value SREM key member HSETNX key field value STRLEN key HVALS key SUNION key [key ...] INCR key SUNIONSTORE destination key [key ...] INCRBY key increment TTL keyevery operation is atomic KEYS pattern TYPE key LINDEX key index ZADD key score member LINSERT key BEFORE|AFTER pivot value ZCARD key LLEN key ZCOUNT key min max LPOP key ZINCRBY key increment member LPUSH key value ZINTERSTORE destination numkeys key [key ...] LPUSHX key value ZRANGE key start stop [WITHSCORES] LRANGE key start stop ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count] LREM key count value ZRANK key member LSET key index value ZREM key member LTRIM key start stop ZREMRANGEBYRANK key start stop MGET key [key ...] ZREMRANGEBYSCORE key min max MOVE key db ZREVRANGE key start stop [WITHSCORES] MSET key value [key value ...] ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count] MSETNX key value [key value ...] ZREVRANK key member PERSIST key ZSCORE key member RANDOMKEY ZUNIONSTORE destination numkeys key [key ...]
  • 52. APPEND key value RENAME key newkey BLPOP key [key ...] RENAMENX key newkey BRPOP key [key ...] timeout RPOP keyFundamental data structures BRPOPLPUSH source destination timeout DECR key DECRBY key decrement RPOPLPUSH source destination RPUSH key valuefor a fundamental API DEL key [key ...] RPUSHX key value EXISTS key SADD key member EXPIRE key seconds SCARD key EXPIREAT key timestamp SDIFF key [key ...] a data processing server FLUSHALL R FLUSHDB GET key SDIFFSTORE destination key [key ...] SELECT index SET key value GETBIT key offset SETBIT key offset value GETRANGE key start end SETEX key seconds value GETSET key value SETNX key value HDEL key field SETRANGE key offset value HEXISTS key field SINTER key [key ...] HGET key field SINTERSTORE destination key [key ...] HGETALL key SISMEMBER key member HINCRBY key field increment SMEMBERS key HKEYS key SMOVE source destination member HLEN key SORT key [BY pattern] [LIMIT offset count] [GET pattern [GET pattern ...]] [ASC|DESC] [ALPHA] HMGET key field [field ...] SPOP key HMSET key field value [field value ...] SRANDMEMBER key HSET key field value SREM key member HSETNX key field value STRLEN key HVALS key SUNION key [key ...] INCR key SUNIONSTORE destination key [key ...] INCRBY key increment TTL keyevery operation is atomic KEYS pattern TYPE key LINDEX key index ZADD key score member LINSERT key BEFORE|AFTER pivot value ZCARD key LLEN key ZCOUNT key min max LPOP key ZINCRBY key increment member LPUSH key value ZINTERSTORE destination numkeys key [key ...] LPUSHX key value ZRANGE key start stop [WITHSCORES] LRANGE key start stop ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count] LREM key count value ZRANK key member LSET key index value ZREM key member LTRIM key start stop MGET key [key ...] MOVE key db MSET key value [key value ...] MSETNX key value [key value ...] ZREMRANGEBYRANK key start stop ZREMRANGEBYSCORE key min max Mind blown. ZREVRANGE key start stop [WITHSCORES] ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count] ZREVRANK key member PERSIST key ZSCORE key member RANDOMKEY ZUNIONSTORE destination numkeys key [key ...]
  • 53. Fundamental data structuresfor a fundamental API CAP & ACID promises atomicity consistency availability durability consistency isolation partition tolerance “kicks ACID out the door” –Mathias Meyer, Chief Cloud Officer
  • 54. Fundamental data structuresfor a fundamental API CAP & ACID promises atomicity consistency “that’s availability like saying durability consistency why aren’t filesystems ACID” isolation partition tolerance “kicks ACID out the door” –Mathias Meyer, Chief Cloud Officer
  • 55. Code is like a poem C 23.000 lines of code
  • 56. We’re against complexity
  • 57. Two levels of APIs transactions redis> MULTI OK redis> GET key QUEUED redis> DEL key QUEUED redis> EXEC 1) “helloworld” 2) (integer) 1
  • 58. Two levels of APIs transactions redis> MULTI OK redis> GET key QUEUED redis> DEL key QUEUED redis> EXEC DISCARD 1) “helloworld” 2) (integer) 1
  • 59. Two levels of APIs transactions redis> MULTI OK redis> GET key QUEUED atomic redis> DEL key QUEUED redis> EXEC 1) “helloworld” 2) (integer) 1
  • 60. Two levels of there’s always a but: APIsserver crash = partial execution transactions redis> MULTI OK redis> GET key QUEUED atomic redis> DEL key QUEUED redis> EXEC 1) “helloworld” 2) (integer) 1
  • 61. Two levels of there’s always a but: APIsserver crash = partial execution transactions redis> MULTI OK there’s always a but but: redis> GET key detected on startup atomic QUEUED => exit redis> DEL key QUEUED redis> EXEC 1) “helloworld” 2) (integer) 1
  • 62. Two levels of there’s always a but: APIsserver crash = partial execution transactions redis> MULTI OK there’s always a but but: redis> GET key detected on startup atomic QUEUED => exit redis> DEL key QUEUED always a but but but: there’s redis> EXEC redis-check-aof repairs it 1) “helloworld” 2) (integer) 1
  • 63. Two levels of APIs optimistic locking redis> WATCH key OK redis> MULTI OK
  • 64. Two levels of APIs optimistic locking redis> WATCH key OK redis> MULTI OK SET key ciao <redis OK
  • 65. Two levels of APIs optimistic locking redis> WATCH key OK redis> MULTI OK SET key ciao <redis OK redis> SET key bye QUEUED redis> EXEC (nil)
  • 66. Two levels of APIs optimistic locking redis> WATCH key OK redis> MULTI OK SET key ciao <redis OK redis> SET key bye QUEUED redis> EXEC (nil) GET key <redis “ciao”
  • 67. We optimize for joy Pub/Sub redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1
  • 68. We optimize for joy Pub/Sub redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” type 2) “chat” 3) (integer) 1
  • 69. We optimize for joy Pub/Sub redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” channel 3) (integer) 1
  • 70. We optimize for joy Pub/Sub redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1 message
  • 71. We optimize for joy Pub/Sub redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1 PUBLISH chat “asl?” <redis (integer) 1
  • 72. We optimize for joy Pub/Sub redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1 PUBLISH chat “asl?” <redis (integer) 1 1) “message” 2) “chat” 3) “asl?” ^C
  • 73. We optimize for joy Pub/Sub redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1 PUBLISH chat “asl?” <redis (integer) 1 1) “message” 2) “chat” in other news: 3) “asl?” PSUBSCRIBE ^C UNSUBSCRIBE
  • 74. “Redis ismore than justa key-value store,it’s a lifestyle. –– Mathias Meyer
  • 75. Benchmarks redis-benchmark$ redis-benchmark====== SET ====== 100000 requests completed in 0.88 seconds 50 parallel clients 3 bytes payload====== GET ====== 100000 requests completed in 1.23 seconds Linux 2.6, Xeon X3320 2.5GHz, loopback
  • 76. BenchmarksColin Howe3.7x writes speed improvement for over MySQL2.1x reads
  • 77. Replication master-slave scenariohttp://www.flickr.com/photos/joiearai/4393860397/
  • 78. Replication master-slave scenariohttp://www.flickr.com/photos/joiearai/4393860397/
  • 79. Replication master-slave scenariohttp://www.flickr.com/photos/joiearai/4393860397/
  • 80. Replication master-slave scenario SLAVEOF host porthttp://www.flickr.com/photos/joiearai/4393860397/
  • 81. Replication master-slave scenario SLAVEOF host port save + re-issuinghttp://www.flickr.com/photos/joiearai/4393860397/
  • 82. Who’s using itGithub Resque message queue service Configuration Management Routing Information
  • 83. Who’s using itDigg Digg Streaming API personalized news data real time view click counts
  • 84. Who’s using itwooga Monster World inventory system Happy Hospital user system
  • 85. Who’s using it Peter Noordhuis# Author: Peter Noordhuis# Description: Simple demo to showcase Redis PubSub with EventMachine<170 lines of Ruby code><220 lines of HTML+JS>
  • 86. Who’s using it Hurl HTTP request testing WildlifeNearYou API rate limiting Crowdsourcing taxonomies Vanity A/B testing The GuardianCrowdsourced analysis of MP expenses BNP membership list analysis
  • 87. Who’s using it Hurl HTTP request testing WildlifeNearYou API rate limiting Crowdsourcing taxonomies general blogosphere hype stormhttp://redis.io/topics/using-redis lists about 30 others Vanity A/B testing The Guardian Crowdsourced analysis of MP expenses BNP membership list analysis
  • 88. “I think the… problem is…believing thatthere can be“one true datastore”.Different technologiesexcel at different things. –– Weixi Yen
  • 89. Check out…Simon Willison’s tutorialhttp://simonwillison.net/static/2010/redis-tutorial
  • 90. Check out… Redis, from the Ground Uphttp://blog.mjrusso.com/2010/10/17/redis-from-the-ground-up Try Redis The Redis Cookbook http://try.redis-db.com/ http://rediscookbook.org/ Redis: The Definitive Guide Data modeling, caching, and messagingcoming August 2011, Salvatore & Pieter, published by O’Reilly also check my Redis bookmarks: http://del.icio.us/lehmannro/redis
  • 91. man, that was a hell of a ride. Thanks. Questions?Now go use Redis.io!