REDIS – Advanced Key Value
Store
Rajan Bhatt
What REDIS is ?
• REDIS is an open source, BSD licensed,
advanced key-value store. It is often referred
to as data structure server since keys can
contain String, Hashes, List, Sets and Sorted
sets..
• REDIS stands for REmote DIctionary Server
REDIS - Features
• Simplicity
• Speed
• Low Footprint
• Versatility
• Predictability
• Reliability
How to Install and Start REDIS server,
REDIS client and Bindings
• Go to http://redis.io/download and download
• Start Redis server ( TCP server, Single
Threaded, Listens on default port 6379 ).
• Command – redis-server <path to conf. file )
• Bundled REDIS client – redis-cli
• Client bindings available for all popular
languages, Java, Python, Ruby, Scala, Ruby,
Eralang, Go ..( I am running out of space )
Data Structure
• Strings
• List
• Sets
• Sorted Set
• Hash
• And Publish/Subscribe Design Pattern
Strings
• Set Key “Value” ( 4 billion Keys )
• Get Key returns “Value”
• Del Key
• Fetch Multiple Keys at once … MGET
• Set EXPIRATION key, e.g. Expire Key 5
• Command TTL returns remaining time before
key expires
Use Case for Expire, TTL
• Cache – LRU
http://antirez.com/post/redis-as-LRU-
cache.html
• Storage for Session ( HTTP or others ..)
Atomic Counter
• GET Key
 Nil
• INCR Key
1
• INCR Key
2
• Get Key
 2
Usage of INCR
• Great Use case Global Counters ( Download, Hits,
Votes ..)
• INCR downloads:total
• INCR downloads:total:today
• INCR downloads:total:2011-05-10
• INCR downloads:/downloads/file1.mpg:total
• INCR downloads:/downloads/file1.mpg:today
• INCR downloads:/downloads/file1.mpg:2011-05-
10
Counters
# Total downloads for server, all time
• GET downloads:total
# Total downloads for server, today
• GET downloads:total:today
# Total downloads for file
• GET downloads:/downloads/file1.mpg:total
# Total downloads for file today
• GET downloads:/downloads/file1.mpg:today
How do you reset daily counter ?
# Expire at 2011-05-10 23:59:59
• EXPIREAT downloads:total:today 1305064799
One more Use Case – API rate Limiting
• $ curl http://api.example.com/list.json
• INCR api:<TOKEN>:hits
1
• Pseudo-Code
if INCR('api:abc123:hits') > LIMIT
return 420 Enhance Your Calm
end
One more Use case Generating unique
IDs
• INCR global:users_ids
1
• SET users:1:username "john"
• INCR global:users_ids
2
• SET users:2:username "mary”
List
• LPUSH key 1
 1
• LPUSH Key 2
 2
• LPUSH Key 3
 3
• RPOP Key
 What should I get ?
• LPOP Key
 What should I get ?
• LLEN Key ( Length if List )
• LRANGE key 0 -1
• LTRIM 0 1
Usage of List
• Indexes ( List of comments ..)
LPUSH article:comments <ID>
Timelines (of all sorts: messages, logs, ...)
LPUSH user:<ID>:inbox "message from Alice"
LPUSH user:<ID>:inbox "message from Bob"
# Limit the messages to 100
LTRIM user:<ID>:inbox 0 99
# Get last 10 messages
LRANGE user:<ID>:inbox 0 9
# Get next 10 messages
LRANGE user:<ID>:inbox 10 19
List is “Queue”
 Publisher
• RPUSH queue “task-1”
• RPUSH queue “task-2”
Worker ( blocks and waits for tasks, 0 means
waits for ever )
• BLPOP queue 0
Basic Set
• SADD key 1  1
• SADD key 2  2
• SADD key 3  3
• SMEMBERS key  “3”, “2”, “1”
• SISMEMBER key 1  “1”
• SISMEMBER key 5  “0”
• SREM key 3  1
Set Operations
• SADD A 1
• SADD A 2
• SMEMBERS A  “1”, “2”
• SADD B 1
• SADD B 3
• SMEMBERS  “1”, “3”
Set Continues..
• SUNION A B  “1”, “2”, “3”
• SINTER A B  “1”
• SDIFF A B  “2”
Set Usages - Relationship
• SADD users:A:follows B
• SADD users:B:follows C
• SADD users:B:follows D
• SADD users:C:follows A
• SADD users:C:follows D
Set Relationship
• Joint network of A & B
SUNION users:A:follows users:B:follows
“C”,”D”,”B”
• Common for A & B
 SINTER users:B:follows users:C:follows
[]
• Unique to B compared to C
 “C”
Set – Friends & Followers
# Whom “swid1” follows
• SADD swid1:follows A
• SADD swid1:follows B
# Who are friends of “swid1”
• SADD swid1:friend C
• SADD swid1:friend B
# Whom swid1 follows and he is friend
• SINTER swid1:follows swid1:friend  “B”
# Whom swid1 follows and he is not friend
• SDIFF swid1:follows swid1:friend  “A”
# Whom Swid1 is friend with but does not follow
• SDIFF swid1:friend swid1:follows  “C”
Sorted Set
• ZADD key 100 A
• ZADD key 10 C
• ZADD key 80 B
• ZRANGE key 0 -1  “C”, “B”, “A”
• ZREVANGE key 0 -1  “A”, “B”, “C”
• ZINCRBY key 10 C  “20”
Sorted Set - Continues
• ZREVRANGE key 0 -1 WITHSCORES 
1) "A" 2) "100" 3) "B" 4) "80" 5) "C" 6) "20"
• ZREVRANGEBYSCORE key 100 50 
• 1) "A"
• 2) "B"
Leaderboards
# User A got 10 points
ZINCRBY scores 10 A
# User B got 15 points
 ZINCRBY scores 15 B
# User A got another 10 points
 ZINCRBY scores 10 A
# Display scores
ZREVRANGE scores 0 -1 WITHSCORES
1) "A" 2) "20" 3) "B" 4) "15"
Hashes
• Redis Hashes are maps string filed and string
values.
• This data type is perfect to represent an
object.
• A hash with few fields ( 100s ) is stored in a
way that takes very little space so you can
store millions of object is small REDIS
instances.
REDIS Hash commands
• HMSET user:1001 username john cid 1234
dept finance eaccount 5000
• HMGET user:1001 username cid ( Multi-field
GETs )
• HINCRBY user:1001 eaccount 5 ( Incrementing
specific field )
• HGET user:1001 eaccount ( Get specific field )
• HSETNX user:1001 username
REDIS - Persistence
• All datasets are stored in Memory like
Memcached, extremely fast read/write
operations.
• Datasets can be saved to Disk, Two ways
– RDB
– AOF ( Append only File )
RDB snapshots and AOF logs
• Persistence REDIS is a matter of configuration,
balancing the trade-off between performance,
disk I/O and data durability.
– RDB is a very compact single-file point-in-time
representation of REDIS dataset.
– AOF is a simple text log of write operations.
REDIS HA
• REDIS support Master/Slave configuration for HA
• Master can have multiple Slaves and Slaves can also be
connected other slaves in Graph like structure.
• REDIS replication is non-blocking on Master and Slave side.
• Replication is used both for scalability or data redundancy.
• To configure Slave is very simple just needs to add following to
slave configuration file,
Slaveof <ip-addr-of-server> 6379
• By default slaves are read only.
• What if Master fails ? How do client behave ? What are
options ?
Redis Scalability
• Client Side Sharding – Client shards key based on number of
Redis servers. This is static partitioning. Each Server is
configured with Master/Slave for redundancy.
• Server Side Sharding – twemproxy
(https://github.com/twitter/twemproxy) by Tweeter,
Consistent Hashing implementation. Another resource
http://antirez.com/news/44.
• Redis sever Pre-Sharding strategy - e.g Pinterest
• Great resource to start - http://redis.io/topics/partitioning.
• Many options.. Roll your Own which meets needs..
How to Monitor REDIS ?
• Local Monitoring : Monitd to monitor REDIS instance locally.
Restart if process fails. AS team has prototype for this.
• Remote Monitoring and Integration with Zenoss -
https://community.zenoss.org/docs/DOC-5333.
• Accessing REDIS metrics – Start redis-cli – info command with
following options, server, clients, memory, persistence, stats,
replication, cpu, commandstats, cluster,keyspace
• Many commercial SAAS vendors, DATADOG, LIBRATO, more
analysis and statistical Metrics collection.
REDIS as a Service
• Redislabs - http://redislabs.com/
• Amazon Elastic Cache service -
http://aws.amazon.com/about-aws/whats-
new/2013/09/04/amazon-elasticache-for-
redis/
How to Measure REDIS performance ?
• Redis-benchmark is a tool which comes
bundled with Redis installation.
• Check out at
http://redis.io/topics/benchmarks.
• This tool simulates N client sending M queries.
• Let us run.. See what happens.
Other feature of REDIS
• Transactions
– MULTI,EXEC,DISCARD AND WATCH are the transactions in Redis.
– Redis transactions are atomic.
– Redis transactions are started with MULTI command and then all
commands are executed once EXEC is executed.
• LUA scripting
– Lua scripting is used to implement custom command and/or
extension. This is very similar to stored procedure in RDBMS.
– Redis Lua interpreter loads seven libraries:
base,string,table,math,debug,cjson and cmsgpack.
Summary
• Redis offers excellent caching and persistence
options along with various data structure for
applications.
• This is true Lego building block for a service
developer. Use case is only limited by
imagination.
• At least start with, replacing Memcached.
• Excellent Live Documentation..check it out
http://redis.io
Q&A

REDIS327

  • 1.
    REDIS – AdvancedKey Value Store Rajan Bhatt
  • 2.
    What REDIS is? • REDIS is an open source, BSD licensed, advanced key-value store. It is often referred to as data structure server since keys can contain String, Hashes, List, Sets and Sorted sets.. • REDIS stands for REmote DIctionary Server
  • 3.
    REDIS - Features •Simplicity • Speed • Low Footprint • Versatility • Predictability • Reliability
  • 4.
    How to Installand Start REDIS server, REDIS client and Bindings • Go to http://redis.io/download and download • Start Redis server ( TCP server, Single Threaded, Listens on default port 6379 ). • Command – redis-server <path to conf. file ) • Bundled REDIS client – redis-cli • Client bindings available for all popular languages, Java, Python, Ruby, Scala, Ruby, Eralang, Go ..( I am running out of space )
  • 5.
    Data Structure • Strings •List • Sets • Sorted Set • Hash • And Publish/Subscribe Design Pattern
  • 6.
    Strings • Set Key“Value” ( 4 billion Keys ) • Get Key returns “Value” • Del Key • Fetch Multiple Keys at once … MGET • Set EXPIRATION key, e.g. Expire Key 5 • Command TTL returns remaining time before key expires
  • 7.
    Use Case forExpire, TTL • Cache – LRU http://antirez.com/post/redis-as-LRU- cache.html • Storage for Session ( HTTP or others ..)
  • 8.
    Atomic Counter • GETKey  Nil • INCR Key 1 • INCR Key 2 • Get Key  2
  • 9.
    Usage of INCR •Great Use case Global Counters ( Download, Hits, Votes ..) • INCR downloads:total • INCR downloads:total:today • INCR downloads:total:2011-05-10 • INCR downloads:/downloads/file1.mpg:total • INCR downloads:/downloads/file1.mpg:today • INCR downloads:/downloads/file1.mpg:2011-05- 10
  • 10.
    Counters # Total downloadsfor server, all time • GET downloads:total # Total downloads for server, today • GET downloads:total:today # Total downloads for file • GET downloads:/downloads/file1.mpg:total # Total downloads for file today • GET downloads:/downloads/file1.mpg:today
  • 11.
    How do youreset daily counter ? # Expire at 2011-05-10 23:59:59 • EXPIREAT downloads:total:today 1305064799
  • 12.
    One more UseCase – API rate Limiting • $ curl http://api.example.com/list.json • INCR api:<TOKEN>:hits 1 • Pseudo-Code if INCR('api:abc123:hits') > LIMIT return 420 Enhance Your Calm end
  • 13.
    One more Usecase Generating unique IDs • INCR global:users_ids 1 • SET users:1:username "john" • INCR global:users_ids 2 • SET users:2:username "mary”
  • 14.
    List • LPUSH key1  1 • LPUSH Key 2  2 • LPUSH Key 3  3 • RPOP Key  What should I get ? • LPOP Key  What should I get ? • LLEN Key ( Length if List ) • LRANGE key 0 -1 • LTRIM 0 1
  • 15.
    Usage of List •Indexes ( List of comments ..) LPUSH article:comments <ID> Timelines (of all sorts: messages, logs, ...) LPUSH user:<ID>:inbox "message from Alice" LPUSH user:<ID>:inbox "message from Bob" # Limit the messages to 100 LTRIM user:<ID>:inbox 0 99 # Get last 10 messages LRANGE user:<ID>:inbox 0 9 # Get next 10 messages LRANGE user:<ID>:inbox 10 19
  • 16.
    List is “Queue” Publisher • RPUSH queue “task-1” • RPUSH queue “task-2” Worker ( blocks and waits for tasks, 0 means waits for ever ) • BLPOP queue 0
  • 17.
    Basic Set • SADDkey 1  1 • SADD key 2  2 • SADD key 3  3 • SMEMBERS key  “3”, “2”, “1” • SISMEMBER key 1  “1” • SISMEMBER key 5  “0” • SREM key 3  1
  • 18.
    Set Operations • SADDA 1 • SADD A 2 • SMEMBERS A  “1”, “2” • SADD B 1 • SADD B 3 • SMEMBERS  “1”, “3”
  • 19.
    Set Continues.. • SUNIONA B  “1”, “2”, “3” • SINTER A B  “1” • SDIFF A B  “2”
  • 20.
    Set Usages -Relationship • SADD users:A:follows B • SADD users:B:follows C • SADD users:B:follows D • SADD users:C:follows A • SADD users:C:follows D
  • 21.
    Set Relationship • Jointnetwork of A & B SUNION users:A:follows users:B:follows “C”,”D”,”B” • Common for A & B  SINTER users:B:follows users:C:follows [] • Unique to B compared to C  “C”
  • 22.
    Set – Friends& Followers # Whom “swid1” follows • SADD swid1:follows A • SADD swid1:follows B # Who are friends of “swid1” • SADD swid1:friend C • SADD swid1:friend B # Whom swid1 follows and he is friend • SINTER swid1:follows swid1:friend  “B” # Whom swid1 follows and he is not friend • SDIFF swid1:follows swid1:friend  “A” # Whom Swid1 is friend with but does not follow • SDIFF swid1:friend swid1:follows  “C”
  • 23.
    Sorted Set • ZADDkey 100 A • ZADD key 10 C • ZADD key 80 B • ZRANGE key 0 -1  “C”, “B”, “A” • ZREVANGE key 0 -1  “A”, “B”, “C” • ZINCRBY key 10 C  “20”
  • 24.
    Sorted Set -Continues • ZREVRANGE key 0 -1 WITHSCORES  1) "A" 2) "100" 3) "B" 4) "80" 5) "C" 6) "20" • ZREVRANGEBYSCORE key 100 50  • 1) "A" • 2) "B"
  • 25.
    Leaderboards # User Agot 10 points ZINCRBY scores 10 A # User B got 15 points  ZINCRBY scores 15 B # User A got another 10 points  ZINCRBY scores 10 A # Display scores ZREVRANGE scores 0 -1 WITHSCORES 1) "A" 2) "20" 3) "B" 4) "15"
  • 26.
    Hashes • Redis Hashesare maps string filed and string values. • This data type is perfect to represent an object. • A hash with few fields ( 100s ) is stored in a way that takes very little space so you can store millions of object is small REDIS instances.
  • 27.
    REDIS Hash commands •HMSET user:1001 username john cid 1234 dept finance eaccount 5000 • HMGET user:1001 username cid ( Multi-field GETs ) • HINCRBY user:1001 eaccount 5 ( Incrementing specific field ) • HGET user:1001 eaccount ( Get specific field ) • HSETNX user:1001 username
  • 28.
    REDIS - Persistence •All datasets are stored in Memory like Memcached, extremely fast read/write operations. • Datasets can be saved to Disk, Two ways – RDB – AOF ( Append only File )
  • 29.
    RDB snapshots andAOF logs • Persistence REDIS is a matter of configuration, balancing the trade-off between performance, disk I/O and data durability. – RDB is a very compact single-file point-in-time representation of REDIS dataset. – AOF is a simple text log of write operations.
  • 30.
    REDIS HA • REDISsupport Master/Slave configuration for HA • Master can have multiple Slaves and Slaves can also be connected other slaves in Graph like structure. • REDIS replication is non-blocking on Master and Slave side. • Replication is used both for scalability or data redundancy. • To configure Slave is very simple just needs to add following to slave configuration file, Slaveof <ip-addr-of-server> 6379 • By default slaves are read only. • What if Master fails ? How do client behave ? What are options ?
  • 31.
    Redis Scalability • ClientSide Sharding – Client shards key based on number of Redis servers. This is static partitioning. Each Server is configured with Master/Slave for redundancy. • Server Side Sharding – twemproxy (https://github.com/twitter/twemproxy) by Tweeter, Consistent Hashing implementation. Another resource http://antirez.com/news/44. • Redis sever Pre-Sharding strategy - e.g Pinterest • Great resource to start - http://redis.io/topics/partitioning. • Many options.. Roll your Own which meets needs..
  • 32.
    How to MonitorREDIS ? • Local Monitoring : Monitd to monitor REDIS instance locally. Restart if process fails. AS team has prototype for this. • Remote Monitoring and Integration with Zenoss - https://community.zenoss.org/docs/DOC-5333. • Accessing REDIS metrics – Start redis-cli – info command with following options, server, clients, memory, persistence, stats, replication, cpu, commandstats, cluster,keyspace • Many commercial SAAS vendors, DATADOG, LIBRATO, more analysis and statistical Metrics collection.
  • 33.
    REDIS as aService • Redislabs - http://redislabs.com/ • Amazon Elastic Cache service - http://aws.amazon.com/about-aws/whats- new/2013/09/04/amazon-elasticache-for- redis/
  • 34.
    How to MeasureREDIS performance ? • Redis-benchmark is a tool which comes bundled with Redis installation. • Check out at http://redis.io/topics/benchmarks. • This tool simulates N client sending M queries. • Let us run.. See what happens.
  • 35.
    Other feature ofREDIS • Transactions – MULTI,EXEC,DISCARD AND WATCH are the transactions in Redis. – Redis transactions are atomic. – Redis transactions are started with MULTI command and then all commands are executed once EXEC is executed. • LUA scripting – Lua scripting is used to implement custom command and/or extension. This is very similar to stored procedure in RDBMS. – Redis Lua interpreter loads seven libraries: base,string,table,math,debug,cjson and cmsgpack.
  • 36.
    Summary • Redis offersexcellent caching and persistence options along with various data structure for applications. • This is true Lego building block for a service developer. Use case is only limited by imagination. • At least start with, replacing Memcached. • Excellent Live Documentation..check it out http://redis.io
  • 37.