Redis in Practice

Noah Davis
Noah DavisCo-Founder at Code Climate
Redis in Practice
NoSQL NYC • December 2nd, 2010
    Noah Davis & Luke Melia
About Noah & Luke

Weplay.com
Rubyists
Running Redis in production since mid-2009
@noahd1 and @lukemelia on Twitter
Pronouncing Redis
Tonight’s gameplan
Fly-by intro to Redis
Redis in Practice
   View counts                     Q&A
                           throughout, please.
   Global locks               We love being
   Presence                    interrupted.

   Social activity feeds
   Friend suggestions
   Caching
Salvatore
     Sanfilippo
 Original author of Redis.
   @antirez on Twitter.
       Lives in Italy.
Recently hired by VMWare.
  Great project leader.
Describing Redis

Remote dictionary server
“advanced, fast, persistent key- value database”
“Data structures server”
“memcached on steroids”
In Salvatore’s words

 “I see Redis definitely more as a flexible tool
than as a solution specialized to solve a specific
  problem: his mixed soul of cache, store, and
    messaging server shows this very well.”
             - Salvatore Sanfilippo
Qualities
Written in C
Few dependencies
Fast
Single-threaded
Lots of clients available
Initial release was March 2009
Features
Key-value store where a value is one of:
   scalar/string, list, set, sorted sets, hash
Persistence: in-memory, snapshots, append-only log, VM
Replication: master-slave, configureable
Pub-Sub
Expiry
“Transaction”-y
In development: Redis Cluster
What Redis ain’t (...yet)

Big data
Seamless scaling
Ad-hoc query tool
The only data store you’ll ever need
Who’s using Redis?
VMWare
                 Grooveshark
Github
                 Superfeedr
Craigslist
                 Ravelry
EngineYard
                 mediaFAIL
The Guardian
                 Weplay
Forrst
                 More...
PostRank
Our infrastructure
App server           App server   Utility server




      MySQL master                   Redis master




       MySQL slave                   Redis slave
Redis in Practice #1

View Counts
View counts

potential for massive amounts of simple writes/reads
valuable data, but not mission critical
lots of row contention in SQL
Redis incrementing
    $ redis-cli
    redis> incr "medium:301:views"
    (integer) 1
    redis> incr "medium:301:views"
    (integer) 2
    redis> incrby "medium:301:views" 5
    (integer) 7
    redis> decr "medium:301:views"
    (integer) 6
Redis in Practice #2

Distributed Locks
Subscribe to a Weplay Calendar




                        Subscribe to “ics”
Subscription challenges


Generally static content for long periods of time, but with
short sessions of frequent updates
Expensive to compute on the fly
Without Redis/Locking
     CANCEL EVENT                 BACKGROUND                  Pubisher
                                     QUEUE


                Queue: Publish ICS



                                                  Publish




                                                                     Contention/Redundancy


       ADD EVENT                     BACKGROUND                Pubisher
                                        QUEUE


                    Queue: Publish ICS



                                                    Publish
Redis Locking: SetNX
  redis = Redis.new
  redis.setnx "locking.key", Time.now + 2.hours
  => true
  redis.setnx "locking.key", Time.now + 2.hours
  => false
  redis.del "locking.key"
  => true¨
  redis.setnx "locking.key", Time.now + 2.hours
  => true
Redis: Obtained Lock
ADD EVENT                                  REDIS




            SETNX "group_34_publish_ics"



                   Returns: 1




                                       BACKGROUND                              PUBLISHER                       REDIS
                                          QUEUE




                                                    5 minutes later: PUBLISH....
        Queue Job: Publish.publish_ics("34")
                                                                                      DEL "group_34_publish_ics"
Redis: Locked Out
   CANCEL EVENT                                                  REDIS




                         SETNX "group_34_publish_ics"

                                Returns: 0




                  ADD EVENT                                              REDIS




                                       SETNX "group_34_publish_ics"

                                              Returns: 0
Redis in Practice #3

 Presence
“Who’s online?”
Weplay members told us
they wanted to be able
to see which of their
friends were online...
About sets
0 to N elements       Adding a value to a set
                      does not require you
Unordered
                      to check if the value
No repeated members   exists in the set first
Working with Redis sets 1/3
# SADD key, member
# Adds the specified member to the set stored at key

redis = Redis.new
redis.sadd 'my_set', 'foo'   #   =>   true
redis.sadd 'my_set', 'bar'   #   =>   true
redis.sadd 'my_set', 'bar'   #   =>   false
redis.smembers 'my_set'      #   =>   ["foo", "bar"]
Working with Redis sets 2/3
# SUNION key1 key2 ... keyN
# Returns the members of a set resulting from the union of all
# the sets stored at the specified keys.

# SUNIONSTORE <i>dstkey key1 key2 ... keyN</i></b>
# Works like SUNION but instead of being returned the resulting
# set is stored as dstkey.

redis = Redis.new
redis.sadd 'set_a', 'foo'
redis.sadd 'set_a', 'bar'
redis.sadd 'set_b', 'bar'
redis.sadd 'set_b', 'baz'
redis.sunion 'set_a', 'set_b'        # => ["foo", "baz", "bar"]

redis.sunionstore 'set_ab', 'set_a', 'set_b'
redis.smembers 'set_ab'              # => ["foo", "baz", "bar"]
Working with Redis sets 3/3
# SINTER key1 key2 ... keyN
# Returns the members that are present in all
# sets stored at the specified keys.

redis = Redis.new
redis.sadd 'set_a', 'foo'
redis.sadd 'set_a', 'bar'
redis.sadd 'set_b', 'bar'
redis.sadd 'set_b', 'baz'
redis.sinter 'set_a', 'set_b'      # => ["bar"]
Approach 1/2
Approach 2/2
Implementation 1/2
# Defining the keys

def current_key
  key(Time.now.strftime("%M"))
end

def keys_in_last_5_minutes
  now = Time.now
  times = (0..5).collect {|n| now - n.minutes }
  times.collect{ |t| key(t.strftime("%M")) }
end

def key(minute)
  "online_users_minute_#{minute}"
end
Implementation 2/2
# Tracking an Active User, and calculating who’s online

def track_user_id(id)
  key = current_key
  redis.sadd(key, id)
end

def online_user_ids
  redis.sunion(*keys_in_last_5_minutes)
end

def online_friend_ids(interested_user_id)
  redis.sunionstore("online_users", *keys_in_last_5_minutes)
  redis.sinter("online_users",
               "user:#{interested_user_id}:friend_ids")
end
Redis in Practice #4

Social Activity Feeds
Social Activity Feeds
Via SQL, Approach 1/2
 Doesn’t scale to more
 complex social graphs
 e.g. friends who are         SELECT activities.*
                                FROM activities
                                JOIN friendships f1 ON f1.from_id =
 ‘hidden’ from your feed      activities.actor_id
                                JOIN friendships f2 ON f2.to_id   =
                              activities.actor_id
 May still require multiple     WHERE f1.to_id = ? OR f2.from_id = ?
                               ORDER BY activities.id DESC

 queries to grab               LIMIT 15



 unindexed data
Via SQL, Approach 2/2                             user_id



                                                   11
                                                            activity_id



                                                               96

                                                   22          96

                                                   11          97

                                                   22          97

                                                   33          97

                                                   11          98

                                                   22          98

                          Friend: 11               11          99

    Activity: 100
                                                   11        100
               Actor 1    Friend: 22
                                        INSERTS
                                                   22        100
                         Teammate: 33
                                                   33        100
Large SQL table

Grew quickly
Difficult to maintain
Difficult to prune
Redis Lists 1/2
      redis> lpush "teams" "yankees"
      (integer) 1
      redis> lpush "teams" "redsox"
      (integer) 2
      redis> llen "teams"
      (integer) 2
      redis> lrange "teams" 0 1
      1. "redsox"
      2. "yankees"
      redis> lrange "teams" 0 -1
      1. "redsox"
      2. "yankees"
                                       LTRIM, LLEN, LRANGE
Redis Lists 2/2
      redis> ltrim "teams" 0 0
      OK
      redis> lrange "teams" 0 -1
      1. "redsox"




                                   LTRIM
via Redis, 1/2
                            LPUSH “user:11:feed”, “100”
                            LTRIM “user:11:feed”, 0, 50
                                     LPUSH 'user:11:feed', '100'        Key             Value
                      Friend: 11
                                     LTRIM 'user:11:feed', 0, 50

Activity: 100
                                                                    user:11:feed   [100,99,97,96]
                                     LPUSH 'user:22:feed', '100'
           Actor 1    Friend: 22
                                     LTRIM 'user:22:feed', 0, 50
                                                                    user:22:feed    [100,99,98]

                                     LPUSH 'user:33:feed', '100'
                     Teammate: 33
                                     LTRIM 'user:33:feed', 0, 100   user:33:feed      [100,99]
via Redis, 2/2
                                          Key             Value
                       Friend: 11


                                      user:11:feed   [100,99,97,96]

                       Friend: 22
                                     user:22:feed     [100,99,98]
 Activity: 100


            Actor 1
                                     user:33:feed       [100,99]
                      Teammate: 33




                                          Key             Value

                       New York
                                     new_york:feed      [100,67]


                         Boston       boston:feed       [100,99]
Rendering the Feed
Redis in Practice #5

Friend Suggestions
Friend Suggestions

suggest new connections based on
existing connections
expanding the social graph and
mirroring real world connections is key
The Concept

            Paul




Me         George   John




            Ringo
The Yoko Factor

            Paul




Me         George      John   Yoko




            Ringo
The Approach 1/2
           Sarah




                       2

           Frank           Donny




     Me            2

             Ted

                            Erika
The Approach 2/2
           Sarah




                   2

           Frank       Donny




     Me



             Ted
                   3
                        Erika




            Sue
ZSETS in Redis

“Sorted Sets”
Each member of the set has a score
Ordered by the score at all times
Friend Suggestions in Redis
zincrby “user:1:suggestions” 1 “donny”
zincrby “user:1:suggestions” 1 “donny”
                                          Sarah




                                          Frank   Donny




                                     Me




zincrby “user:1:suggestions” 1 “erika”
                                            Ted

                                                   Erika


zincrby “user:1:suggestions” 1 “erika”

     [ Donny   2   , Erika   2   ]
Friend Suggestions in Redis
zincrby “user:1:suggestions” 1 “erika”
                                          Sarah




                                          Frank   Donny




                                     Me



                                            Ted



zrevrange “user:1:suggestions” 0 1                 Erika




                                           Sue




     [ Erika   3   , Donny   2   ]
Redis in Practice #6

  Caching
Suitability for caching
Excellent match for managed denormalization
   ex. friendships, teams, teammates
Excellent match where you would benefit from persistence
and/or replication
Historically, not a good match for a “generational cache,” in
which you want to optimize memory use by evicting least-
recently used (LRU) keys
As an LRU cache
TTL-support since inception, but with unintuitive behavior
   Writing to volatile key replaced it and cleared the TTL
Redis 2.2 changes this behavior and adds key features:
   ability to write to, and update expiry of volatile keys
   maxmemory [bytes], maxmemory-policy [policy]
   policies: volatile-lru, volatile-ttl, volatile-random,
   allkeys-lru, allkeys-random
Easy to adapt



Namespaced Rack::Session, Rack::Cache, I18n and cache
Redis cache stores for Ruby web frameworks
   implementation is under 1,000 LOC
Contentious benchmarks
Any
questions?




                  Thanks!
    Follow us on Twitter: @noahd1 and @lukemelia
   Tell your friends in youth sports about Weplay.com
1 of 57

Recommended

Redis introduction by
Redis introductionRedis introduction
Redis introductionFederico Daniel Colombo Gennarelli
5.9K views22 slides
Introduction to Redis by
Introduction to RedisIntroduction to Redis
Introduction to RedisTO THE NEW | Technology
1.5K views41 slides
Introduction to Redis by
Introduction to RedisIntroduction to Redis
Introduction to RedisMaarten Smeets
3K views36 slides
An Introduction to Redis for Developers.pdf by
An Introduction to Redis for Developers.pdfAn Introduction to Redis for Developers.pdf
An Introduction to Redis for Developers.pdfStephen Lorello
113 views100 slides
An Introduction to REDIS NoSQL database by
An Introduction to REDIS NoSQL databaseAn Introduction to REDIS NoSQL database
An Introduction to REDIS NoSQL databaseAli MasudianPour
5.4K views17 slides
Introduction to redis by
Introduction to redisIntroduction to redis
Introduction to redisTanu Siwag
826 views58 slides

More Related Content

What's hot

Caching solutions with Redis by
Caching solutions   with RedisCaching solutions   with Redis
Caching solutions with RedisGeorge Platon
4.8K views14 slides
redis basics by
redis basicsredis basics
redis basicsManoj Kumar
366 views10 slides
Redis - Usability and Use Cases by
Redis - Usability and Use CasesRedis - Usability and Use Cases
Redis - Usability and Use CasesFabrizio Farinacci
1.1K views29 slides
Introduction to redis by
Introduction to redisIntroduction to redis
Introduction to redisNexThoughts Technologies
984 views21 slides
Redis cluster by
Redis clusterRedis cluster
Redis clusteriammutex
7K views17 slides
Redis Reliability, Performance & Innovation by
Redis Reliability, Performance & InnovationRedis Reliability, Performance & Innovation
Redis Reliability, Performance & InnovationRedis Labs
1.8K views94 slides

What's hot(20)

Caching solutions with Redis by George Platon
Caching solutions   with RedisCaching solutions   with Redis
Caching solutions with Redis
George Platon4.8K views
Redis cluster by iammutex
Redis clusterRedis cluster
Redis cluster
iammutex7K views
Redis Reliability, Performance & Innovation by Redis Labs
Redis Reliability, Performance & InnovationRedis Reliability, Performance & Innovation
Redis Reliability, Performance & Innovation
Redis Labs1.8K views
Google Cloud Platform monitoring with Zabbix by Max Kuzkin
Google Cloud Platform monitoring with ZabbixGoogle Cloud Platform monitoring with Zabbix
Google Cloud Platform monitoring with Zabbix
Max Kuzkin11.1K views
Redis Introduction by Alex Su
Redis IntroductionRedis Introduction
Redis Introduction
Alex Su1.8K views
Introduction to Redis by Arnab Mitra
Introduction to RedisIntroduction to Redis
Introduction to Redis
Arnab Mitra11K views
plProxy, pgBouncer, pgBalancer by elliando dias
plProxy, pgBouncer, pgBalancerplProxy, pgBouncer, pgBalancer
plProxy, pgBouncer, pgBalancer
elliando dias7.1K views
Abusing Microsoft Kerberos - Sorry you guys don't get it by Benjamin Delpy
Abusing Microsoft Kerberos - Sorry you guys don't get itAbusing Microsoft Kerberos - Sorry you guys don't get it
Abusing Microsoft Kerberos - Sorry you guys don't get it
Benjamin Delpy43.1K views
MySQL operator for_kubernetes by rockplace
MySQL operator for_kubernetesMySQL operator for_kubernetes
MySQL operator for_kubernetes
rockplace427 views
Designing Apache Hudi for Incremental Processing With Vinoth Chandar and Etha... by HostedbyConfluent
Designing Apache Hudi for Incremental Processing With Vinoth Chandar and Etha...Designing Apache Hudi for Incremental Processing With Vinoth Chandar and Etha...
Designing Apache Hudi for Incremental Processing With Vinoth Chandar and Etha...
HostedbyConfluent992 views
Cassandra Introduction & Features by DataStax Academy
Cassandra Introduction & FeaturesCassandra Introduction & Features
Cassandra Introduction & Features
DataStax Academy31.9K views
Scaling Redis To 1M Ops/Sec: Jane Paek by Redis Labs
Scaling Redis To 1M Ops/Sec: Jane PaekScaling Redis To 1M Ops/Sec: Jane Paek
Scaling Redis To 1M Ops/Sec: Jane Paek
Redis Labs784 views
A simple introduction to redis by Zhichao Liang
A simple introduction to redisA simple introduction to redis
A simple introduction to redis
Zhichao Liang2.9K views
MongoDB Replica Sets by MongoDB
MongoDB Replica SetsMongoDB Replica Sets
MongoDB Replica Sets
MongoDB4.5K views

Viewers also liked

Redis data modeling examples by
Redis data modeling examplesRedis data modeling examples
Redis data modeling examplesTerry Cho
19.9K views11 slides
Scaling Crashlytics: Building Analytics on Redis 2.6 by
Scaling Crashlytics: Building Analytics on Redis 2.6Scaling Crashlytics: Building Analytics on Redis 2.6
Scaling Crashlytics: Building Analytics on Redis 2.6Crashlytics
15.5K views41 slides
Redis Use Patterns (DevconTLV June 2014) by
Redis Use Patterns (DevconTLV June 2014)Redis Use Patterns (DevconTLV June 2014)
Redis Use Patterns (DevconTLV June 2014)Itamar Haber
12.9K views25 slides
High-Volume Data Collection and Real Time Analytics Using Redis by
High-Volume Data Collection and Real Time Analytics Using RedisHigh-Volume Data Collection and Real Time Analytics Using Redis
High-Volume Data Collection and Real Time Analytics Using Rediscacois
25.2K views75 slides
Redis data design by usecase by
Redis data design by usecaseRedis data design by usecase
Redis data design by usecaseKris Jeong
11.1K views38 slides
Kicking ass with redis by
Kicking ass with redisKicking ass with redis
Kicking ass with redisDvir Volk
39.1K views22 slides

Viewers also liked(7)

Redis data modeling examples by Terry Cho
Redis data modeling examplesRedis data modeling examples
Redis data modeling examples
Terry Cho19.9K views
Scaling Crashlytics: Building Analytics on Redis 2.6 by Crashlytics
Scaling Crashlytics: Building Analytics on Redis 2.6Scaling Crashlytics: Building Analytics on Redis 2.6
Scaling Crashlytics: Building Analytics on Redis 2.6
Crashlytics15.5K views
Redis Use Patterns (DevconTLV June 2014) by Itamar Haber
Redis Use Patterns (DevconTLV June 2014)Redis Use Patterns (DevconTLV June 2014)
Redis Use Patterns (DevconTLV June 2014)
Itamar Haber12.9K views
High-Volume Data Collection and Real Time Analytics Using Redis by cacois
High-Volume Data Collection and Real Time Analytics Using RedisHigh-Volume Data Collection and Real Time Analytics Using Redis
High-Volume Data Collection and Real Time Analytics Using Redis
cacois25.2K views
Redis data design by usecase by Kris Jeong
Redis data design by usecaseRedis data design by usecase
Redis data design by usecase
Kris Jeong11.1K views
Kicking ass with redis by Dvir Volk
Kicking ass with redisKicking ass with redis
Kicking ass with redis
Dvir Volk39.1K views
Everything you always wanted to know about Redis but were afraid to ask by Carlos Abalde
Everything you always wanted to know about Redis but were afraid to askEverything you always wanted to know about Redis but were afraid to ask
Everything you always wanted to know about Redis but were afraid to ask
Carlos Abalde26.9K views

Similar to Redis in Practice

REDIS intro and how to use redis by
REDIS intro and how to use redisREDIS intro and how to use redis
REDIS intro and how to use redisKris Jeong
7.2K views17 slides
Reinforcement Learning On Hundreds Of Thousands Of Cores: Henrique Pondedeoli... by
Reinforcement Learning On Hundreds Of Thousands Of Cores: Henrique Pondedeoli...Reinforcement Learning On Hundreds Of Thousands Of Cores: Henrique Pondedeoli...
Reinforcement Learning On Hundreds Of Thousands Of Cores: Henrique Pondedeoli...Redis Labs
127 views72 slides
Indexing thousands of writes per second with redis by
Indexing thousands of writes per second with redisIndexing thousands of writes per second with redis
Indexing thousands of writes per second with redispauldix
13.4K views138 slides
10 Ways to Scale with Redis - LA Redis Meetup 2019 by
10 Ways to Scale with Redis - LA Redis Meetup 201910 Ways to Scale with Redis - LA Redis Meetup 2019
10 Ways to Scale with Redis - LA Redis Meetup 2019Dave Nielsen
339 views50 slides
Redis — The AK-47 of Post-relational Databases by
Redis — The AK-47 of Post-relational DatabasesRedis — The AK-47 of Post-relational Databases
Redis — The AK-47 of Post-relational DatabasesKarel Minarik
23.5K views70 slides
Postgres & Redis Sitting in a Tree- Rimas Silkaitis, Heroku by
Postgres & Redis Sitting in a Tree- Rimas Silkaitis, HerokuPostgres & Redis Sitting in a Tree- Rimas Silkaitis, Heroku
Postgres & Redis Sitting in a Tree- Rimas Silkaitis, HerokuRedis Labs
2.4K views78 slides

Similar to Redis in Practice(20)

REDIS intro and how to use redis by Kris Jeong
REDIS intro and how to use redisREDIS intro and how to use redis
REDIS intro and how to use redis
Kris Jeong7.2K views
Reinforcement Learning On Hundreds Of Thousands Of Cores: Henrique Pondedeoli... by Redis Labs
Reinforcement Learning On Hundreds Of Thousands Of Cores: Henrique Pondedeoli...Reinforcement Learning On Hundreds Of Thousands Of Cores: Henrique Pondedeoli...
Reinforcement Learning On Hundreds Of Thousands Of Cores: Henrique Pondedeoli...
Redis Labs127 views
Indexing thousands of writes per second with redis by pauldix
Indexing thousands of writes per second with redisIndexing thousands of writes per second with redis
Indexing thousands of writes per second with redis
pauldix13.4K views
10 Ways to Scale with Redis - LA Redis Meetup 2019 by Dave Nielsen
10 Ways to Scale with Redis - LA Redis Meetup 201910 Ways to Scale with Redis - LA Redis Meetup 2019
10 Ways to Scale with Redis - LA Redis Meetup 2019
Dave Nielsen339 views
Redis — The AK-47 of Post-relational Databases by Karel Minarik
Redis — The AK-47 of Post-relational DatabasesRedis — The AK-47 of Post-relational Databases
Redis — The AK-47 of Post-relational Databases
Karel Minarik23.5K views
Postgres & Redis Sitting in a Tree- Rimas Silkaitis, Heroku by Redis Labs
Postgres & Redis Sitting in a Tree- Rimas Silkaitis, HerokuPostgres & Redis Sitting in a Tree- Rimas Silkaitis, Heroku
Postgres & Redis Sitting in a Tree- Rimas Silkaitis, Heroku
Redis Labs2.4K views
10 Ways to Scale Your Website Silicon Valley Code Camp 2019 by Dave Nielsen
10 Ways to Scale Your Website Silicon Valley Code Camp 201910 Ways to Scale Your Website Silicon Valley Code Camp 2019
10 Ways to Scale Your Website Silicon Valley Code Camp 2019
Dave Nielsen225 views
Amir Salihefendic: Redis - the hacker's database by it-people
Amir Salihefendic: Redis - the hacker's databaseAmir Salihefendic: Redis - the hacker's database
Amir Salihefendic: Redis - the hacker's database
it-people1.4K views
Extend Redis with Modules by Itamar Haber
Extend Redis with ModulesExtend Redis with Modules
Extend Redis with Modules
Itamar Haber2K views
WebClusters, Redis by Filip Tepper
WebClusters, RedisWebClusters, Redis
WebClusters, Redis
Filip Tepper1.2K views
Advanced Redis data structures by amix3k
Advanced Redis data structuresAdvanced Redis data structures
Advanced Redis data structures
amix3k16.2K views
Drupalcamp gent - Node access by Jasper Knops
Drupalcamp gent - Node accessDrupalcamp gent - Node access
Drupalcamp gent - Node access
Jasper Knops1.8K views
Service Discovery & Load-Balancing under Docker 1.12.0 @ Docker Meetup #22 by Ajeet Singh Raina
Service Discovery & Load-Balancing under Docker 1.12.0 @ Docker Meetup #22Service Discovery & Load-Balancing under Docker 1.12.0 @ Docker Meetup #22
Service Discovery & Load-Balancing under Docker 1.12.0 @ Docker Meetup #22
Ajeet Singh Raina2.8K views
Redis学习笔记 by yongboy
Redis学习笔记Redis学习笔记
Redis学习笔记
yongboy1.8K views
Redis SoCraTes 2014 by steffenbauer
Redis SoCraTes 2014Redis SoCraTes 2014
Redis SoCraTes 2014
steffenbauer3.5K views
Redispresentation apac2012 by Ankur Gupta
Redispresentation apac2012Redispresentation apac2012
Redispresentation apac2012
Ankur Gupta2K views

Recently uploaded

Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ... by
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...ShapeBlue
52 views10 slides
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ... by
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...ShapeBlue
48 views17 slides
The Power of Heat Decarbonisation Plans in the Built Environment by
The Power of Heat Decarbonisation Plans in the Built EnvironmentThe Power of Heat Decarbonisation Plans in the Built Environment
The Power of Heat Decarbonisation Plans in the Built EnvironmentIES VE
67 views20 slides
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online by
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineKVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineShapeBlue
154 views19 slides
Kyo - Functional Scala 2023.pdf by
Kyo - Functional Scala 2023.pdfKyo - Functional Scala 2023.pdf
Kyo - Functional Scala 2023.pdfFlavio W. Brasil
443 views92 slides
Confidence in CloudStack - Aron Wagner, Nathan Gleason - Americ by
Confidence in CloudStack - Aron Wagner, Nathan Gleason - AmericConfidence in CloudStack - Aron Wagner, Nathan Gleason - Americ
Confidence in CloudStack - Aron Wagner, Nathan Gleason - AmericShapeBlue
58 views9 slides

Recently uploaded(20)

Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ... by ShapeBlue
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...
ShapeBlue52 views
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ... by ShapeBlue
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
ShapeBlue48 views
The Power of Heat Decarbonisation Plans in the Built Environment by IES VE
The Power of Heat Decarbonisation Plans in the Built EnvironmentThe Power of Heat Decarbonisation Plans in the Built Environment
The Power of Heat Decarbonisation Plans in the Built Environment
IES VE67 views
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online by ShapeBlue
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineKVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online
ShapeBlue154 views
Confidence in CloudStack - Aron Wagner, Nathan Gleason - Americ by ShapeBlue
Confidence in CloudStack - Aron Wagner, Nathan Gleason - AmericConfidence in CloudStack - Aron Wagner, Nathan Gleason - Americ
Confidence in CloudStack - Aron Wagner, Nathan Gleason - Americ
ShapeBlue58 views
Digital Personal Data Protection (DPDP) Practical Approach For CISOs by Priyanka Aash
Digital Personal Data Protection (DPDP) Practical Approach For CISOsDigital Personal Data Protection (DPDP) Practical Approach For CISOs
Digital Personal Data Protection (DPDP) Practical Approach For CISOs
Priyanka Aash103 views
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N... by James Anderson
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
James Anderson142 views
Business Analyst Series 2023 - Week 4 Session 7 by DianaGray10
Business Analyst Series 2023 -  Week 4 Session 7Business Analyst Series 2023 -  Week 4 Session 7
Business Analyst Series 2023 - Week 4 Session 7
DianaGray10110 views
Extending KVM Host HA for Non-NFS Storage - Alex Ivanov - StorPool by ShapeBlue
Extending KVM Host HA for Non-NFS Storage -  Alex Ivanov - StorPoolExtending KVM Host HA for Non-NFS Storage -  Alex Ivanov - StorPool
Extending KVM Host HA for Non-NFS Storage - Alex Ivanov - StorPool
ShapeBlue56 views
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT by ShapeBlue
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBITUpdates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT
ShapeBlue138 views
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit... by ShapeBlue
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
ShapeBlue86 views
Backroll, News and Demo - Pierre Charton, Matthias Dhellin, Ousmane Diarra - ... by ShapeBlue
Backroll, News and Demo - Pierre Charton, Matthias Dhellin, Ousmane Diarra - ...Backroll, News and Demo - Pierre Charton, Matthias Dhellin, Ousmane Diarra - ...
Backroll, News and Demo - Pierre Charton, Matthias Dhellin, Ousmane Diarra - ...
ShapeBlue121 views
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O... by ShapeBlue
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...
ShapeBlue59 views
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue by ShapeBlue
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlueElevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue
ShapeBlue149 views
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue by ShapeBlue
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlueCloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue
ShapeBlue63 views
Future of AR - Facebook Presentation by Rob McCarty
Future of AR - Facebook PresentationFuture of AR - Facebook Presentation
Future of AR - Facebook Presentation
Rob McCarty54 views
NTGapps NTG LowCode Platform by Mustafa Kuğu
NTGapps NTG LowCode Platform NTGapps NTG LowCode Platform
NTGapps NTG LowCode Platform
Mustafa Kuğu287 views
Data Integrity for Banking and Financial Services by Precisely
Data Integrity for Banking and Financial ServicesData Integrity for Banking and Financial Services
Data Integrity for Banking and Financial Services
Precisely76 views

Redis in Practice

  • 1. Redis in Practice NoSQL NYC • December 2nd, 2010 Noah Davis & Luke Melia
  • 2. About Noah & Luke Weplay.com Rubyists Running Redis in production since mid-2009 @noahd1 and @lukemelia on Twitter
  • 4. Tonight’s gameplan Fly-by intro to Redis Redis in Practice View counts Q&A throughout, please. Global locks We love being Presence interrupted. Social activity feeds Friend suggestions Caching
  • 5. Salvatore Sanfilippo Original author of Redis. @antirez on Twitter. Lives in Italy. Recently hired by VMWare. Great project leader.
  • 6. Describing Redis Remote dictionary server “advanced, fast, persistent key- value database” “Data structures server” “memcached on steroids”
  • 7. In Salvatore’s words “I see Redis definitely more as a flexible tool than as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well.” - Salvatore Sanfilippo
  • 8. Qualities Written in C Few dependencies Fast Single-threaded Lots of clients available Initial release was March 2009
  • 9. Features Key-value store where a value is one of: scalar/string, list, set, sorted sets, hash Persistence: in-memory, snapshots, append-only log, VM Replication: master-slave, configureable Pub-Sub Expiry “Transaction”-y In development: Redis Cluster
  • 10. What Redis ain’t (...yet) Big data Seamless scaling Ad-hoc query tool The only data store you’ll ever need
  • 11. Who’s using Redis? VMWare Grooveshark Github Superfeedr Craigslist Ravelry EngineYard mediaFAIL The Guardian Weplay Forrst More... PostRank
  • 12. Our infrastructure App server App server Utility server MySQL master Redis master MySQL slave Redis slave
  • 13. Redis in Practice #1 View Counts
  • 14. View counts potential for massive amounts of simple writes/reads valuable data, but not mission critical lots of row contention in SQL
  • 15. Redis incrementing $ redis-cli redis> incr "medium:301:views" (integer) 1 redis> incr "medium:301:views" (integer) 2 redis> incrby "medium:301:views" 5 (integer) 7 redis> decr "medium:301:views" (integer) 6
  • 16. Redis in Practice #2 Distributed Locks
  • 17. Subscribe to a Weplay Calendar Subscribe to “ics”
  • 18. Subscription challenges Generally static content for long periods of time, but with short sessions of frequent updates Expensive to compute on the fly
  • 19. Without Redis/Locking CANCEL EVENT BACKGROUND Pubisher QUEUE Queue: Publish ICS Publish Contention/Redundancy ADD EVENT BACKGROUND Pubisher QUEUE Queue: Publish ICS Publish
  • 20. Redis Locking: SetNX redis = Redis.new redis.setnx "locking.key", Time.now + 2.hours => true redis.setnx "locking.key", Time.now + 2.hours => false redis.del "locking.key" => true¨ redis.setnx "locking.key", Time.now + 2.hours => true
  • 21. Redis: Obtained Lock ADD EVENT REDIS SETNX "group_34_publish_ics" Returns: 1 BACKGROUND PUBLISHER REDIS QUEUE 5 minutes later: PUBLISH.... Queue Job: Publish.publish_ics("34") DEL "group_34_publish_ics"
  • 22. Redis: Locked Out CANCEL EVENT REDIS SETNX "group_34_publish_ics" Returns: 0 ADD EVENT REDIS SETNX "group_34_publish_ics" Returns: 0
  • 23. Redis in Practice #3 Presence
  • 24. “Who’s online?” Weplay members told us they wanted to be able to see which of their friends were online...
  • 25. About sets 0 to N elements Adding a value to a set does not require you Unordered to check if the value No repeated members exists in the set first
  • 26. Working with Redis sets 1/3 # SADD key, member # Adds the specified member to the set stored at key redis = Redis.new redis.sadd 'my_set', 'foo' # => true redis.sadd 'my_set', 'bar' # => true redis.sadd 'my_set', 'bar' # => false redis.smembers 'my_set' # => ["foo", "bar"]
  • 27. Working with Redis sets 2/3 # SUNION key1 key2 ... keyN # Returns the members of a set resulting from the union of all # the sets stored at the specified keys. # SUNIONSTORE <i>dstkey key1 key2 ... keyN</i></b> # Works like SUNION but instead of being returned the resulting # set is stored as dstkey. redis = Redis.new redis.sadd 'set_a', 'foo' redis.sadd 'set_a', 'bar' redis.sadd 'set_b', 'bar' redis.sadd 'set_b', 'baz' redis.sunion 'set_a', 'set_b' # => ["foo", "baz", "bar"] redis.sunionstore 'set_ab', 'set_a', 'set_b' redis.smembers 'set_ab' # => ["foo", "baz", "bar"]
  • 28. Working with Redis sets 3/3 # SINTER key1 key2 ... keyN # Returns the members that are present in all # sets stored at the specified keys. redis = Redis.new redis.sadd 'set_a', 'foo' redis.sadd 'set_a', 'bar' redis.sadd 'set_b', 'bar' redis.sadd 'set_b', 'baz' redis.sinter 'set_a', 'set_b' # => ["bar"]
  • 31. Implementation 1/2 # Defining the keys def current_key   key(Time.now.strftime("%M")) end def keys_in_last_5_minutes   now = Time.now   times = (0..5).collect {|n| now - n.minutes }   times.collect{ |t| key(t.strftime("%M")) } end def key(minute)   "online_users_minute_#{minute}" end
  • 32. Implementation 2/2 # Tracking an Active User, and calculating who’s online def track_user_id(id)   key = current_key   redis.sadd(key, id) end def online_user_ids   redis.sunion(*keys_in_last_5_minutes) end def online_friend_ids(interested_user_id)   redis.sunionstore("online_users", *keys_in_last_5_minutes)   redis.sinter("online_users", "user:#{interested_user_id}:friend_ids") end
  • 33. Redis in Practice #4 Social Activity Feeds
  • 35. Via SQL, Approach 1/2 Doesn’t scale to more complex social graphs e.g. friends who are SELECT activities.* FROM activities JOIN friendships f1 ON f1.from_id = ‘hidden’ from your feed activities.actor_id JOIN friendships f2 ON f2.to_id = activities.actor_id May still require multiple WHERE f1.to_id = ? OR f2.from_id = ? ORDER BY activities.id DESC queries to grab LIMIT 15 unindexed data
  • 36. Via SQL, Approach 2/2 user_id 11 activity_id 96 22 96 11 97 22 97 33 97 11 98 22 98 Friend: 11 11 99 Activity: 100 11 100 Actor 1 Friend: 22 INSERTS 22 100 Teammate: 33 33 100
  • 37. Large SQL table Grew quickly Difficult to maintain Difficult to prune
  • 38. Redis Lists 1/2 redis> lpush "teams" "yankees" (integer) 1 redis> lpush "teams" "redsox" (integer) 2 redis> llen "teams" (integer) 2 redis> lrange "teams" 0 1 1. "redsox" 2. "yankees" redis> lrange "teams" 0 -1 1. "redsox" 2. "yankees" LTRIM, LLEN, LRANGE
  • 39. Redis Lists 2/2 redis> ltrim "teams" 0 0 OK redis> lrange "teams" 0 -1 1. "redsox" LTRIM
  • 40. via Redis, 1/2 LPUSH “user:11:feed”, “100” LTRIM “user:11:feed”, 0, 50 LPUSH 'user:11:feed', '100' Key Value Friend: 11 LTRIM 'user:11:feed', 0, 50 Activity: 100 user:11:feed [100,99,97,96] LPUSH 'user:22:feed', '100' Actor 1 Friend: 22 LTRIM 'user:22:feed', 0, 50 user:22:feed [100,99,98] LPUSH 'user:33:feed', '100' Teammate: 33 LTRIM 'user:33:feed', 0, 100 user:33:feed [100,99]
  • 41. via Redis, 2/2 Key Value Friend: 11 user:11:feed [100,99,97,96] Friend: 22 user:22:feed [100,99,98] Activity: 100 Actor 1 user:33:feed [100,99] Teammate: 33 Key Value New York new_york:feed [100,67] Boston boston:feed [100,99]
  • 43. Redis in Practice #5 Friend Suggestions
  • 44. Friend Suggestions suggest new connections based on existing connections expanding the social graph and mirroring real world connections is key
  • 45. The Concept Paul Me George John Ringo
  • 46. The Yoko Factor Paul Me George John Yoko Ringo
  • 47. The Approach 1/2 Sarah 2 Frank Donny Me 2 Ted Erika
  • 48. The Approach 2/2 Sarah 2 Frank Donny Me Ted 3 Erika Sue
  • 49. ZSETS in Redis “Sorted Sets” Each member of the set has a score Ordered by the score at all times
  • 50. Friend Suggestions in Redis zincrby “user:1:suggestions” 1 “donny” zincrby “user:1:suggestions” 1 “donny” Sarah Frank Donny Me zincrby “user:1:suggestions” 1 “erika” Ted Erika zincrby “user:1:suggestions” 1 “erika” [ Donny 2 , Erika 2 ]
  • 51. Friend Suggestions in Redis zincrby “user:1:suggestions” 1 “erika” Sarah Frank Donny Me Ted zrevrange “user:1:suggestions” 0 1 Erika Sue [ Erika 3 , Donny 2 ]
  • 52. Redis in Practice #6 Caching
  • 53. Suitability for caching Excellent match for managed denormalization ex. friendships, teams, teammates Excellent match where you would benefit from persistence and/or replication Historically, not a good match for a “generational cache,” in which you want to optimize memory use by evicting least- recently used (LRU) keys
  • 54. As an LRU cache TTL-support since inception, but with unintuitive behavior Writing to volatile key replaced it and cleared the TTL Redis 2.2 changes this behavior and adds key features: ability to write to, and update expiry of volatile keys maxmemory [bytes], maxmemory-policy [policy] policies: volatile-lru, volatile-ttl, volatile-random, allkeys-lru, allkeys-random
  • 55. Easy to adapt Namespaced Rack::Session, Rack::Cache, I18n and cache Redis cache stores for Ruby web frameworks implementation is under 1,000 LOC
  • 57. Any questions? Thanks! Follow us on Twitter: @noahd1 and @lukemelia Tell your friends in youth sports about Weplay.com

Editor's Notes

  1. \n
  2. \n
  3. \n
  4. \n
  5. \n
  6. \n
  7. \n
  8. \n
  9. \n
  10. \n
  11. \n
  12. \n
  13. \n
  14. \n
  15. \n
  16. \n
  17. \n
  18. \n
  19. \n
  20. \n
  21. \n
  22. \n
  23. \n
  24. \n
  25. \n
  26. \n
  27. \n
  28. \n
  29. \n
  30. \n
  31. \n
  32. \n
  33. \n
  34. \n
  35. \n
  36. \n
  37. \n
  38. \n
  39. \n
  40. \n
  41. \n
  42. \n
  43. \n
  44. \n
  45. \n
  46. \n
  47. \n
  48. \n
  49. \n
  50. \n
  51. \n
  52. \n
  53. \n
  54. http://antirez.com/m/p.php?i=220\nvolatile-lru remove a key among the ones with an expire set, trying to remove keys not recently used.\nvolatile-ttl remove a key among the ones with an expire set, trying to remove keys with short remaining time to live.\nvolatile-random remove a random key among the ones with an expire set.\nallkeys-lru like volatile-lru, but will remove every kind of key, both normal keys or keys with an expire set.\nallkeys-random like volatile-random, but will remove every kind of keys, both normal keys and keys with an expire set.\n\n
  55. Another 1500 LOC for specs\n
  56. \n
  57. \n