Redis, Resque and Friends               Christopher Spring         https://github.com/autonomous                  @autonom...
WTF is Redis?
WTF is Redis?Remote Dictionary Server
Key-value storage...   # Insert a value for a key   SET some:key “value”   # Retrieve value for a key   GET some:key # => ...
... on steroids!# ListsRPUSH chores "Mow lawn"LPUSH chores "Get dressed"# SetsSADD sith "Darth Maul"SUNION sith evil# Sort...
Strings
Hashes
Lists
Sets
Sorted sets
$ brew install redis$ redis-server /usr/local/etc/redis-conf
Example: Tagging$ redis-cli...redis> SADD post:17:tag "ruby"redis> SADD post:17:tag "rails"redis> SADD post:17:tag "redis"...
Transactions
Example: Transactions $ redis-cli ... redis> SET "player:1:coins" 30 redis> SET "player:2:coins" 63 # ... Players decide t...
In-memory storage • Simple single process event   driven design • No locks • 100K+ operations per second
Asynchronous persistence  • Snapshotting -   save 60 1000    # dump every 60 seconds if at least 1000    keys changed  • A...
Master-slave replication      slaveof 192.168.1.1 6379         • Multiple slaves         • Slaves of slaves         • Scal...
... I’ve got too much data!
Redis Virtual Memory  • Only values swapped to disk  • Most recent values in memory  • Predictable memory usage...
1M keys: 160 MB10M keys: 1.6 GB100M keys: 16 GB
MySQL + memcached  • Multiple copies of data  • Mainly scales reads  • No higher level functions
Client libraries for all:     •   C             •   Java     •   C#            •   Lua     •   C++           •   Node.js  ...
Client libraries for all:     •   C             •   Java     •   C#            •   Lua     •   C++           •   Node.js  ...
$ sudo gem install redis
Redis as a database...
require rubygems; require redis; require jsonclass RubyFuZa  def initialize   args                                      cl...
Publish/Subscribe
require rubygems                             $ redis-clirequire "redis"                                ...                ...
Resque• Backed by redis• Asynchronous job server• Multiple queues• Sinatra based web-ui
class PostalWorker  @queue :post_box  def self.perform( user_id )    User.where( :id => user_id )    msg = UserMail.welcom...
Resque-Scheduler• Queue jobs in the future!• Recurring cron-style queueing• Delayed jobs
database_cleaning:  cron: "0 0 * * *"  class: CleanupWorker  args:  description: "This jobs removes junk from the DB"
Resque.enqueue_at(  5.days.from_now,  SendFollowUpEmail,  :user_id => current_user.id)
Resque-retry• Retry failed jobs• Set number of retries• Set delay between retries• Exponential back-off• Delays built on r...
class WebHookWorker  extend Resque::Plugins::Retry  @queue = :web_hooks  @retry_limit = 10  @retry_delay = 5*60 # Seconds ...
In summary...
Data structure server      • Strings      • Hashes      • Lists      • Sets      • Sorted sets
Main use cases:    • Database    • Cache    • Messaging
Features:• Fast• Atomic operations• Transactions• Master-Slave• Persist to disk• Expiring keys• Pub - Sub• Redis Virtual M...
Baie dankie!
Questions?
Links• https://github.com/bblimke/copy-with-style-tmbundle• http://redis.io/• https://github.com/antirez/redis• https://gi...
Redis, Resque & Friends
Upcoming SlideShare
Loading in...5
×

Redis, Resque & Friends

6,270

Published on

RubyFuZa presentation giving an overview of redis with a couple of examples. Also quick mentions of Resque, Resque-Scheduler and Resque-Retry

Published in: Technology
0 Comments
11 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
6,270
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
110
Comments
0
Likes
11
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Redis, Resque & Friends

    1. 1. Redis, Resque and Friends Christopher Spring https://github.com/autonomous @autonomous www.christopherspring.com
    2. 2. WTF is Redis?
    3. 3. WTF is Redis?Remote Dictionary Server
    4. 4. Key-value storage... # Insert a value for a key SET some:key “value” # Retrieve value for a key GET some:key # => “value”
    5. 5. ... on steroids!# ListsRPUSH chores "Mow lawn"LPUSH chores "Get dressed"# SetsSADD sith "Darth Maul"SUNION sith evil# Sorted setsZADD background:workers 50 "w1.oss.com"ZADD background:workers 19 "w3.oss.com"# HashHMSET client username "womble" password "secret_womble"HGET client password
    6. 6. Strings
    7. 7. Hashes
    8. 8. Lists
    9. 9. Sets
    10. 10. Sorted sets
    11. 11. $ brew install redis$ redis-server /usr/local/etc/redis-conf
    12. 12. Example: Tagging$ redis-cli...redis> SADD post:17:tag "ruby"redis> SADD post:17:tag "rails"redis> SADD post:17:tag "redis"redis> SADD post:20:tag "resque"redis> SADD post:20:tag "redis"redis> SINTER post:17:tag post:20:tag1. "redis"
    13. 13. Transactions
    14. 14. Example: Transactions $ redis-cli ... redis> SET "player:1:coins" 30 redis> SET "player:2:coins" 63 # ... Players decide to trade some coin redis> MULTI redis> INCRBY "player:1:coins" 15 redis> DECRBY "player:2:coins" 15 redis> EXEC # 1. (integer) 45 # 2. (integer) 48
    15. 15. In-memory storage • Simple single process event driven design • No locks • 100K+ operations per second
    16. 16. Asynchronous persistence • Snapshotting - save 60 1000 # dump every 60 seconds if at least 1000 keys changed • Append-only file - durable with log rewrite support
    17. 17. Master-slave replication slaveof 192.168.1.1 6379 • Multiple slaves • Slaves of slaves • Scalability • Defer save to slaves
    18. 18. ... I’ve got too much data!
    19. 19. Redis Virtual Memory • Only values swapped to disk • Most recent values in memory • Predictable memory usage...
    20. 20. 1M keys: 160 MB10M keys: 1.6 GB100M keys: 16 GB
    21. 21. MySQL + memcached • Multiple copies of data • Mainly scales reads • No higher level functions
    22. 22. Client libraries for all: • C • Java • C# • Lua • C++ • Node.js • Clojure • Objective-c • Common List • Perl • Erlang • PHP • Go • Python • Haskell • Scala • haXe • Smalltalk • Io • Tcl
    23. 23. Client libraries for all: • C • Java • C# • Lua • C++ • Node.js • Clojure • Objective-c • Common List • Perl • Erlang • PHP • Go • Python • Haskell • Scala • haXe • Smalltalk • Io • Tcl
    24. 24. $ sudo gem install redis
    25. 25. Redis as a database...
    26. 26. require rubygems; require redis; require jsonclass RubyFuZa def initialize args class Dude < Struct.new(:first_name, :last_name); end @db = args[:db] @duration = args[:duration] # seconds @namespace = "#{self.class}:#{args[:id]}" chris = Dude.new Chris, Spring @s_key = "#{@namespace}:speakers" marc = Dude.new Marc, Heiligers @a_key = "#{@namespace}:attendees" steve = Dude.new Steve, Hoffmeier # Is jy my pa? end def start! redis = Redis.new return "Already started!" if started? two_days = 2*24*60*60 # seconds @db[@namespace] = started conf = RubyFuZa.new( @db.expire( @namespace, @duration ) :db => redis, end :duration => two_days, :id => "2011") def started? @db[@namespace] == started end conf.add_speaker chris # true conf.add_attendee marc # true def how_much_longer? conf.attendees # [chris, marc] t = @db.ttl(@namespace) t == -1 ? Its over! : t end conf.started? # false conf.start! # true def add_speaker( speaker ) conf.started? # true @db.sadd( @s_key, speaker.to_json ) end sleep( 2000 ) def add_attendee( attendee ) conf.how_much_longer? # two_days - 2000 @db.sadd(@a_key, attendee.to_json) end def attendees @db.sinter(@s_key, @a_key).map{ |sa| JSON.parse( sa )} endend
    27. 27. Publish/Subscribe
    28. 28. require rubygems $ redis-clirequire "redis" ... redis> PUBLISH rubyfuza:chat "Smoke me aredis = Redis.connect kipper..." redis> PUBLISH rubyfuza:chat "Ill be backtrap(:INT) { puts; exit } for breakfast." redis> PUBLISH rubyfuza:chat "exit"redis.subscribe(rubyfuza:chat) do |on| on.subscribe do |channel, subscriptions| puts "Subscribed to ##{channel}" end on.message do |channel, message| puts "##{channel}: #{message}" redis.unsubscribe if message == "exit" end on.unsubscribe do |channel, subscriptions| puts "Unsubscribed from ##{channel}" endend
    29. 29. Resque• Backed by redis• Asynchronous job server• Multiple queues• Sinatra based web-ui
    30. 30. class PostalWorker @queue :post_box def self.perform( user_id ) User.where( :id => user_id ) msg = UserMail.welcome_message( user ) msg.deliver endend...Resque.enqueue( PostalWorker, user.id )
    31. 31. Resque-Scheduler• Queue jobs in the future!• Recurring cron-style queueing• Delayed jobs
    32. 32. database_cleaning: cron: "0 0 * * *" class: CleanupWorker args: description: "This jobs removes junk from the DB"
    33. 33. Resque.enqueue_at( 5.days.from_now, SendFollowUpEmail, :user_id => current_user.id)
    34. 34. Resque-retry• Retry failed jobs• Set number of retries• Set delay between retries• Exponential back-off• Delays built on resque-scheduler
    35. 35. class WebHookWorker extend Resque::Plugins::Retry @queue = :web_hooks @retry_limit = 10 @retry_delay = 5*60 # Seconds def self.perform(*args) # ... trigger web-hook endend
    36. 36. In summary...
    37. 37. Data structure server • Strings • Hashes • Lists • Sets • Sorted sets
    38. 38. Main use cases: • Database • Cache • Messaging
    39. 39. Features:• Fast• Atomic operations• Transactions• Master-Slave• Persist to disk• Expiring keys• Pub - Sub• Redis Virtual Memory
    40. 40. Baie dankie!
    41. 41. Questions?
    42. 42. Links• https://github.com/bblimke/copy-with-style-tmbundle• http://redis.io/• https://github.com/antirez/redis• https://github.com/ezmobius/redis-rb• http://antirez.com/• https://github.com/defunkt/resque• https://github.com/bvandenbos/resque-scheduler• https://github.com/lantins/resque-retry
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×