Redis, Resque & Friends

Redis, Resque and Friends Christopher Spring https://github.com/autonomous @autonomous www.christopherspring.com

WTF is Redis?

WTF is Redis?Remote Dictionary Server

Key-value storage... # Insert a value for a key SET some:key “value” # Retrieve value for a key GET some:key # => “value”

... on steroids!# ListsRPUSH chores "Mow lawn"LPUSH chores "Get dressed"# SetsSADD sith "Darth Maul"SUNION sith evil# Sorted setsZADD background:workers 50 "w1.oss.com"ZADD background:workers 19 "w3.oss.com"# HashHMSET client username "womble" password "secret_womble"HGET client password

Strings

Hashes

Sorted sets

$ brew install redis$ redis-server /usr/local/etc/redis-conf

Example: Tagging$ redis-cli...redis> SADD post:17:tag "ruby"redis> SADD post:17:tag "rails"redis> SADD post:17:tag "redis"redis> SADD post:20:tag "resque"redis> SADD post:20:tag "redis"redis> SINTER post:17:tag post:20:tag1. "redis"

Transactions

Example: Transactions $ redis-cli ... redis> SET "player:1:coins" 30 redis> SET "player:2:coins" 63 # ... Players decide to trade some coin redis> MULTI redis> INCRBY "player:1:coins" 15 redis> DECRBY "player:2:coins" 15 redis> EXEC # 1. (integer) 45 # 2. (integer) 48

In-memory storage • Simple single process event driven design • No locks • 100K+ operations per second

Asynchronous persistence • Snapshotting - save 60 1000 # dump every 60 seconds if at least 1000 keys changed • Append-only ﬁle - durable with log rewrite support

Master-slave replication slaveof 192.168.1.1 6379 • Multiple slaves • Slaves of slaves • Scalability • Defer save to slaves

... I’ve got too much data!

Redis Virtual Memory • Only values swapped to disk • Most recent values in memory • Predictable memory usage...

1M keys: 160 MB10M keys: 1.6 GB100M keys: 16 GB

MySQL + memcached • Multiple copies of data • Mainly scales reads • No higher level functions

Client libraries for all: • C • Java • C# • Lua • C++ • Node.js • Clojure • Objective-c • Common List • Perl • Erlang • PHP • Go • Python • Haskell • Scala • haXe • Smalltalk • Io • Tcl

$ sudo gem install redis

Redis as a database...

require rubygems; require redis; require jsonclass RubyFuZa def initialize args class Dude < Struct.new(:first_name, :last_name); end @db = args[:db] @duration = args[:duration] # seconds @namespace = "#{self.class}:#{args[:id]}" chris = Dude.new Chris, Spring @s_key = "#{@namespace}:speakers" marc = Dude.new Marc, Heiligers @a_key = "#{@namespace}:attendees" steve = Dude.new Steve, Hoffmeier # Is jy my pa? end def start! redis = Redis.new return "Already started!" if started? two_days = 2*24*60*60 # seconds @db[@namespace] = started conf = RubyFuZa.new( @db.expire( @namespace, @duration ) :db => redis, end :duration => two_days, :id => "2011") def started? @db[@namespace] == started end conf.add_speaker chris # true conf.add_attendee marc # true def how_much_longer? conf.attendees # [chris, marc] t = @db.ttl(@namespace) t == -1 ? Its over! : t end conf.started? # false conf.start! # true def add_speaker( speaker ) conf.started? # true @db.sadd( @s_key, speaker.to_json ) end sleep( 2000 ) def add_attendee( attendee ) conf.how_much_longer? # two_days - 2000 @db.sadd(@a_key, attendee.to_json) end def attendees @db.sinter(@s_key, @a_key).map{ |sa| JSON.parse( sa )} endend

Publish/Subscribe

require rubygems $ redis-clirequire "redis" ... redis> PUBLISH rubyfuza:chat "Smoke me aredis = Redis.connect kipper..." redis> PUBLISH rubyfuza:chat "Ill be backtrap(:INT) { puts; exit } for breakfast." redis> PUBLISH rubyfuza:chat "exit"redis.subscribe(rubyfuza:chat) do |on| on.subscribe do |channel, subscriptions| puts "Subscribed to ##{channel}" end on.message do |channel, message| puts "##{channel}: #{message}" redis.unsubscribe if message == "exit" end on.unsubscribe do |channel, subscriptions| puts "Unsubscribed from ##{channel}" endend

Resque• Backed by redis• Asynchronous job server• Multiple queues• Sinatra based web-ui

class PostalWorker @queue :post_box def self.perform( user_id ) User.where( :id => user_id ) msg = UserMail.welcome_message( user ) msg.deliver endend...Resque.enqueue( PostalWorker, user.id )

Resque-Scheduler• Queue jobs in the future!• Recurring cron-style queueing• Delayed jobs

database_cleaning: cron: "0 0 * * *" class: CleanupWorker args: description: "This jobs removes junk from the DB"

Resque.enqueue_at( 5.days.from_now, SendFollowUpEmail, :user_id => current_user.id)

Resque-retry• Retry failed jobs• Set number of retries• Set delay between retries• Exponential back-off• Delays built on resque-scheduler

class WebHookWorker extend Resque::Plugins::Retry @queue = :web_hooks @retry_limit = 10 @retry_delay = 5*60 # Seconds def self.perform(*args) # ... trigger web-hook endend

In summary...

Data structure server • Strings • Hashes • Lists • Sets • Sorted sets

Main use cases: • Database • Cache • Messaging

Features:• Fast• Atomic operations• Transactions• Master-Slave• Persist to disk• Expiring keys• Pub - Sub• Redis Virtual Memory

Baie dankie!

Questions?

Links• https://github.com/bblimke/copy-with-style-tmbundle• http://redis.io/• https://github.com/antirez/redis• https://github.com/ezmobius/redis-rb• http://antirez.com/• https://github.com/defunkt/resque• https://github.com/bvandenbos/resque-scheduler• https://github.com/lantins/resque-retry

http://christopherspring.com	166
http://chrisspring.posterous.com	3
http://translate.googleusercontent.com	2
http://import-ydsf.posterous.com	1

Redis, Resque & Friends

by Christopher Spring on Feb 06, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 172

Statistics

Redis, Resque & Friends — Presentation Transcript