Building Scalable, Distributed Job Queues with Redis and Redis::Client
Upcoming SlideShare
Loading in...5
×
 

Building Scalable, Distributed Job Queues with Redis and Redis::Client

on

  • 12,895 views

Presentation at YAPC NA 2012 by Mike Friedman

Presentation at YAPC NA 2012 by Mike Friedman

Statistics

Views

Total Views
12,895
Slideshare-icon Views on SlideShare
12,890
Embed Views
5

Actions

Likes
15
Downloads
104
Comments
0

3 Embeds 5

http://pinterest.com 3
https://twitter.com 1
http://www.docshut.com 1

Accessibility

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Building Scalable, Distributed Job Queues with Redis and Redis::Client Building Scalable, Distributed Job Queues with Redis and Redis::Client Presentation Transcript

  • Building Scalable,Distributed Job Queues with Redis and Redis::Client Mike Friedman (friedo) YAPC NA 2012
  • What Redis is• Open-source• Key-value store• Simple• Fast
  • What Redis isn’t• Relational (e.g. MySQL, Postgres, Oracle)• Document store (e.g. MongoDB)• Structured (e.g. Cassandra)• HTTP
  • Redis Data Types• Strings • e.g. ‘foo’, ’42’, or a JSON blob • Think Perl scalars
  • Redis Data Types• Lists • Zero or more strings, ordered • RPUSH, RPOP, LPUSH, and LPOP == push, pop, unshift, and shift in Perl
  • Redis Data Types• Hashes • Zero or more key-value pairs, unordered • HDEL, HEXISTS, HKEYS, and HVALS == delete, exists, keys, and values in Perl
  • Redis Data Types• Some types are not directly analogous to Perl concepts
  • Redis Data Types• Sets • Zero or more keys, unordered • No values • Think of a Perl hash where all values are undef
  • Redis Data Types• Sets • SREM, SISMEMBER, SMEMBERS == delete, exists, and keys in Perl
  • Redis Data Types• Set operations • Union • Intersection • Add / remove • Cardinality
  • Redis Data Types• Set operations can be implemented with Perl hashes• How?
  • Redis Data Types• Sorted Sets (zsets) • Zero or more keys, each with a numeric “score” • No values • Can be modeled as a Perl hash where the values are the scores
  • Redis Data Types• Sorted Sets (zsets) • ZREM, ZRANK, and ZRANGE (loosely) == delete, exists, and keys in Perl
  • Redis Data Types• Sorted set operations • Cardinality (within ranges) • Union / intersection • Lots more
  • Redis Data Types• Many more commands for working with Redis types• See http://redis.io/commands for the full list
  • Redis Protocol• Not HTTP• Stateful, interactive protocol
  • Redis Protocol• Actually, there are TWO protocols
  • Redis Protocol• Old Protocol• Client commands sent one per line• Server responses sent immediately
  • Redis Protocol• Old Protocol example: SET mystring foobar +OK
  • Redis Protocol• Old Protocol problems: • Command and data separated with whitespace • Difficult to parse - escaping becomes an issue • Difficult to deal with binary data or encoded text • Inconsistent
  • Redis Protocol• New Protocol (Unified Request Protocol) • Also called the URP
  • Redis Protocol• URP • Consistent command syntax • All data are prefixed with a byte length • No escaping or encoding/decoding required • Binary round-trip safe
  • Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
  • Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
  • Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
  • Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
  • Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
  • Redis Dists on CPAN
  • Redis Dists on CPAN• Information as of when I began work on Redis::Client, so some of this may have changed by now.
  • Redis Dists on CPAN• Redis.pm • Simple interface; works • Does some odd things with encoding • No newer Redis types like hashes • AUTOLOAD :(
  • Redis Dists on CPAN• Redis.pm • Forces UTF-8 flag on returned data by default • This is horribly broken • It will be fixed
  • Redis Dists on CPAN• Redis::hiredis • Wrapper around hiredis binary client • Works well if you have hiredis available • External binary dependency • Slow IPC
  • Redis Dists on CPAN• There was no Perl Redis module with: • Native support for Redis hashes • No outside binary dependencies • Full URP Support
  • Redis Dists on CPAN Until now!
  • Redis Dists on CPAN (Applause)
  • Redis Dists on CPAN (Applause)
  • Redis::Client• On CPAN • https://metacpan.org/release/Redis-Client
  • Redis::Client• On Github • https://github.com/friedo/perl-redis-client
  • Redis::Client• Distribution structure • Redis/Client.pm • Primary interface, implements the Redis commands
  • Redis::Client• Distribution structure • Redis/Client/Role/URP.pm • Moose role; implements the Unified Request Protocol • Abstract enough to be used by other projects
  • Redis::Client• Distribution structure • Redis/Client/Role/Tied.pm • Moose role; common functionality for classes which map Redis types to Perl objects
  • Redis::Client• Distribution structure • Redis/Client/*.pm • Tied object classes for Redis strings, lists, hashes, sets, and sorted sets
  • Redis::Client• Distribution structure • Redis/Client/String.pm • Redis/Client/List.pm • Redis/Client/Hash.pm • Redis/Client/Set.pm • Redis/Client/Zset.pm
  • Redis::Client• Encoding Caveats • Redis::Client makes no assumptions about your data encoding • Character data MUST be encoded prior to being sent to Redis • Redis URP relies on accurate BYTE counts, NOT character counts • Data returned from Redis NOT decoded
  • Redis::Client• Future features • Callback registration system for • Encoding and decoding • Inflating and deflating
  • Redis::Client Examplesmy $redis = Redis::Client->new;# store a string$redis->set( mystring => ‘foo’ );# retrieve stringmy $str = $redis->get(‘mystring’);
  • Redis::Client Examples# work with lists$redis->lpush( ‘mylist’, ‘one’, ‘two’, ‘three’);my $tail = $redis->rpop(‘mylist’);# three
  • Redis::Client Examples# work with hashes$redis->hset( ‘myhash’, key => 42);my $val = $redis->hget( ‘myhash’, ‘key’);# 42
  • Redis::Client Examples# work with tied classestie my %hash, ‘Redis::Client::Hash’, key => ‘myhash’, client => $redis;my @keys = keys %hash;if ( exists $hash{foo} ) { ... }delete $hash{some_key};while ( my ( $k, $v ) = each %hash ) { ... }# etc.
  • Job Queues• Goals • Add jobs with arbitrary data to queue • Fetch and execute jobs as soon as possible • Prevent duplicate job execution • Thousands of jobs per hour
  • Job Queues• Old Model • Jobs table in relational DB • INSERT to add job • Poll DB to find new jobs • Set a status field when job is running • Transactions to prevent duplicates • Set status to ‘done’ or ‘error’ for historical data
  • Job Queues• Problems: • Relational DB is slow • Jobs table grows quickly; not scalable • Wrong tool for the job
  • Job Queues• Using Redis: • A single Redis list implements a queue • LPOP (shift) jobs off the front of the queue • RPUSH (push) jobs onto the end of the queue • Use JSON to store job arguments and metadata
  • Job Queues• Redis blocking push/pop • LPOP returns undef if no jobs • BLPOP command blocks until an item to shift exists • No need to poll server; just wait
  • Job Queues# Simple Example of Job / Dispatcheruse Redis::Client;use JSON::XS;use Module::Load;use TryCatch;my $redis = Redis::Client->new;
  • Job Queues# Add a jobmy $job = { class => ‘Some::Job’, method => ‘do_something’, constructor_args => { foo => 42 }, method_args => { bar => 43 } };$redis->lpush( jobs => encode_json $job );
  • Job Queues# Simple dispatcher loopmy $job_str = $redis->blpop(‘jobs’);my $job = decode_json $job_str;my %c_args = %{ $job->{constructor_args} };my $m_args = %{ $job->{method_args} };my $class = ‘MyApp::’ . $job->{class};my $meth = $job->{method};
  • Job Queuesload $class;my $obj = $class->new( %c_args );try { $obj->$meth( %m_args );} catch ( $err ) { # store error data --> relational DB};# store success data --> relational DB
  • Benchmarks• Disclaimer: • These are bad benchmarks • Dependent on highly system-specific architecture • Didn’t try very hard • Do your own evaluation
  • Benchmarks• Old system (MySQL jobs table with polling, 5,000,000 dummy jobs populated) 100 1 Worker 10 Workers Workers 88 jobs/ 710 jobs/ 3892 jobs/ minute minute minute avg. avg. avg.
  • Benchmarks• New system (Redis queue, historical data only in MySQL, 5,000,000 dummy jobs populated 100 1 Worker 10 Workers Workers 92 jobs/ 1011 jobs/ 9222 jobs/ minute minute minute avg. avg. avg.
  • Caveats• These benchmarks are stupid• Highly specific test; do your own tests• Most job time is spent doing the job, not talking to the dispatcher• But it works for my purposes
  • Thank YouHave fun with Redis and Redis::Client Mike Friedman friedo@friedo.com
  • Questions