Building Scalable,Distributed Job Queues    with Redis and  Redis::Client       Mike Friedman          (friedo)      YAPC ...
What Redis is• Open-source• Key-value store• Simple• Fast
What Redis isn’t• Relational (e.g. MySQL, Postgres, Oracle)• Document store (e.g. MongoDB)• Structured (e.g. Cassandra)• H...
Redis Data Types• Strings • e.g. ‘foo’, ’42’, or a JSON blob • Think Perl scalars
Redis Data Types• Lists • Zero or more strings, ordered • RPUSH, RPOP, LPUSH, and LPOP ==   push, pop, unshift, and shift ...
Redis Data Types• Hashes • Zero or more key-value pairs, unordered   • HDEL, HEXISTS, HKEYS, and HVALS      == delete, exi...
Redis Data Types• Some types are not directly analogous to  Perl concepts
Redis Data Types• Sets • Zero or more keys, unordered • No values • Think of a Perl hash where all values are    undef
Redis Data Types• Sets • SREM, SISMEMBER, SMEMBERS ==   delete, exists, and keys in Perl
Redis Data Types• Set operations • Union • Intersection • Add / remove • Cardinality
Redis Data Types• Set operations can be implemented with  Perl hashes• How?
Redis Data Types• Sorted Sets (zsets) • Zero or more keys, each with a numeric    “score” • No values • Can be modeled as ...
Redis Data Types• Sorted Sets (zsets) • ZREM, ZRANK, and ZRANGE (loosely)   == delete, exists, and keys in Perl
Redis Data Types• Sorted set operations • Cardinality (within ranges) • Union / intersection • Lots more
Redis Data Types• Many more commands for working with  Redis types• See http://redis.io/commands for the full list
Redis Protocol• Not HTTP• Stateful, interactive protocol
Redis Protocol• Actually, there are TWO protocols
Redis Protocol• Old Protocol• Client commands sent one per line• Server responses sent immediately
Redis Protocol• Old Protocol example:  SET mystring foobar  +OK
Redis Protocol• Old Protocol problems: • Command and data separated with    whitespace • Difficult to parse - escaping beco...
Redis Protocol• New Protocol (Unified Request Protocol) • Also called the URP
Redis Protocol• URP • Consistent command syntax • All data are prefixed with a byte length • No escaping or encoding/decodi...
Redis Protocol• URP Example  *3  $3  SET  $8  mystring  $6  foobar  +OK
Redis Protocol• URP Example  *3  $3  SET  $8  mystring  $6  foobar  +OK
Redis Protocol• URP Example  *3  $3  SET  $8  mystring  $6  foobar  +OK
Redis Protocol• URP Example  *3  $3  SET  $8  mystring  $6  foobar  +OK
Redis Protocol• URP Example  *3  $3  SET  $8  mystring  $6  foobar  +OK
Redis Dists on CPAN
Redis Dists on CPAN• Information as of when I began work on  Redis::Client, so some of this may have  changed by now.
Redis Dists on CPAN• Redis.pm • Simple interface; works • Does some odd things with encoding • No newer Redis types like h...
Redis Dists on CPAN• Redis.pm • Forces UTF-8 flag on returned data by    default • This is horribly broken • It will be fixed
Redis Dists on CPAN• Redis::hiredis • Wrapper around hiredis binary client • Works well if you have hiredis    available •...
Redis Dists on CPAN• There was no Perl Redis module with: • Native support for Redis hashes • No outside binary dependenci...
Redis Dists on CPAN Until now!
Redis Dists on CPAN     (Applause)
Redis Dists on CPAN (Applause)
Redis::Client• On CPAN • https://metacpan.org/release/Redis-Client
Redis::Client• On Github • https://github.com/friedo/perl-redis-client
Redis::Client• Distribution structure • Redis/Client.pm   • Primary interface, implements the      Redis commands
Redis::Client• Distribution structure • Redis/Client/Role/URP.pm   • Moose role; implements the Unified      Request Protoc...
Redis::Client• Distribution structure • Redis/Client/Role/Tied.pm   • Moose role; common functionality for      classes wh...
Redis::Client• Distribution structure • Redis/Client/*.pm   • Tied object classes for Redis strings,      lists, hashes, s...
Redis::Client• Distribution structure • Redis/Client/String.pm • Redis/Client/List.pm • Redis/Client/Hash.pm • Redis/Clien...
Redis::Client• Encoding Caveats • Redis::Client makes no assumptions    about your data encoding • Character data MUST be ...
Redis::Client• Future features • Callback registration system for   • Encoding and decoding   • Inflating and deflating
Redis::Client Examplesmy $redis = Redis::Client->new;# store a string$redis->set( mystring => ‘foo’ );# retrieve stringmy ...
Redis::Client Examples# work with lists$redis->lpush(   ‘mylist’, ‘one’, ‘two’, ‘three’);my $tail = $redis->rpop(‘mylist’)...
Redis::Client Examples# work with hashes$redis->hset(   ‘myhash’, key => 42);my $val = $redis->hget(   ‘myhash’, ‘key’);# 42
Redis::Client Examples# work with tied classestie my %hash, ‘Redis::Client::Hash’,  key => ‘myhash’, client => $redis;my @...
Job Queues• Goals • Add jobs with arbitrary data to queue • Fetch and execute jobs as soon as    possible • Prevent duplic...
Job Queues• Old Model • Jobs table in relational DB • INSERT to add job • Poll DB to find new jobs • Set a status field when...
Job Queues• Problems: • Relational DB is slow • Jobs table grows quickly; not scalable • Wrong tool for the job
Job Queues• Using Redis: • A single Redis list implements a queue • LPOP (shift) jobs off the front of the    queue • RPUS...
Job Queues• Redis blocking push/pop • LPOP returns undef if no jobs • BLPOP command blocks until an item to    shift exist...
Job Queues# Simple Example of Job / Dispatcheruse   Redis::Client;use   JSON::XS;use   Module::Load;use   TryCatch;my $red...
Job Queues# Add a jobmy $job =  { class => ‘Some::Job’,    method => ‘do_something’,    constructor_args => { foo => 42 },...
Job Queues# Simple dispatcher loopmy $job_str = $redis->blpop(‘jobs’);my $job = decode_json $job_str;my %c_args  = %{ $job...
Job Queuesload $class;my $obj = $class->new( %c_args );try {    $obj->$meth( %m_args );} catch ( $err ) {    # store error...
Benchmarks• Disclaimer: • These are bad benchmarks • Dependent on highly system-specific    architecture • Didn’t try very ...
Benchmarks• Old system (MySQL jobs table with polling,  5,000,000 dummy jobs populated)                            100    ...
Benchmarks• New system (Redis queue, historical data  only in MySQL, 5,000,000 dummy jobs  populated                      ...
Caveats• These benchmarks are stupid• Highly specific test; do your own tests• Most job time is spent doing the job, not  t...
Thank YouHave fun with Redis and Redis::Client    Mike Friedman   friedo@friedo.com
Questions
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Upcoming SlideShare
Loading in...5
×

Building Scalable, Distributed Job Queues with Redis and Redis::Client

15,890

Published on

Presentation at YAPC NA 2012 by Mike Friedman

Published in: Technology, Art & Photos
0 Comments
30 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
15,890
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
209
Comments
0
Likes
30
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Building Scalable, Distributed Job Queues with Redis and Redis::Client

    1. 1. Building Scalable,Distributed Job Queues with Redis and Redis::Client Mike Friedman (friedo) YAPC NA 2012
    2. 2. What Redis is• Open-source• Key-value store• Simple• Fast
    3. 3. What Redis isn’t• Relational (e.g. MySQL, Postgres, Oracle)• Document store (e.g. MongoDB)• Structured (e.g. Cassandra)• HTTP
    4. 4. Redis Data Types• Strings • e.g. ‘foo’, ’42’, or a JSON blob • Think Perl scalars
    5. 5. Redis Data Types• Lists • Zero or more strings, ordered • RPUSH, RPOP, LPUSH, and LPOP == push, pop, unshift, and shift in Perl
    6. 6. Redis Data Types• Hashes • Zero or more key-value pairs, unordered • HDEL, HEXISTS, HKEYS, and HVALS == delete, exists, keys, and values in Perl
    7. 7. Redis Data Types• Some types are not directly analogous to Perl concepts
    8. 8. Redis Data Types• Sets • Zero or more keys, unordered • No values • Think of a Perl hash where all values are undef
    9. 9. Redis Data Types• Sets • SREM, SISMEMBER, SMEMBERS == delete, exists, and keys in Perl
    10. 10. Redis Data Types• Set operations • Union • Intersection • Add / remove • Cardinality
    11. 11. Redis Data Types• Set operations can be implemented with Perl hashes• How?
    12. 12. Redis Data Types• Sorted Sets (zsets) • Zero or more keys, each with a numeric “score” • No values • Can be modeled as a Perl hash where the values are the scores
    13. 13. Redis Data Types• Sorted Sets (zsets) • ZREM, ZRANK, and ZRANGE (loosely) == delete, exists, and keys in Perl
    14. 14. Redis Data Types• Sorted set operations • Cardinality (within ranges) • Union / intersection • Lots more
    15. 15. Redis Data Types• Many more commands for working with Redis types• See http://redis.io/commands for the full list
    16. 16. Redis Protocol• Not HTTP• Stateful, interactive protocol
    17. 17. Redis Protocol• Actually, there are TWO protocols
    18. 18. Redis Protocol• Old Protocol• Client commands sent one per line• Server responses sent immediately
    19. 19. Redis Protocol• Old Protocol example: SET mystring foobar +OK
    20. 20. Redis Protocol• Old Protocol problems: • Command and data separated with whitespace • Difficult to parse - escaping becomes an issue • Difficult to deal with binary data or encoded text • Inconsistent
    21. 21. Redis Protocol• New Protocol (Unified Request Protocol) • Also called the URP
    22. 22. Redis Protocol• URP • Consistent command syntax • All data are prefixed with a byte length • No escaping or encoding/decoding required • Binary round-trip safe
    23. 23. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
    24. 24. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
    25. 25. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
    26. 26. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
    27. 27. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
    28. 28. Redis Dists on CPAN
    29. 29. Redis Dists on CPAN• Information as of when I began work on Redis::Client, so some of this may have changed by now.
    30. 30. Redis Dists on CPAN• Redis.pm • Simple interface; works • Does some odd things with encoding • No newer Redis types like hashes • AUTOLOAD :(
    31. 31. Redis Dists on CPAN• Redis.pm • Forces UTF-8 flag on returned data by default • This is horribly broken • It will be fixed
    32. 32. Redis Dists on CPAN• Redis::hiredis • Wrapper around hiredis binary client • Works well if you have hiredis available • External binary dependency • Slow IPC
    33. 33. Redis Dists on CPAN• There was no Perl Redis module with: • Native support for Redis hashes • No outside binary dependencies • Full URP Support
    34. 34. Redis Dists on CPAN Until now!
    35. 35. Redis Dists on CPAN (Applause)
    36. 36. Redis Dists on CPAN (Applause)
    37. 37. Redis::Client• On CPAN • https://metacpan.org/release/Redis-Client
    38. 38. Redis::Client• On Github • https://github.com/friedo/perl-redis-client
    39. 39. Redis::Client• Distribution structure • Redis/Client.pm • Primary interface, implements the Redis commands
    40. 40. Redis::Client• Distribution structure • Redis/Client/Role/URP.pm • Moose role; implements the Unified Request Protocol • Abstract enough to be used by other projects
    41. 41. Redis::Client• Distribution structure • Redis/Client/Role/Tied.pm • Moose role; common functionality for classes which map Redis types to Perl objects
    42. 42. Redis::Client• Distribution structure • Redis/Client/*.pm • Tied object classes for Redis strings, lists, hashes, sets, and sorted sets
    43. 43. Redis::Client• Distribution structure • Redis/Client/String.pm • Redis/Client/List.pm • Redis/Client/Hash.pm • Redis/Client/Set.pm • Redis/Client/Zset.pm
    44. 44. Redis::Client• Encoding Caveats • Redis::Client makes no assumptions about your data encoding • Character data MUST be encoded prior to being sent to Redis • Redis URP relies on accurate BYTE counts, NOT character counts • Data returned from Redis NOT decoded
    45. 45. Redis::Client• Future features • Callback registration system for • Encoding and decoding • Inflating and deflating
    46. 46. Redis::Client Examplesmy $redis = Redis::Client->new;# store a string$redis->set( mystring => ‘foo’ );# retrieve stringmy $str = $redis->get(‘mystring’);
    47. 47. Redis::Client Examples# work with lists$redis->lpush( ‘mylist’, ‘one’, ‘two’, ‘three’);my $tail = $redis->rpop(‘mylist’);# three
    48. 48. Redis::Client Examples# work with hashes$redis->hset( ‘myhash’, key => 42);my $val = $redis->hget( ‘myhash’, ‘key’);# 42
    49. 49. Redis::Client Examples# work with tied classestie my %hash, ‘Redis::Client::Hash’, key => ‘myhash’, client => $redis;my @keys = keys %hash;if ( exists $hash{foo} ) { ... }delete $hash{some_key};while ( my ( $k, $v ) = each %hash ) { ... }# etc.
    50. 50. Job Queues• Goals • Add jobs with arbitrary data to queue • Fetch and execute jobs as soon as possible • Prevent duplicate job execution • Thousands of jobs per hour
    51. 51. Job Queues• Old Model • Jobs table in relational DB • INSERT to add job • Poll DB to find new jobs • Set a status field when job is running • Transactions to prevent duplicates • Set status to ‘done’ or ‘error’ for historical data
    52. 52. Job Queues• Problems: • Relational DB is slow • Jobs table grows quickly; not scalable • Wrong tool for the job
    53. 53. Job Queues• Using Redis: • A single Redis list implements a queue • LPOP (shift) jobs off the front of the queue • RPUSH (push) jobs onto the end of the queue • Use JSON to store job arguments and metadata
    54. 54. Job Queues• Redis blocking push/pop • LPOP returns undef if no jobs • BLPOP command blocks until an item to shift exists • No need to poll server; just wait
    55. 55. Job Queues# Simple Example of Job / Dispatcheruse Redis::Client;use JSON::XS;use Module::Load;use TryCatch;my $redis = Redis::Client->new;
    56. 56. Job Queues# Add a jobmy $job = { class => ‘Some::Job’, method => ‘do_something’, constructor_args => { foo => 42 }, method_args => { bar => 43 } };$redis->lpush( jobs => encode_json $job );
    57. 57. Job Queues# Simple dispatcher loopmy $job_str = $redis->blpop(‘jobs’);my $job = decode_json $job_str;my %c_args = %{ $job->{constructor_args} };my $m_args = %{ $job->{method_args} };my $class = ‘MyApp::’ . $job->{class};my $meth = $job->{method};
    58. 58. Job Queuesload $class;my $obj = $class->new( %c_args );try { $obj->$meth( %m_args );} catch ( $err ) { # store error data --> relational DB};# store success data --> relational DB
    59. 59. Benchmarks• Disclaimer: • These are bad benchmarks • Dependent on highly system-specific architecture • Didn’t try very hard • Do your own evaluation
    60. 60. Benchmarks• Old system (MySQL jobs table with polling, 5,000,000 dummy jobs populated) 100 1 Worker 10 Workers Workers 88 jobs/ 710 jobs/ 3892 jobs/ minute minute minute avg. avg. avg.
    61. 61. Benchmarks• New system (Redis queue, historical data only in MySQL, 5,000,000 dummy jobs populated 100 1 Worker 10 Workers Workers 92 jobs/ 1011 jobs/ 9222 jobs/ minute minute minute avg. avg. avg.
    62. 62. Caveats• These benchmarks are stupid• Highly specific test; do your own tests• Most job time is spent doing the job, not talking to the dispatcher• But it works for my purposes
    63. 63. Thank YouHave fun with Redis and Redis::Client Mike Friedman friedo@friedo.com
    64. 64. Questions
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×