Building Scalable, Distributed Job Queues with Redis and Redis::Client

  • 13,586 views
Uploaded on

Presentation at YAPC NA 2012 by Mike Friedman

Presentation at YAPC NA 2012 by Mike Friedman

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
13,586
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
125
Comments
0
Likes
23

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Transcript

  • 1. Building Scalable,Distributed Job Queues with Redis and Redis::Client Mike Friedman (friedo) YAPC NA 2012
  • 2. What Redis is• Open-source• Key-value store• Simple• Fast
  • 3. What Redis isn’t• Relational (e.g. MySQL, Postgres, Oracle)• Document store (e.g. MongoDB)• Structured (e.g. Cassandra)• HTTP
  • 4. Redis Data Types• Strings • e.g. ‘foo’, ’42’, or a JSON blob • Think Perl scalars
  • 5. Redis Data Types• Lists • Zero or more strings, ordered • RPUSH, RPOP, LPUSH, and LPOP == push, pop, unshift, and shift in Perl
  • 6. Redis Data Types• Hashes • Zero or more key-value pairs, unordered • HDEL, HEXISTS, HKEYS, and HVALS == delete, exists, keys, and values in Perl
  • 7. Redis Data Types• Some types are not directly analogous to Perl concepts
  • 8. Redis Data Types• Sets • Zero or more keys, unordered • No values • Think of a Perl hash where all values are undef
  • 9. Redis Data Types• Sets • SREM, SISMEMBER, SMEMBERS == delete, exists, and keys in Perl
  • 10. Redis Data Types• Set operations • Union • Intersection • Add / remove • Cardinality
  • 11. Redis Data Types• Set operations can be implemented with Perl hashes• How?
  • 12. Redis Data Types• Sorted Sets (zsets) • Zero or more keys, each with a numeric “score” • No values • Can be modeled as a Perl hash where the values are the scores
  • 13. Redis Data Types• Sorted Sets (zsets) • ZREM, ZRANK, and ZRANGE (loosely) == delete, exists, and keys in Perl
  • 14. Redis Data Types• Sorted set operations • Cardinality (within ranges) • Union / intersection • Lots more
  • 15. Redis Data Types• Many more commands for working with Redis types• See http://redis.io/commands for the full list
  • 16. Redis Protocol• Not HTTP• Stateful, interactive protocol
  • 17. Redis Protocol• Actually, there are TWO protocols
  • 18. Redis Protocol• Old Protocol• Client commands sent one per line• Server responses sent immediately
  • 19. Redis Protocol• Old Protocol example: SET mystring foobar +OK
  • 20. Redis Protocol• Old Protocol problems: • Command and data separated with whitespace • Difficult to parse - escaping becomes an issue • Difficult to deal with binary data or encoded text • Inconsistent
  • 21. Redis Protocol• New Protocol (Unified Request Protocol) • Also called the URP
  • 22. Redis Protocol• URP • Consistent command syntax • All data are prefixed with a byte length • No escaping or encoding/decoding required • Binary round-trip safe
  • 23. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
  • 24. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
  • 25. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
  • 26. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
  • 27. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
  • 28. Redis Dists on CPAN
  • 29. Redis Dists on CPAN• Information as of when I began work on Redis::Client, so some of this may have changed by now.
  • 30. Redis Dists on CPAN• Redis.pm • Simple interface; works • Does some odd things with encoding • No newer Redis types like hashes • AUTOLOAD :(
  • 31. Redis Dists on CPAN• Redis.pm • Forces UTF-8 flag on returned data by default • This is horribly broken • It will be fixed
  • 32. Redis Dists on CPAN• Redis::hiredis • Wrapper around hiredis binary client • Works well if you have hiredis available • External binary dependency • Slow IPC
  • 33. Redis Dists on CPAN• There was no Perl Redis module with: • Native support for Redis hashes • No outside binary dependencies • Full URP Support
  • 34. Redis Dists on CPAN Until now!
  • 35. Redis Dists on CPAN (Applause)
  • 36. Redis Dists on CPAN (Applause)
  • 37. Redis::Client• On CPAN • https://metacpan.org/release/Redis-Client
  • 38. Redis::Client• On Github • https://github.com/friedo/perl-redis-client
  • 39. Redis::Client• Distribution structure • Redis/Client.pm • Primary interface, implements the Redis commands
  • 40. Redis::Client• Distribution structure • Redis/Client/Role/URP.pm • Moose role; implements the Unified Request Protocol • Abstract enough to be used by other projects
  • 41. Redis::Client• Distribution structure • Redis/Client/Role/Tied.pm • Moose role; common functionality for classes which map Redis types to Perl objects
  • 42. Redis::Client• Distribution structure • Redis/Client/*.pm • Tied object classes for Redis strings, lists, hashes, sets, and sorted sets
  • 43. Redis::Client• Distribution structure • Redis/Client/String.pm • Redis/Client/List.pm • Redis/Client/Hash.pm • Redis/Client/Set.pm • Redis/Client/Zset.pm
  • 44. Redis::Client• Encoding Caveats • Redis::Client makes no assumptions about your data encoding • Character data MUST be encoded prior to being sent to Redis • Redis URP relies on accurate BYTE counts, NOT character counts • Data returned from Redis NOT decoded
  • 45. Redis::Client• Future features • Callback registration system for • Encoding and decoding • Inflating and deflating
  • 46. Redis::Client Examplesmy $redis = Redis::Client->new;# store a string$redis->set( mystring => ‘foo’ );# retrieve stringmy $str = $redis->get(‘mystring’);
  • 47. Redis::Client Examples# work with lists$redis->lpush( ‘mylist’, ‘one’, ‘two’, ‘three’);my $tail = $redis->rpop(‘mylist’);# three
  • 48. Redis::Client Examples# work with hashes$redis->hset( ‘myhash’, key => 42);my $val = $redis->hget( ‘myhash’, ‘key’);# 42
  • 49. Redis::Client Examples# work with tied classestie my %hash, ‘Redis::Client::Hash’, key => ‘myhash’, client => $redis;my @keys = keys %hash;if ( exists $hash{foo} ) { ... }delete $hash{some_key};while ( my ( $k, $v ) = each %hash ) { ... }# etc.
  • 50. Job Queues• Goals • Add jobs with arbitrary data to queue • Fetch and execute jobs as soon as possible • Prevent duplicate job execution • Thousands of jobs per hour
  • 51. Job Queues• Old Model • Jobs table in relational DB • INSERT to add job • Poll DB to find new jobs • Set a status field when job is running • Transactions to prevent duplicates • Set status to ‘done’ or ‘error’ for historical data
  • 52. Job Queues• Problems: • Relational DB is slow • Jobs table grows quickly; not scalable • Wrong tool for the job
  • 53. Job Queues• Using Redis: • A single Redis list implements a queue • LPOP (shift) jobs off the front of the queue • RPUSH (push) jobs onto the end of the queue • Use JSON to store job arguments and metadata
  • 54. Job Queues• Redis blocking push/pop • LPOP returns undef if no jobs • BLPOP command blocks until an item to shift exists • No need to poll server; just wait
  • 55. Job Queues# Simple Example of Job / Dispatcheruse Redis::Client;use JSON::XS;use Module::Load;use TryCatch;my $redis = Redis::Client->new;
  • 56. Job Queues# Add a jobmy $job = { class => ‘Some::Job’, method => ‘do_something’, constructor_args => { foo => 42 }, method_args => { bar => 43 } };$redis->lpush( jobs => encode_json $job );
  • 57. Job Queues# Simple dispatcher loopmy $job_str = $redis->blpop(‘jobs’);my $job = decode_json $job_str;my %c_args = %{ $job->{constructor_args} };my $m_args = %{ $job->{method_args} };my $class = ‘MyApp::’ . $job->{class};my $meth = $job->{method};
  • 58. Job Queuesload $class;my $obj = $class->new( %c_args );try { $obj->$meth( %m_args );} catch ( $err ) { # store error data --> relational DB};# store success data --> relational DB
  • 59. Benchmarks• Disclaimer: • These are bad benchmarks • Dependent on highly system-specific architecture • Didn’t try very hard • Do your own evaluation
  • 60. Benchmarks• Old system (MySQL jobs table with polling, 5,000,000 dummy jobs populated) 100 1 Worker 10 Workers Workers 88 jobs/ 710 jobs/ 3892 jobs/ minute minute minute avg. avg. avg.
  • 61. Benchmarks• New system (Redis queue, historical data only in MySQL, 5,000,000 dummy jobs populated 100 1 Worker 10 Workers Workers 92 jobs/ 1011 jobs/ 9222 jobs/ minute minute minute avg. avg. avg.
  • 62. Caveats• These benchmarks are stupid• Highly specific test; do your own tests• Most job time is spent doing the job, not talking to the dispatcher• But it works for my purposes
  • 63. Thank YouHave fun with Redis and Redis::Client Mike Friedman friedo@friedo.com
  • 64. Questions