Building Scalable, Distributed Job Queues with Redis and Redis::Client
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Building Scalable, Distributed Job Queues with Redis and Redis::Client

  • 14,082 views
Uploaded on

Presentation at YAPC NA 2012 by Mike Friedman

Presentation at YAPC NA 2012 by Mike Friedman

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
14,082
On Slideshare
14,077
From Embeds
5
Number of Embeds
3

Actions

Shares
Downloads
122
Comments
0
Likes
22

Embeds 5

http://pinterest.com 3
https://twitter.com 1
http://www.docshut.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Transcript

  • 1. Building Scalable,Distributed Job Queues with Redis and Redis::Client Mike Friedman (friedo) YAPC NA 2012
  • 2. What Redis is• Open-source• Key-value store• Simple• Fast
  • 3. What Redis isn’t• Relational (e.g. MySQL, Postgres, Oracle)• Document store (e.g. MongoDB)• Structured (e.g. Cassandra)• HTTP
  • 4. Redis Data Types• Strings • e.g. ‘foo’, ’42’, or a JSON blob • Think Perl scalars
  • 5. Redis Data Types• Lists • Zero or more strings, ordered • RPUSH, RPOP, LPUSH, and LPOP == push, pop, unshift, and shift in Perl
  • 6. Redis Data Types• Hashes • Zero or more key-value pairs, unordered • HDEL, HEXISTS, HKEYS, and HVALS == delete, exists, keys, and values in Perl
  • 7. Redis Data Types• Some types are not directly analogous to Perl concepts
  • 8. Redis Data Types• Sets • Zero or more keys, unordered • No values • Think of a Perl hash where all values are undef
  • 9. Redis Data Types• Sets • SREM, SISMEMBER, SMEMBERS == delete, exists, and keys in Perl
  • 10. Redis Data Types• Set operations • Union • Intersection • Add / remove • Cardinality
  • 11. Redis Data Types• Set operations can be implemented with Perl hashes• How?
  • 12. Redis Data Types• Sorted Sets (zsets) • Zero or more keys, each with a numeric “score” • No values • Can be modeled as a Perl hash where the values are the scores
  • 13. Redis Data Types• Sorted Sets (zsets) • ZREM, ZRANK, and ZRANGE (loosely) == delete, exists, and keys in Perl
  • 14. Redis Data Types• Sorted set operations • Cardinality (within ranges) • Union / intersection • Lots more
  • 15. Redis Data Types• Many more commands for working with Redis types• See http://redis.io/commands for the full list
  • 16. Redis Protocol• Not HTTP• Stateful, interactive protocol
  • 17. Redis Protocol• Actually, there are TWO protocols
  • 18. Redis Protocol• Old Protocol• Client commands sent one per line• Server responses sent immediately
  • 19. Redis Protocol• Old Protocol example: SET mystring foobar +OK
  • 20. Redis Protocol• Old Protocol problems: • Command and data separated with whitespace • Difficult to parse - escaping becomes an issue • Difficult to deal with binary data or encoded text • Inconsistent
  • 21. Redis Protocol• New Protocol (Unified Request Protocol) • Also called the URP
  • 22. Redis Protocol• URP • Consistent command syntax • All data are prefixed with a byte length • No escaping or encoding/decoding required • Binary round-trip safe
  • 23. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
  • 24. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
  • 25. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
  • 26. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
  • 27. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
  • 28. Redis Dists on CPAN
  • 29. Redis Dists on CPAN• Information as of when I began work on Redis::Client, so some of this may have changed by now.
  • 30. Redis Dists on CPAN• Redis.pm • Simple interface; works • Does some odd things with encoding • No newer Redis types like hashes • AUTOLOAD :(
  • 31. Redis Dists on CPAN• Redis.pm • Forces UTF-8 flag on returned data by default • This is horribly broken • It will be fixed
  • 32. Redis Dists on CPAN• Redis::hiredis • Wrapper around hiredis binary client • Works well if you have hiredis available • External binary dependency • Slow IPC
  • 33. Redis Dists on CPAN• There was no Perl Redis module with: • Native support for Redis hashes • No outside binary dependencies • Full URP Support
  • 34. Redis Dists on CPAN Until now!
  • 35. Redis Dists on CPAN (Applause)
  • 36. Redis Dists on CPAN (Applause)
  • 37. Redis::Client• On CPAN • https://metacpan.org/release/Redis-Client
  • 38. Redis::Client• On Github • https://github.com/friedo/perl-redis-client
  • 39. Redis::Client• Distribution structure • Redis/Client.pm • Primary interface, implements the Redis commands
  • 40. Redis::Client• Distribution structure • Redis/Client/Role/URP.pm • Moose role; implements the Unified Request Protocol • Abstract enough to be used by other projects
  • 41. Redis::Client• Distribution structure • Redis/Client/Role/Tied.pm • Moose role; common functionality for classes which map Redis types to Perl objects
  • 42. Redis::Client• Distribution structure • Redis/Client/*.pm • Tied object classes for Redis strings, lists, hashes, sets, and sorted sets
  • 43. Redis::Client• Distribution structure • Redis/Client/String.pm • Redis/Client/List.pm • Redis/Client/Hash.pm • Redis/Client/Set.pm • Redis/Client/Zset.pm
  • 44. Redis::Client• Encoding Caveats • Redis::Client makes no assumptions about your data encoding • Character data MUST be encoded prior to being sent to Redis • Redis URP relies on accurate BYTE counts, NOT character counts • Data returned from Redis NOT decoded
  • 45. Redis::Client• Future features • Callback registration system for • Encoding and decoding • Inflating and deflating
  • 46. Redis::Client Examplesmy $redis = Redis::Client->new;# store a string$redis->set( mystring => ‘foo’ );# retrieve stringmy $str = $redis->get(‘mystring’);
  • 47. Redis::Client Examples# work with lists$redis->lpush( ‘mylist’, ‘one’, ‘two’, ‘three’);my $tail = $redis->rpop(‘mylist’);# three
  • 48. Redis::Client Examples# work with hashes$redis->hset( ‘myhash’, key => 42);my $val = $redis->hget( ‘myhash’, ‘key’);# 42
  • 49. Redis::Client Examples# work with tied classestie my %hash, ‘Redis::Client::Hash’, key => ‘myhash’, client => $redis;my @keys = keys %hash;if ( exists $hash{foo} ) { ... }delete $hash{some_key};while ( my ( $k, $v ) = each %hash ) { ... }# etc.
  • 50. Job Queues• Goals • Add jobs with arbitrary data to queue • Fetch and execute jobs as soon as possible • Prevent duplicate job execution • Thousands of jobs per hour
  • 51. Job Queues• Old Model • Jobs table in relational DB • INSERT to add job • Poll DB to find new jobs • Set a status field when job is running • Transactions to prevent duplicates • Set status to ‘done’ or ‘error’ for historical data
  • 52. Job Queues• Problems: • Relational DB is slow • Jobs table grows quickly; not scalable • Wrong tool for the job
  • 53. Job Queues• Using Redis: • A single Redis list implements a queue • LPOP (shift) jobs off the front of the queue • RPUSH (push) jobs onto the end of the queue • Use JSON to store job arguments and metadata
  • 54. Job Queues• Redis blocking push/pop • LPOP returns undef if no jobs • BLPOP command blocks until an item to shift exists • No need to poll server; just wait
  • 55. Job Queues# Simple Example of Job / Dispatcheruse Redis::Client;use JSON::XS;use Module::Load;use TryCatch;my $redis = Redis::Client->new;
  • 56. Job Queues# Add a jobmy $job = { class => ‘Some::Job’, method => ‘do_something’, constructor_args => { foo => 42 }, method_args => { bar => 43 } };$redis->lpush( jobs => encode_json $job );
  • 57. Job Queues# Simple dispatcher loopmy $job_str = $redis->blpop(‘jobs’);my $job = decode_json $job_str;my %c_args = %{ $job->{constructor_args} };my $m_args = %{ $job->{method_args} };my $class = ‘MyApp::’ . $job->{class};my $meth = $job->{method};
  • 58. Job Queuesload $class;my $obj = $class->new( %c_args );try { $obj->$meth( %m_args );} catch ( $err ) { # store error data --> relational DB};# store success data --> relational DB
  • 59. Benchmarks• Disclaimer: • These are bad benchmarks • Dependent on highly system-specific architecture • Didn’t try very hard • Do your own evaluation
  • 60. Benchmarks• Old system (MySQL jobs table with polling, 5,000,000 dummy jobs populated) 100 1 Worker 10 Workers Workers 88 jobs/ 710 jobs/ 3892 jobs/ minute minute minute avg. avg. avg.
  • 61. Benchmarks• New system (Redis queue, historical data only in MySQL, 5,000,000 dummy jobs populated 100 1 Worker 10 Workers Workers 92 jobs/ 1011 jobs/ 9222 jobs/ minute minute minute avg. avg. avg.
  • 62. Caveats• These benchmarks are stupid• Highly specific test; do your own tests• Most job time is spent doing the job, not talking to the dispatcher• But it works for my purposes
  • 63. Thank YouHave fun with Redis and Redis::Client Mike Friedman friedo@friedo.com
  • 64. Questions