Your SlideShare is downloading. ×
0
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Building Scalable, Distributed Job Queues with Redis and Redis::Client

15,360

Published on

Presentation at YAPC NA 2012 by Mike Friedman

Presentation at YAPC NA 2012 by Mike Friedman

Published in: Technology, Art & Photos
0 Comments
30 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
15,360
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
200
Comments
0
Likes
30
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Transcript

    • 1. Building Scalable,Distributed Job Queues with Redis and Redis::Client Mike Friedman (friedo) YAPC NA 2012
    • 2. What Redis is• Open-source• Key-value store• Simple• Fast
    • 3. What Redis isn’t• Relational (e.g. MySQL, Postgres, Oracle)• Document store (e.g. MongoDB)• Structured (e.g. Cassandra)• HTTP
    • 4. Redis Data Types• Strings • e.g. ‘foo’, ’42’, or a JSON blob • Think Perl scalars
    • 5. Redis Data Types• Lists • Zero or more strings, ordered • RPUSH, RPOP, LPUSH, and LPOP == push, pop, unshift, and shift in Perl
    • 6. Redis Data Types• Hashes • Zero or more key-value pairs, unordered • HDEL, HEXISTS, HKEYS, and HVALS == delete, exists, keys, and values in Perl
    • 7. Redis Data Types• Some types are not directly analogous to Perl concepts
    • 8. Redis Data Types• Sets • Zero or more keys, unordered • No values • Think of a Perl hash where all values are undef
    • 9. Redis Data Types• Sets • SREM, SISMEMBER, SMEMBERS == delete, exists, and keys in Perl
    • 10. Redis Data Types• Set operations • Union • Intersection • Add / remove • Cardinality
    • 11. Redis Data Types• Set operations can be implemented with Perl hashes• How?
    • 12. Redis Data Types• Sorted Sets (zsets) • Zero or more keys, each with a numeric “score” • No values • Can be modeled as a Perl hash where the values are the scores
    • 13. Redis Data Types• Sorted Sets (zsets) • ZREM, ZRANK, and ZRANGE (loosely) == delete, exists, and keys in Perl
    • 14. Redis Data Types• Sorted set operations • Cardinality (within ranges) • Union / intersection • Lots more
    • 15. Redis Data Types• Many more commands for working with Redis types• See http://redis.io/commands for the full list
    • 16. Redis Protocol• Not HTTP• Stateful, interactive protocol
    • 17. Redis Protocol• Actually, there are TWO protocols
    • 18. Redis Protocol• Old Protocol• Client commands sent one per line• Server responses sent immediately
    • 19. Redis Protocol• Old Protocol example: SET mystring foobar +OK
    • 20. Redis Protocol• Old Protocol problems: • Command and data separated with whitespace • Difficult to parse - escaping becomes an issue • Difficult to deal with binary data or encoded text • Inconsistent
    • 21. Redis Protocol• New Protocol (Unified Request Protocol) • Also called the URP
    • 22. Redis Protocol• URP • Consistent command syntax • All data are prefixed with a byte length • No escaping or encoding/decoding required • Binary round-trip safe
    • 23. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
    • 24. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
    • 25. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
    • 26. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
    • 27. Redis Protocol• URP Example *3 $3 SET $8 mystring $6 foobar +OK
    • 28. Redis Dists on CPAN
    • 29. Redis Dists on CPAN• Information as of when I began work on Redis::Client, so some of this may have changed by now.
    • 30. Redis Dists on CPAN• Redis.pm • Simple interface; works • Does some odd things with encoding • No newer Redis types like hashes • AUTOLOAD :(
    • 31. Redis Dists on CPAN• Redis.pm • Forces UTF-8 flag on returned data by default • This is horribly broken • It will be fixed
    • 32. Redis Dists on CPAN• Redis::hiredis • Wrapper around hiredis binary client • Works well if you have hiredis available • External binary dependency • Slow IPC
    • 33. Redis Dists on CPAN• There was no Perl Redis module with: • Native support for Redis hashes • No outside binary dependencies • Full URP Support
    • 34. Redis Dists on CPAN Until now!
    • 35. Redis Dists on CPAN (Applause)
    • 36. Redis Dists on CPAN (Applause)
    • 37. Redis::Client• On CPAN • https://metacpan.org/release/Redis-Client
    • 38. Redis::Client• On Github • https://github.com/friedo/perl-redis-client
    • 39. Redis::Client• Distribution structure • Redis/Client.pm • Primary interface, implements the Redis commands
    • 40. Redis::Client• Distribution structure • Redis/Client/Role/URP.pm • Moose role; implements the Unified Request Protocol • Abstract enough to be used by other projects
    • 41. Redis::Client• Distribution structure • Redis/Client/Role/Tied.pm • Moose role; common functionality for classes which map Redis types to Perl objects
    • 42. Redis::Client• Distribution structure • Redis/Client/*.pm • Tied object classes for Redis strings, lists, hashes, sets, and sorted sets
    • 43. Redis::Client• Distribution structure • Redis/Client/String.pm • Redis/Client/List.pm • Redis/Client/Hash.pm • Redis/Client/Set.pm • Redis/Client/Zset.pm
    • 44. Redis::Client• Encoding Caveats • Redis::Client makes no assumptions about your data encoding • Character data MUST be encoded prior to being sent to Redis • Redis URP relies on accurate BYTE counts, NOT character counts • Data returned from Redis NOT decoded
    • 45. Redis::Client• Future features • Callback registration system for • Encoding and decoding • Inflating and deflating
    • 46. Redis::Client Examplesmy $redis = Redis::Client->new;# store a string$redis->set( mystring => ‘foo’ );# retrieve stringmy $str = $redis->get(‘mystring’);
    • 47. Redis::Client Examples# work with lists$redis->lpush( ‘mylist’, ‘one’, ‘two’, ‘three’);my $tail = $redis->rpop(‘mylist’);# three
    • 48. Redis::Client Examples# work with hashes$redis->hset( ‘myhash’, key => 42);my $val = $redis->hget( ‘myhash’, ‘key’);# 42
    • 49. Redis::Client Examples# work with tied classestie my %hash, ‘Redis::Client::Hash’, key => ‘myhash’, client => $redis;my @keys = keys %hash;if ( exists $hash{foo} ) { ... }delete $hash{some_key};while ( my ( $k, $v ) = each %hash ) { ... }# etc.
    • 50. Job Queues• Goals • Add jobs with arbitrary data to queue • Fetch and execute jobs as soon as possible • Prevent duplicate job execution • Thousands of jobs per hour
    • 51. Job Queues• Old Model • Jobs table in relational DB • INSERT to add job • Poll DB to find new jobs • Set a status field when job is running • Transactions to prevent duplicates • Set status to ‘done’ or ‘error’ for historical data
    • 52. Job Queues• Problems: • Relational DB is slow • Jobs table grows quickly; not scalable • Wrong tool for the job
    • 53. Job Queues• Using Redis: • A single Redis list implements a queue • LPOP (shift) jobs off the front of the queue • RPUSH (push) jobs onto the end of the queue • Use JSON to store job arguments and metadata
    • 54. Job Queues• Redis blocking push/pop • LPOP returns undef if no jobs • BLPOP command blocks until an item to shift exists • No need to poll server; just wait
    • 55. Job Queues# Simple Example of Job / Dispatcheruse Redis::Client;use JSON::XS;use Module::Load;use TryCatch;my $redis = Redis::Client->new;
    • 56. Job Queues# Add a jobmy $job = { class => ‘Some::Job’, method => ‘do_something’, constructor_args => { foo => 42 }, method_args => { bar => 43 } };$redis->lpush( jobs => encode_json $job );
    • 57. Job Queues# Simple dispatcher loopmy $job_str = $redis->blpop(‘jobs’);my $job = decode_json $job_str;my %c_args = %{ $job->{constructor_args} };my $m_args = %{ $job->{method_args} };my $class = ‘MyApp::’ . $job->{class};my $meth = $job->{method};
    • 58. Job Queuesload $class;my $obj = $class->new( %c_args );try { $obj->$meth( %m_args );} catch ( $err ) { # store error data --> relational DB};# store success data --> relational DB
    • 59. Benchmarks• Disclaimer: • These are bad benchmarks • Dependent on highly system-specific architecture • Didn’t try very hard • Do your own evaluation
    • 60. Benchmarks• Old system (MySQL jobs table with polling, 5,000,000 dummy jobs populated) 100 1 Worker 10 Workers Workers 88 jobs/ 710 jobs/ 3892 jobs/ minute minute minute avg. avg. avg.
    • 61. Benchmarks• New system (Redis queue, historical data only in MySQL, 5,000,000 dummy jobs populated 100 1 Worker 10 Workers Workers 92 jobs/ 1011 jobs/ 9222 jobs/ minute minute minute avg. avg. avg.
    • 62. Caveats• These benchmarks are stupid• Highly specific test; do your own tests• Most job time is spent doing the job, not talking to the dispatcher• But it works for my purposes
    • 63. Thank YouHave fun with Redis and Redis::Client Mike Friedman friedo@friedo.com
    • 64. Questions

    ×