Presentation at YAPC NA 2012 by Mike Friedman

1. Building Scalable,
Distributed Job Queues
with Redis and
Redis::Client
Mike Friedman
(friedo)
YAPC NA 2012

2. What Redis is
• Open-source
• Key-value store
• Simple
• Fast

3. What Redis isn’t
• Relational (e.g. MySQL, Postgres, Oracle)
• Document store (e.g. MongoDB)
• Structured (e.g. Cassandra)
• HTTP

4. Redis Data Types
• Strings
• e.g. ‘foo’, ’42’, or a JSON blob
• Think Perl scalars

5. Redis Data Types
• Lists
• Zero or more strings, ordered
• RPUSH, RPOP, LPUSH, and LPOP ==
push, pop, unshift, and shift in
Perl

6. Redis Data Types
• Hashes
• Zero or more key-value pairs, unordered
• HDEL, HEXISTS, HKEYS, and HVALS
== delete, exists, keys, and
values in Perl

7. Redis Data Types
• Some types are not directly analogous to
Perl concepts

8. Redis Data Types
• Sets
• Zero or more keys, unordered
• No values
• Think of a Perl hash where all values are
undef

9. Redis Data Types
• Sets
• SREM, SISMEMBER, SMEMBERS ==
delete, exists, and keys in Perl

10. Redis Data Types
• Set operations
• Union
• Intersection
• Add / remove
• Cardinality

11. Redis Data Types
• Set operations can be implemented with
Perl hashes
• How?

12. Redis Data Types
• Sorted Sets (zsets)
• Zero or more keys, each with a numeric
“score”
• No values
• Can be modeled as a Perl hash where the
values are the scores

13. Redis Data Types
• Sorted Sets (zsets)
• ZREM, ZRANK, and ZRANGE (loosely)
== delete, exists, and keys in Perl

14. Redis Data Types
• Sorted set operations
• Cardinality (within ranges)
• Union / intersection
• Lots more

15. Redis Data Types
• Many more commands for working with
Redis types
• See http://redis.io/commands for the full list

16. Redis Protocol
• Not HTTP
• Stateful, interactive protocol

17. Redis Protocol
• Actually, there are TWO protocols

18. Redis Protocol
• Old Protocol
• Client commands sent one per line
• Server responses sent immediately

19. Redis Protocol
• Old Protocol example:
SET mystring foobar
+OK

20. Redis Protocol
• Old Protocol problems:
• Command and data separated with
whitespace
• Difficult to parse - escaping becomes an
issue
• Difficult to deal with binary data or
encoded text
• Inconsistent

21. Redis Protocol
• New Protocol (Unified Request Protocol)
• Also called the URP

22. Redis Protocol
• URP
• Consistent command syntax
• All data are prefixed with a byte length
• No escaping or encoding/decoding
required
• Binary round-trip safe

23. Redis Protocol
• URP Example
*3
$3
SET
$8
mystring
$6
foobar
+OK

24. Redis Protocol
• URP Example
*3
$3
SET
$8
mystring
$6
foobar
+OK

25. Redis Protocol
• URP Example
*3
$3
SET
$8
mystring
$6
foobar
+OK

26. Redis Protocol
• URP Example
*3
$3
SET
$8
mystring
$6
foobar
+OK

27. Redis Protocol
• URP Example
*3
$3
SET
$8
mystring
$6
foobar
+OK

28. Redis Dists on CPAN

29. Redis Dists on CPAN
• Information as of when I began work on
Redis::Client, so some of this may have
changed by now.

30. Redis Dists on CPAN
• Redis.pm
• Simple interface; works
• Does some odd things with encoding
• No newer Redis types like hashes
• AUTOLOAD :(

31. Redis Dists on CPAN
• Redis.pm
• Forces UTF-8 flag on returned data by
default
• This is horribly broken
• It will be fixed

32. Redis Dists on CPAN
• Redis::hiredis
• Wrapper around hiredis binary client
• Works well if you have hiredis
available
• External binary dependency
• Slow IPC

33. Redis Dists on CPAN
• There was no Perl Redis module with:
• Native support for Redis hashes
• No outside binary dependencies
• Full URP Support

34. Redis Dists on CPAN
Until now!

35. Redis Dists on CPAN
(Applause)

36. Redis Dists on CPAN
(Applause)

37. Redis::Client
• On CPAN
• https://metacpan.org/release/Redis-Client

39. Redis::Client
• On Github
• https://github.com/friedo/perl-redis-client

41. Redis::Client
• Distribution structure
• Redis/Client.pm
• Primary interface, implements the
Redis commands

42. Redis::Client
• Distribution structure
• Redis/Client/Role/URP.pm
• Moose role; implements the Unified
Request Protocol
• Abstract enough to be used by other
projects

43. Redis::Client
• Distribution structure
• Redis/Client/Role/Tied.pm
• Moose role; common functionality for
classes which map Redis types to Perl
objects

44. Redis::Client
• Distribution structure
• Redis/Client/*.pm
• Tied object classes for Redis strings,
lists, hashes, sets, and sorted sets

45. Redis::Client
• Distribution structure
• Redis/Client/String.pm
• Redis/Client/List.pm
• Redis/Client/Hash.pm
• Redis/Client/Set.pm
• Redis/Client/Zset.pm

46. Redis::Client
• Encoding Caveats
• Redis::Client makes no assumptions
about your data encoding
• Character data MUST be encoded prior
to being sent to Redis
• Redis URP relies on accurate BYTE
counts, NOT character counts
• Data returned from Redis NOT decoded

47. Redis::Client
• Future features
• Callback registration system for
• Encoding and decoding
• Inflating and deflating

48. Redis::Client Examples
my $redis = Redis::Client->new;
# store a string
$redis->set( mystring => ‘foo’ );
# retrieve string
my $str = $redis->get(‘mystring’);

49. Redis::Client Examples
# work with lists
$redis->lpush(
‘mylist’, ‘one’, ‘two’, ‘three’
);
my $tail = $redis->rpop(‘mylist’);
# three

50. Redis::Client Examples
# work with hashes
$redis->hset(
‘myhash’, key => 42
);
my $val = $redis->hget(
‘myhash’, ‘key’
);
# 42

51. Redis::Client Examples
# work with tied classes
tie my %hash, ‘Redis::Client::Hash’,
key => ‘myhash’, client => $redis;
my @keys = keys %hash;
if ( exists $hash{foo} ) { ... }
delete $hash{some_key};
while ( my ( $k, $v ) = each %hash )
{ ... }
# etc.

52. Job Queues
• Goals
• Add jobs with arbitrary data to queue
• Fetch and execute jobs as soon as
possible
• Prevent duplicate job execution
• Thousands of jobs per hour

53. Job Queues
• Old Model
• Jobs table in relational DB
• INSERT to add job
• Poll DB to find new jobs
• Set a status field when job is running
• Transactions to prevent duplicates
• Set status to ‘done’ or ‘error’ for
historical data

54. Job Queues
• Problems:
• Relational DB is slow
• Jobs table grows quickly; not scalable
• Wrong tool for the job

55. Job Queues
• Using Redis:
• A single Redis list implements a queue
• LPOP (shift) jobs off the front of the
queue
• RPUSH (push) jobs onto the end of the
queue
• Use JSON to store job arguments and
metadata

56. Job Queues
• Redis blocking push/pop
• LPOP returns undef if no jobs
• BLPOP command blocks until an item to
shift exists
• No need to poll server; just wait

57. Job Queues
# Simple Example of Job / Dispatcher
use Redis::Client;
use JSON::XS;
use Module::Load;
use TryCatch;
my $redis = Redis::Client->new;

58. Job Queues
# Add a job
my $job =
{ class => ‘Some::Job’,
method => ‘do_something’,
constructor_args => { foo => 42 },
method_args => { bar => 43 } };
$redis->lpush( jobs => encode_json $job );

59. Job Queues
# Simple dispatcher loop
my $job_str = $redis->blpop(‘jobs’);
my $job = decode_json $job_str;
my %c_args
= %{ $job->{constructor_args} };
my $m_args
= %{ $job->{method_args} };
my $class
= ‘MyApp::’ . $job->{class};
my $meth = $job->{method};

60. Job Queues
load $class;
my $obj = $class->new( %c_args );
try {
$obj->$meth( %m_args );
} catch ( $err ) {
# store error data --> relational DB
};
# store success data --> relational DB

61. Benchmarks
• Disclaimer:
• These are bad benchmarks
• Dependent on highly system-specific
architecture
• Didn’t try very hard
• Do your own evaluation

62. Benchmarks
• Old system (MySQL jobs table with polling,
5,000,000 dummy jobs populated)
100
1 Worker 10 Workers
Workers
88 jobs/ 710 jobs/ 3892 jobs/
minute minute minute
avg. avg. avg.

63. Benchmarks
• New system (Redis queue, historical data
only in MySQL, 5,000,000 dummy jobs
populated
100
1 Worker 10 Workers
Workers
92 jobs/ 1011 jobs/ 9222 jobs/
minute minute minute
avg. avg. avg.

64. Caveats
• These benchmarks are stupid
• Highly specific test; do your own tests
• Most job time is spent doing the job, not
talking to the dispatcher
• But it works for my purposes

65. Thank You
Have fun with Redis
and Redis::Client
Mike Friedman
friedo@friedo.com

66. Questions