Presented at #PHPLX 11 July 2013 When you need to do some heavy processing how do you scale you application? You can use Redis and ZeroMQ to leverage the heavy work for you! With this presentation we will know more about this two technologies and how they can be used to help solve problems with the performance and scalability of your application.

1. Follow this topic:
@rjsmelo
Redis & ZeroMQ: How to
scale your application
RICARDO MELO
Presented at #PHPLX – 11 July 2013

2. @rjsmelo 2
RICARDO MELO
● CTO @ DRI
● PHP, Mysql, Linux and lots of other
OSS
● ZCE, RHCE, LPI 3, ITIL, etc
● +10 years building (and breaking)
things

3. @rjsmelo 3
About
● 14 Year old academic spin-off
● Pragmatic OSS Orientation
● PHP, Mysql, SugarCRM, Drupal,
JavaScript, Linux, etc.
● Crafters, Integrators
● Always looking for software developers
– Yes, right now!

4. 1999 - 2013 DRI. Some Rights Reserved. 4
Outline
● Redis
● ZeroMQ
● Use Cases
● Conclusions

5. 1999 - 2013 DRI. Some Rights Reserved. 5
Redis
“Redis is an open source, BSD licensed,
advanced key-value store. It is often
referred to as a data structure server
since keys can contain strings, hashes,
lists, sets and sorted sets.”
source: http://redis.io

6. 1999 - 2013 DRI. Some Rights Reserved. 6
Predis
● Pure PHP Redis Client
– https://github.com/nrk/predis
● Use composer:
– composer install
{
"name": "rjsmelo/talk-redis-zmq",
"description": "Sample code for Redis & ZeroMQ presentation",
"require": {
"ext-zmq": "*",
"predis/predis": "dev-master"
},
"license": "Apache-2.0",
"authors": [
{
"name": "Ricardo Melo",
"email": "ricardo.melo@dri.pt"
}
]
}

7. 1999 - 2013 DRI. Some Rights Reserved. 7
Strings
1 <?php
2 //redis_strings.php
3 require_once __DIR__ . '/../vendor/autoload.php';
4 $redis = new PredisClient('tcp://localhost:6379');
5
6 // Using SET
7 $redis->set("SampleKey", "Throw me any thing here....");
8 echo $redis->get("SampleKey") . "n"; // Throw me any thing here....
9
10 // Remove Key
11 $redis->del("SampleKey");
12
13 // Using APPEND
14 $redis->append("SampleKey", "Hello"); // Hello
15 $redis->append("SampleKey", " World!"); // Hello World!
16 echo $redis->get("SampleKey") . "n";
17
18 // Other commands: incr, decr, incrby, getrange, setrange
19 // http://redis.io/commands/#string

8. 1999 - 2013 DRI. Some Rights Reserved. 8
Lists
1 <?php
2 //redis_lists.php
3 require_once __DIR__ . '/../vendor/autoload.php';
4 $redis = new PredisClient('tcp://localhost:6379');
5
6 $redis->del('SampleList');
7
8 // Using {L,R}PUSH
9 $redis->lpush('SampleList', 'a'); // a
10 $redis->lpush('SampleList', 'b'); // b, a
11 $redis->rpush('SampleList', 'c'); // b, a, c
12
13 // Using LLEN
14 echo $redis->llen('SampleList') . "n"; // 3
15
16 // Using LINDEX (note: zero indexed)
17 echo $redis->lindex('SampleList', 1) . "n"; // a
18
19 // Using {L,R}POP
20
21 echo $redis->lpop('SampleList') . "n"; // b
22 echo $redis->rpop('SampleList') . "n"; // c
23
24 // Other commands: lrange, rpoplpush
25 // http://redis.io/commands#list

9. 1999 - 2013 DRI. Some Rights Reserved. 9
Sets
1 <?php
2 //redis_sets.php
3 require_once __DIR__ . '/../vendor/autoload.php';
4 $redis = new PredisClient('tcp://localhost:6379');
5
6 $redis->del('SampleSet');
7 $redis->del('OtherSampleSet');
8
9 // Using SADD
10 $redis->sadd('SampleSet', 'Hello'); // Hello
11 $redis->sadd('SampleSet', 'World'); // Hello, World
12 $redis->sadd('SampleSet', 'World'); // Hello, World
13
14 // Using SMEMBERS
15 var_dump($redis->smembers('SampleSet')); // Hello, World
16
17 // Using SINTER
18 $redis->sadd('OtherSampleSet', 'Hello');
19 $redis->sadd('OtherSampleSet', 'All');
20 var_dump($redis->sinter('SampleSet', 'OtherSampleSet')); // Hello
21
22 // Using SUNION
23 var_dump($redis->sunion('SampleSet', 'OtherSampleSet')); // Hello, World, All
24
25 // Other commands: smove, srandmember, srem, scard
26 // http://redis.io/commands#set

10. 1999 - 2013 DRI. Some Rights Reserved. 10
Hashes
1 <?php
2 //redis_hashes.php
3 require_once __DIR__ . '/../vendor/autoload.php';
4 $redis = new PredisClient('tcp://localhost:6379');
5
6 $redis->del('SampleHash');
7
8 // Using HMSET
9 $redis->hmset("SampleHash", array(
10 'prop_a' => 'aaa',
11 'prop_b' => 'bbb',
12 ));
13
14 // Using HGETALL
15 var_dump($redis->hgetall("SampleHash")); // prop_a=>aaa, prop_b=>bbb
16
17 // Using HSET
18 $redis->hset('SampleHash', 'prop_b', 'ccc');
19
20 //USING HGET
21 echo $redis->hget("SampleHash", 'prop_b') ."n"; // ccc
22
23 // Other commands: hexists, hdel, hlen, hkeys, hvals
24 // http://redis.io/commands/#hash

11. 1999 - 2013 DRI. Some Rights Reserved. 11
Sorted Sets
1 <?php
2 //redis_sorted_sets.php
3 require_once __DIR__ . '/../vendor/autoload.php';
4 $redis = new PredisClient('tcp://localhost:6379');
5
6 $redis->del('SampleSortedSet');
7
8 // Using SADD
9 $redis->zadd('SampleSortedSet', 1, 'Hello'); // Hello(1)
10 $redis->zadd('SampleSortedSet', 2, 'World'); // Hello(1), World(2)
11 $redis->zadd('SampleSortedSet', 3, 'World'); // Hello(1), World(3)
12
13 // Using ZRANGE
14 var_dump($redis->zrange('SampleSortedSet', 0, -1,
array('withscores'=>true))); // Hello(1), World(3)
15
16 // Using ZSCORE
17 echo $redis->zscore('SampleSortedSet', 'World') . "n"; // 3
18
19 // Other commands: zrank, zrevrange, zrangebyscore, zremrangebyscore
20 // http://redis.io/commands#sorted_set

12. 1999 - 2013 DRI. Some Rights Reserved. 12
ZeroMQ
“ØMQ is a high-performance asynchronous
messaging library aimed at use in scalable
distributed or concurrent applications. It provides a
message queue, but unlike message-oriented
middleware, a ØMQ system can run without a
dedicated message broker. With bindings for 30+
languages”
source: https://en.wikipedia.org/wiki/%C3%98MQ

13. 1999 - 2013 DRI. Some Rights Reserved. 13
PHP Module - zmq
#
# ZMQ instalation - http://www.zeromq.org/intro:get-the-software
#
tar xzvf zeromq-3.2.2.tar.gz
cd zeromq-3.2.2
./configure
make
make install
echo "/usr/local/lib" > /etc/ld.so.conf.d/usr_local_lib.conf
ldconfig
#
# ZMQ PHP Binding Instalation
#
pear channel-discover pear.zero.mq
pecl install pear.zero.mq/zmq-beta
echo "extension=zmq.so" > /etc/php.d/zmq.ini

14. 1999 - 2013 DRI. Some Rights Reserved. 14
Socket Types
● Threads in a process (inproc://)
● Processes in a box (ipc://)
● Processes over the network (tcp://)
● Multicast group (pgm://)

15. 1999 - 2013 DRI. Some Rights Reserved. 15
Pattern: Request - Reply

16. 1999 - 2013 DRI. Some Rights Reserved. 16
Pattern: Request - Reply
1 <?php
2 // zmq_request.php
3 $context = new ZMQContext();
4
5 $requester = new ZMQSocket($context, ZMQ::SOCKET_REQ);
6 $requester->connect("tcp://localhost:5555");
7
8 for ($number = 0 ; $number <= 10 ; $number++) {
9 $mensage = "Hello " . $number . "!";
10 echo "Sending - " . $mensage . "n";
11 $requester->send($mensage);
12 $reply = $requester->recv();
13 echo "Received - " . $reply . "n";
14 }
1 <?php
2 //zmq_reply.php
3 $context = new ZMQContext();
4
5 $responder = new ZMQSocket($context, ZMQ::SOCKET_REP);
6 $responder->bind("tcp://*:5555");
7
8 while (true) {
9 $request = $responder->recv();
10 echo $request . "n";
11 sleep (1); // some work
12 $responder->send("Reply to: " . $request);
13 }

17. 1999 - 2013 DRI. Some Rights Reserved. 17
Pattern: Publish - Subscribe

18. 1999 - 2013 DRI. Some Rights Reserved. 18
Pattern: Publish - Subscribe
1 <?php
2 // zmq_publisher.php
3 $context = new ZMQContext();
4
5 $publisher = $context->getSocket(ZMQ::SOCKET_PUB);
6 $publisher->bind("tcp://*:5556");
7
8 $timezones = array('UTC', 'EST');
9 while (true) {
10 foreach($timezones as $tz) {
11 date_default_timezone_set($tz);
12 $message = date('T:c'); // the message to broadcast
13 $publisher->send($message);
14 }
15 sleep(1);
16 } 1 <?php
2 // zmq_subscriber.php
3 $context = new ZMQContext();
4
5 $subscriber = $context->getSocket(ZMQ::SOCKET_SUB);
6 $subscriber->connect("tcp://localhost:5556");
7
8 $filter = $_SERVER['argc'] > 1 ? $_SERVER['argv'][1] : "UTC";
9 $subscriber->setSockOpt(ZMQ::SOCKOPT_SUBSCRIBE, $filter);
10
11 while (true) {
12 $message = $subscriber->recv();
13 echo substr($message,4) . "n"; // remove prefix
14 }

19. 1999 - 2013 DRI. Some Rights Reserved. 19
Pattern: Pipeline

20. 20
Pattern: Pipeline
1 <?php
2 // zmq_ventilator.php
3 // extracted from: http://zguide.zeromq.org/
4
5 $context = new ZMQContext();
6
7 // Socket to send messages on
8 $sender = new ZMQSocket($context, ZMQ::SOCKET_PUSH);
9 $sender->bind("tcp://*:5557");
10
11 echo "Press Enter when the workers are ready: ";
12 $fp = fopen('php://stdin', 'r');
13 $line = fgets($fp, 512);
14 fclose($fp);
15 echo "Sending tasks to workersâÀ¦", PHP_EOL;
16
17 // The first message is "0" and signals start of batch
18 $sender->send(0);
19
20 // Send 100 tasks
21 $total_msec = 0; // Total expected cost in msecs
22 for ($task_nbr = 0; $task_nbr < 100; $task_nbr++) {
23 // Random workload from 1 to 100msecs
24 $workload = mt_rand(1, 100);
25 $total_msec += $workload;
26 $sender->send($workload);
27
28 }
29
30 printf ("Total expected cost: %d msecn", $total_msec);
31 sleep (1); // Give 0MQ time to deliver

21. 1999 - 2013 DRI. Some Rights Reserved. 21
Pattern: Pipeline
1 <?php
2 // zmq_worker.php
3 // extracted from: http://zguide.zeromq.org/
4
5 $context = new ZMQContext();
6
7 // Socket to receive messages on
8 $receiver = new ZMQSocket($context, ZMQ::SOCKET_PULL);
9 $receiver->connect("tcp://localhost:5557");
10
11 // Socket to send messages to
12 $sender = new ZMQSocket($context, ZMQ::SOCKET_PUSH);
13 $sender->connect("tcp://localhost:5558");
14
15 // Process tasks forever
16 while (true) {
17 $string = $receiver->recv();
18
19 // Simple progress indicator for the viewer
20 echo $string, PHP_EOL;
21
22 // Do the work
23 usleep($string * 1000);
24
25 // Send results to sink
26 $sender->send("");
27 }

22. 22
Pattern: Pipeline
1 <?php
2 // zmq_sink.php
3 // extracted from: http://zguide.zeromq.org/
4
5 // Prepare our context and socket
6 $context = new ZMQContext();
7 $receiver = new ZMQSocket($context, ZMQ::SOCKET_PULL);
8 $receiver->bind("tcp://*:5558");
9
10 // Wait for start of batch
11 $string = $receiver->recv();
12
13 // Start our clock now
14 $tstart = microtime(true);
15
16 // Process 100 confirmations
17 $total_msec = 0; // Total calculated cost in msecs
18 for ($task_nbr = 0; $task_nbr < 100; $task_nbr++) {
19 $string = $receiver->recv();
20 if ($task_nbr % 10 == 0) {
21 echo ":";
22 } else {
23 echo ".";
24 }
25 }
26
27 $tend = microtime(true);
28
29 $total_msec = ($tend - $tstart) * 1000;
30 echo PHP_EOL;
31 printf ("Total elapsed time: %d msec", $total_msec);
32 echo PHP_EOL;

23. 1999 - 2013 DRI. Some Rights Reserved. 23
Pattern: Pipeline

24. 1999 - 2013 DRI. Some Rights Reserved. 24
Use Case: Service Cluster
Reference: http://zguide.zeromq.org/page:all#Service-Oriented-Reliable-Queuing-Majordomo-Pattern

25. 1999 - 2013 DRI. Some Rights Reserved. 25
Use Case: “Unix Style” Application
● Lots of simple, “focused” programs
– Or in different servers, different languages, etc
● All components collaborate to get the
job done
– cat file | sort | uniq -c
● Glue everything with ZeroMQ

26. 1999 - 2013 DRI. Some Rights Reserved. 26
Use Case: Cache
● We need to cache some values for
speed
● Use SET / GET on Redis
● I could use memcache, but this is a
Redis Talk :-)
● They have similar performance
– http://redis.io/topics/benchmarks

27. 1999 - 2013 DRI. Some Rights Reserved. 27
Use Case: Buffer
● Use Redis as a FIFO (or LIFO)
● Hidden Goods
– Decoupling
– Flow control
– rpoplpush

28. 1999 - 2013 DRI. Some Rights Reserved. 28
Use Case: Background Tasks
● Your application needs to do some
heavy work
– Send Email
– Image Resizing
– Mega Huge map-reduce query to your pentabyte cluster :-)
● You don't want to keep your user
waiting
● Send things to background

29. 1999 - 2013 DRI. Some Rights Reserved. 29
Use Case: Background Tasks
● Use Redis as your Job Queue
– Your application should send job to be run and parameters to the
queue
● Use workers to POP jobs from the
queue and do the heavy work.
● Don't reinvent the well
– Look for php-resqueue

30. 1999 - 2013 DRI. Some Rights Reserved. 30
Conclusions
● Both ZeroMQ and Redis are extremely
fast
● ZeroMQ is great to “glue things” as well
as adding flexibility to scale dynamical
● Redis is great as a queue and allows
you to cope with your load.

31. Thank you

32. Follow this topic:
@rjsmelo
QA
Code: https://github.com/rjsmelo/talk-redis-zmq
Feedback: https://joind.in/talk/view/8937

33. www.dri-global.com
@rjsmelo
ricardo.melo@dri-global.com