Redis & ZeroMQ: How to scale your application

20,043 views

Published on

Presented at #PHPLX 11 July 2013

When you need to do some heavy processing how do you scale you application?

You can use Redis and ZeroMQ to leverage the heavy work for you!

With this presentation we will know more about this two technologies and how they can be used to help solve problems with the performance and scalability of your application.

Published in: Technology
0 Comments
44 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
20,043
On SlideShare
0
From Embeds
0
Number of Embeds
1,832
Actions
Shares
0
Downloads
289
Comments
0
Likes
44
Embeds 0
No embeds

No notes for slide

Redis & ZeroMQ: How to scale your application

  1. 1. Follow this topic: @rjsmelo Redis & ZeroMQ: How to scale your application RICARDO MELO Presented at #PHPLX – 11 July 2013
  2. 2. @rjsmelo 2 RICARDO MELO ● CTO @ DRI ● PHP, Mysql, Linux and lots of other OSS ● ZCE, RHCE, LPI 3, ITIL, etc ● +10 years building (and breaking) things
  3. 3. @rjsmelo 3 About ● 14 Year old academic spin-off ● Pragmatic OSS Orientation ● PHP, Mysql, SugarCRM, Drupal, JavaScript, Linux, etc. ● Crafters, Integrators ● Always looking for software developers – Yes, right now!
  4. 4. 1999 - 2013 DRI. Some Rights Reserved. 4 Outline ● Redis ● ZeroMQ ● Use Cases ● Conclusions
  5. 5. 1999 - 2013 DRI. Some Rights Reserved. 5 Redis “Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.” source: http://redis.io
  6. 6. 1999 - 2013 DRI. Some Rights Reserved. 6 Predis ● Pure PHP Redis Client – https://github.com/nrk/predis ● Use composer: – composer install { "name": "rjsmelo/talk-redis-zmq", "description": "Sample code for Redis & ZeroMQ presentation", "require": { "ext-zmq": "*", "predis/predis": "dev-master" }, "license": "Apache-2.0", "authors": [ { "name": "Ricardo Melo", "email": "ricardo.melo@dri.pt" } ] }
  7. 7. 1999 - 2013 DRI. Some Rights Reserved. 7 Strings 1 <?php 2 //redis_strings.php 3 require_once __DIR__ . '/../vendor/autoload.php'; 4 $redis = new PredisClient('tcp://localhost:6379'); 5 6 // Using SET 7 $redis->set("SampleKey", "Throw me any thing here...."); 8 echo $redis->get("SampleKey") . "n"; // Throw me any thing here.... 9 10 // Remove Key 11 $redis->del("SampleKey"); 12 13 // Using APPEND 14 $redis->append("SampleKey", "Hello"); // Hello 15 $redis->append("SampleKey", " World!"); // Hello World! 16 echo $redis->get("SampleKey") . "n"; 17 18 // Other commands: incr, decr, incrby, getrange, setrange 19 // http://redis.io/commands/#string
  8. 8. 1999 - 2013 DRI. Some Rights Reserved. 8 Lists 1 <?php 2 //redis_lists.php 3 require_once __DIR__ . '/../vendor/autoload.php'; 4 $redis = new PredisClient('tcp://localhost:6379'); 5 6 $redis->del('SampleList'); 7 8 // Using {L,R}PUSH 9 $redis->lpush('SampleList', 'a'); // a 10 $redis->lpush('SampleList', 'b'); // b, a 11 $redis->rpush('SampleList', 'c'); // b, a, c 12 13 // Using LLEN 14 echo $redis->llen('SampleList') . "n"; // 3 15 16 // Using LINDEX (note: zero indexed) 17 echo $redis->lindex('SampleList', 1) . "n"; // a 18 19 // Using {L,R}POP 20 21 echo $redis->lpop('SampleList') . "n"; // b 22 echo $redis->rpop('SampleList') . "n"; // c 23 24 // Other commands: lrange, rpoplpush 25 // http://redis.io/commands#list
  9. 9. 1999 - 2013 DRI. Some Rights Reserved. 9 Sets 1 <?php 2 //redis_sets.php 3 require_once __DIR__ . '/../vendor/autoload.php'; 4 $redis = new PredisClient('tcp://localhost:6379'); 5 6 $redis->del('SampleSet'); 7 $redis->del('OtherSampleSet'); 8 9 // Using SADD 10 $redis->sadd('SampleSet', 'Hello'); // Hello 11 $redis->sadd('SampleSet', 'World'); // Hello, World 12 $redis->sadd('SampleSet', 'World'); // Hello, World 13 14 // Using SMEMBERS 15 var_dump($redis->smembers('SampleSet')); // Hello, World 16 17 // Using SINTER 18 $redis->sadd('OtherSampleSet', 'Hello'); 19 $redis->sadd('OtherSampleSet', 'All'); 20 var_dump($redis->sinter('SampleSet', 'OtherSampleSet')); // Hello 21 22 // Using SUNION 23 var_dump($redis->sunion('SampleSet', 'OtherSampleSet')); // Hello, World, All 24 25 // Other commands: smove, srandmember, srem, scard 26 // http://redis.io/commands#set
  10. 10. 1999 - 2013 DRI. Some Rights Reserved. 10 Hashes 1 <?php 2 //redis_hashes.php 3 require_once __DIR__ . '/../vendor/autoload.php'; 4 $redis = new PredisClient('tcp://localhost:6379'); 5 6 $redis->del('SampleHash'); 7 8 // Using HMSET 9 $redis->hmset("SampleHash", array( 10 'prop_a' => 'aaa', 11 'prop_b' => 'bbb', 12 )); 13 14 // Using HGETALL 15 var_dump($redis->hgetall("SampleHash")); // prop_a=>aaa, prop_b=>bbb 16 17 // Using HSET 18 $redis->hset('SampleHash', 'prop_b', 'ccc'); 19 20 //USING HGET 21 echo $redis->hget("SampleHash", 'prop_b') ."n"; // ccc 22 23 // Other commands: hexists, hdel, hlen, hkeys, hvals 24 // http://redis.io/commands/#hash
  11. 11. 1999 - 2013 DRI. Some Rights Reserved. 11 Sorted Sets 1 <?php 2 //redis_sorted_sets.php 3 require_once __DIR__ . '/../vendor/autoload.php'; 4 $redis = new PredisClient('tcp://localhost:6379'); 5 6 $redis->del('SampleSortedSet'); 7 8 // Using SADD 9 $redis->zadd('SampleSortedSet', 1, 'Hello'); // Hello(1) 10 $redis->zadd('SampleSortedSet', 2, 'World'); // Hello(1), World(2) 11 $redis->zadd('SampleSortedSet', 3, 'World'); // Hello(1), World(3) 12 13 // Using ZRANGE 14 var_dump($redis->zrange('SampleSortedSet', 0, -1, array('withscores'=>true))); // Hello(1), World(3) 15 16 // Using ZSCORE 17 echo $redis->zscore('SampleSortedSet', 'World') . "n"; // 3 18 19 // Other commands: zrank, zrevrange, zrangebyscore, zremrangebyscore 20 // http://redis.io/commands#sorted_set
  12. 12. 1999 - 2013 DRI. Some Rights Reserved. 12 ZeroMQ “ØMQ is a high-performance asynchronous messaging library aimed at use in scalable distributed or concurrent applications. It provides a message queue, but unlike message-oriented middleware, a ØMQ system can run without a dedicated message broker. With bindings for 30+ languages” source: https://en.wikipedia.org/wiki/%C3%98MQ
  13. 13. 1999 - 2013 DRI. Some Rights Reserved. 13 PHP Module - zmq # # ZMQ instalation - http://www.zeromq.org/intro:get-the-software # tar xzvf zeromq-3.2.2.tar.gz cd zeromq-3.2.2 ./configure make make install echo "/usr/local/lib" > /etc/ld.so.conf.d/usr_local_lib.conf ldconfig # # ZMQ PHP Binding Instalation # pear channel-discover pear.zero.mq pecl install pear.zero.mq/zmq-beta echo "extension=zmq.so" > /etc/php.d/zmq.ini
  14. 14. 1999 - 2013 DRI. Some Rights Reserved. 14 Socket Types ● Threads in a process (inproc://) ● Processes in a box (ipc://) ● Processes over the network (tcp://) ● Multicast group (pgm://)
  15. 15. 1999 - 2013 DRI. Some Rights Reserved. 15 Pattern: Request - Reply
  16. 16. 1999 - 2013 DRI. Some Rights Reserved. 16 Pattern: Request - Reply 1 <?php 2 // zmq_request.php 3 $context = new ZMQContext(); 4 5 $requester = new ZMQSocket($context, ZMQ::SOCKET_REQ); 6 $requester->connect("tcp://localhost:5555"); 7 8 for ($number = 0 ; $number <= 10 ; $number++) { 9 $mensage = "Hello " . $number . "!"; 10 echo "Sending - " . $mensage . "n"; 11 $requester->send($mensage); 12 $reply = $requester->recv(); 13 echo "Received - " . $reply . "n"; 14 } 1 <?php 2 //zmq_reply.php 3 $context = new ZMQContext(); 4 5 $responder = new ZMQSocket($context, ZMQ::SOCKET_REP); 6 $responder->bind("tcp://*:5555"); 7 8 while (true) { 9 $request = $responder->recv(); 10 echo $request . "n"; 11 sleep (1); // some work 12 $responder->send("Reply to: " . $request); 13 }
  17. 17. 1999 - 2013 DRI. Some Rights Reserved. 17 Pattern: Publish - Subscribe
  18. 18. 1999 - 2013 DRI. Some Rights Reserved. 18 Pattern: Publish - Subscribe 1 <?php 2 // zmq_publisher.php 3 $context = new ZMQContext(); 4 5 $publisher = $context->getSocket(ZMQ::SOCKET_PUB); 6 $publisher->bind("tcp://*:5556"); 7 8 $timezones = array('UTC', 'EST'); 9 while (true) { 10 foreach($timezones as $tz) { 11 date_default_timezone_set($tz); 12 $message = date('T:c'); // the message to broadcast 13 $publisher->send($message); 14 } 15 sleep(1); 16 } 1 <?php 2 // zmq_subscriber.php 3 $context = new ZMQContext(); 4 5 $subscriber = $context->getSocket(ZMQ::SOCKET_SUB); 6 $subscriber->connect("tcp://localhost:5556"); 7 8 $filter = $_SERVER['argc'] > 1 ? $_SERVER['argv'][1] : "UTC"; 9 $subscriber->setSockOpt(ZMQ::SOCKOPT_SUBSCRIBE, $filter); 10 11 while (true) { 12 $message = $subscriber->recv(); 13 echo substr($message,4) . "n"; // remove prefix 14 }
  19. 19. 1999 - 2013 DRI. Some Rights Reserved. 19 Pattern: Pipeline
  20. 20. 20 Pattern: Pipeline 1 <?php 2 // zmq_ventilator.php 3 // extracted from: http://zguide.zeromq.org/ 4 5 $context = new ZMQContext(); 6 7 // Socket to send messages on 8 $sender = new ZMQSocket($context, ZMQ::SOCKET_PUSH); 9 $sender->bind("tcp://*:5557"); 10 11 echo "Press Enter when the workers are ready: "; 12 $fp = fopen('php://stdin', 'r'); 13 $line = fgets($fp, 512); 14 fclose($fp); 15 echo "Sending tasks to workersâÀ¦", PHP_EOL; 16 17 // The first message is "0" and signals start of batch 18 $sender->send(0); 19 20 // Send 100 tasks 21 $total_msec = 0; // Total expected cost in msecs 22 for ($task_nbr = 0; $task_nbr < 100; $task_nbr++) { 23 // Random workload from 1 to 100msecs 24 $workload = mt_rand(1, 100); 25 $total_msec += $workload; 26 $sender->send($workload); 27 28 } 29 30 printf ("Total expected cost: %d msecn", $total_msec); 31 sleep (1); // Give 0MQ time to deliver
  21. 21. 1999 - 2013 DRI. Some Rights Reserved. 21 Pattern: Pipeline 1 <?php 2 // zmq_worker.php 3 // extracted from: http://zguide.zeromq.org/ 4 5 $context = new ZMQContext(); 6 7 // Socket to receive messages on 8 $receiver = new ZMQSocket($context, ZMQ::SOCKET_PULL); 9 $receiver->connect("tcp://localhost:5557"); 10 11 // Socket to send messages to 12 $sender = new ZMQSocket($context, ZMQ::SOCKET_PUSH); 13 $sender->connect("tcp://localhost:5558"); 14 15 // Process tasks forever 16 while (true) { 17 $string = $receiver->recv(); 18 19 // Simple progress indicator for the viewer 20 echo $string, PHP_EOL; 21 22 // Do the work 23 usleep($string * 1000); 24 25 // Send results to sink 26 $sender->send(""); 27 }
  22. 22. 22 Pattern: Pipeline 1 <?php 2 // zmq_sink.php 3 // extracted from: http://zguide.zeromq.org/ 4 5 // Prepare our context and socket 6 $context = new ZMQContext(); 7 $receiver = new ZMQSocket($context, ZMQ::SOCKET_PULL); 8 $receiver->bind("tcp://*:5558"); 9 10 // Wait for start of batch 11 $string = $receiver->recv(); 12 13 // Start our clock now 14 $tstart = microtime(true); 15 16 // Process 100 confirmations 17 $total_msec = 0; // Total calculated cost in msecs 18 for ($task_nbr = 0; $task_nbr < 100; $task_nbr++) { 19 $string = $receiver->recv(); 20 if ($task_nbr % 10 == 0) { 21 echo ":"; 22 } else { 23 echo "."; 24 } 25 } 26 27 $tend = microtime(true); 28 29 $total_msec = ($tend - $tstart) * 1000; 30 echo PHP_EOL; 31 printf ("Total elapsed time: %d msec", $total_msec); 32 echo PHP_EOL;
  23. 23. 1999 - 2013 DRI. Some Rights Reserved. 23 Pattern: Pipeline
  24. 24. 1999 - 2013 DRI. Some Rights Reserved. 24 Use Case: Service Cluster Reference: http://zguide.zeromq.org/page:all#Service-Oriented-Reliable-Queuing-Majordomo-Pattern
  25. 25. 1999 - 2013 DRI. Some Rights Reserved. 25 Use Case: “Unix Style” Application ● Lots of simple, “focused” programs – Or in different servers, different languages, etc ● All components collaborate to get the job done – cat file | sort | uniq -c ● Glue everything with ZeroMQ
  26. 26. 1999 - 2013 DRI. Some Rights Reserved. 26 Use Case: Cache ● We need to cache some values for speed ● Use SET / GET on Redis ● I could use memcache, but this is a Redis Talk :-) ● They have similar performance – http://redis.io/topics/benchmarks
  27. 27. 1999 - 2013 DRI. Some Rights Reserved. 27 Use Case: Buffer ● Use Redis as a FIFO (or LIFO) ● Hidden Goods – Decoupling – Flow control – rpoplpush
  28. 28. 1999 - 2013 DRI. Some Rights Reserved. 28 Use Case: Background Tasks ● Your application needs to do some heavy work – Send Email – Image Resizing – Mega Huge map-reduce query to your pentabyte cluster :-) ● You don't want to keep your user waiting ● Send things to background
  29. 29. 1999 - 2013 DRI. Some Rights Reserved. 29 Use Case: Background Tasks ● Use Redis as your Job Queue – Your application should send job to be run and parameters to the queue ● Use workers to POP jobs from the queue and do the heavy work. ● Don't reinvent the well – Look for php-resqueue
  30. 30. 1999 - 2013 DRI. Some Rights Reserved. 30 Conclusions ● Both ZeroMQ and Redis are extremely fast ● ZeroMQ is great to “glue things” as well as adding flexibility to scale dynamical ● Redis is great as a queue and allows you to cope with your load.
  31. 31. Thank you
  32. 32. Follow this topic: @rjsmelo QA Code: https://github.com/rjsmelo/talk-redis-zmq Feedback: https://joind.in/talk/view/8937
  33. 33. www.dri-global.com @rjsmelo ricardo.melo@dri-global.com

×