Gearman work queue in php
Upcoming SlideShare
Loading in...5
×
 

Gearman work queue in php

on

  • 5,569 views

Gearman work queue in php

Gearman work queue in php
用 Gearman 打造 PHP 排程

Statistics

Views

Total Views
5,569
Views on SlideShare
2,187
Embed Views
3,382

Actions

Likes
20
Downloads
64
Comments
0

18 Embeds 3,382

http://blog.wu-boy.com 3171
http://cloud.feedly.com 88
http://feeds2.feedburner.com 62
http://10.0.1.42 14
http://jerry54010.kd.io 12
http://digg.com 9
http://www.feedspot.com 7
http://192.168.1.59 3
http://www.plurk.com 3
http://localhost 2
http://feedly.com 2
https://www.google.com.tw 2
http://news.google.com 2
http://webcache.googleusercontent.com 1
http://reader.aol.com 1
http://www.newsblur.com 1
http://www.inoreader.com 1
http://mlb.wu-boy.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Gearman work queue in php Gearman work queue in php Presentation Transcript

  • Gearman work queue in php 用 Gearman 打造 PHP 排程 2013.10.05 @appleboy
  • 2 Who am I Bo-Yi Wu @appleboy http://blog.wu-boy.com https://github.com/appleboy 任職於瑞昱半導體 RealTek(IC Design House)
  • 3 今日來聊聊排程系統
  • 4 排程使用時機 ?
  • 5 不需要即時回覆 都可以使用排程
  • 6 哪些應用需要排程 ?
  • 7 使用者註冊系統
  • 8 使用者註冊 填寫表單完成 寄發認證信 使用者收到回覆 未使用排程 處理資料 透過第三方寄信 Amazone SES
  • 9 缺點 Response time 過長 使用者不需要等伺服器寄信 ( 來自知名插畫家彎彎 )
  • 10 使用者註冊 填寫表單完成 寄發認證信 使用者收到回覆 使用排程 處理資料 寫入排程
  • 11 Email Queue? Github Social Coding
  • 12 Application Server Database Database Database Work Queue
  • 13 執行排程 Linux crontab Ref: http://www.xenstreet.com/2013/03/choosing-the-right-linux/
  • 14 <?php $link = mysqli_connect("host", "user", "passw", "db"); $query = 'SELECT * FROM jobs where status = "on"'; $result = $link->query($query); while($row = mysqli_fecth_array($result)) { // ….... // do work // ….... $sql = 'update jobs set status = "off" where id = {$row["id"]}'; $link->query($sql); }
  • 15 每分鐘執行一次 */1 * * * * root cron.sh
  • 16 優點 ● 容易瞭解 ● 任何程式語言都可以實現 ● 快速 Deploy 到任何機器
  • 17 缺點 ● 每分鐘只能執行一次 ● Race condition ● 浪費 Database 資源
  • 18 換個寫法來試試看
  • 19 每 10 秒執行一次 <?php $link = mysqli_connect("myhost","myuser","mypassw","mybd"); $query = 'SELECT * FROM jobs where status = "on"'; $done = false; while(!$done) { $result = $link->query($query); if ($result->num_rows > 0) { // do work $sql = 'update jobs set status = "off" where id = {$row["id"]}'; $link->query($sql); } else { sleep(10); } }
  • 20 缺點 ● 真的 10 秒執行一次 ? ● Race condition ● 更浪費 Database 資源
  • 21 如果沒有 Race Condition
  • 22 #!/bin/sh (sleep 5 && /path/script.sh) & (sleep 15 && /path/script.sh) & (sleep 25 && /path/script.sh) & (sleep 35 && /path/script.sh) & (sleep 45 && /path/script.sh) & (sleep 55 && /path/script.sh) & 每 10 秒執行一次
  • 23 如何分配處理這些問題 ?
  • 24 Using Gearman
  • 25 What is Gearman? It allows you to do work in parallel, to load balance processing, and to call functions between languages.
  • 26 Client in Perl,Python,PHP Java,C#(.Net)
  • 27 even MySQL,PostgreSQL
  • 28 How Does Gearman Work?
  • 29 Gearman 優點
  • 30 Work Queues ● 淺顯易懂 Got job do job repeat→ → Application Gearmand Worker Got job Do job
  • 31 Work Queues ● 淺顯易懂 Get job do job repeat→ → ● 跨語言 任何平台只要能存入字串即可
  • 32 Multiple Language Application Gearmand Application Worker Worker
  • 33 Work Queues ● 淺顯易懂 Get job do job repeat→ → ● 跨語言 任何平台只要能存入字串即可 ● 跨平台 當 Client 寫入工作時, Worker 不需要啟動
  • 34 跨平台 Application Gearmand Application Gearmand
  • 35 Work Queues ● 淺顯易懂 Get job do job repeat→ → ● 跨語言 任何平台只要能存入字串即可 ● 跨平台 當 Client 寫入工作時, Worker 不需要啟動 ● 即時性 可以為同步或非同步
  • 36 Gearman Architecture 佈署架構
  • 37 Application Gearmand Job
  • 38 Application Gearmand Job Worker Job
  • 39 Application Gearmand Job Worker Job Response
  • 40 Application Gearmand Job Worker Job Response Response
  • 41 Application Gearmand Worker Worker Worker
  • 42 Application Gearmand Worker Worker Gearmand
  • 43 Application Gearmand Worker Worker Gearmand Application
  • 44 結論 :Gearman 擴充性佳
  • 45 不論要幾台 gearmand 跟數個 worker 都可以
  • 46 Application 只會跟 Gearmand 溝通
  • 47 多個 Application 不管 丟出多少個 Jobs
  • 48 最終由多個 Job Server 分配工作
  • 49 Synchronous vs Asynchronous
  • 50 Synchronous Application Gearmand Job Worker Job Response Response
  • 51 Asynchronous Application Gearmand Job Worker Job Done!
  • 52 Installation 安裝方式
  • 53 Installation ● Job Server (C) ● Client library (PHP,Python ...) ● Worker library (PHP,Python ...)
  • 54 Job Server 安裝方式
  • 55 Ubuntu/Debian aptitude -y install gearman gearman-job-server libgearman-dev libdrizzle0
  • 56 CentOS/Fedora yum -y install libgearman gearmand libgearman-devel libdrizzle
  • 57 Tarball install https://gist.github.com/appleboy/6774735
  • 58 Gearmand enabled using MySQL(Mariadb) Postgresql SQL Lite Memcached
  • 59 MySQL Example /etc/default/gearman-job-server -q mysql --mysql-host=localhost --mysql-user=xx --mysql-password=xx --mysql-db=ovoq --mysql-table=gearman_queue
  • 60 Client Library pecl install channel://pecl.php.net/gearman-1.1.2 echo 'extension=gearman.so' > /path/gearman.ini
  • 61 Quick Test php -i | grep 'gearman'
  • 62 Gearman in PHP Client
  • 63 PHP Client # Create a client object $gm = new GearmanClient(); # Add default server, default is localhost:4730 $gm->addServer(); # Do Job $result = $gm->doNormal("reverse", "Hello!"); echo "Success: $resultn";
  • 64 Do background Job $job = $gm->doBackground("reverse", "apple");
  • 65 Gearman in PHP Worker
  • 66 PHP Worker # Define worker job function function reverse_fn($job) { //.... } # fetch message from client function reverse_fn($job) { $data = $job->workload(); }
  • 67 PHP Worker # Create our worker object. $gm= new GearmanWorker(); # Add default server (localhost). $gm->addServer(); # Register function "reverse" $gm->addFunction("reverse", "reverse_fn"); # Waiting for job... while($gm->work());
  • 68 Multiple Jobs $gm->addFunction("reverse", "reverse_fn"); $gm->addFunction("resize", "resize_fn"); $gm->addFunction("queue", "queue_fn"); …...
  • 69 Anonymous function $gm->addFunction("reverse", function($job){ $data = $job->workload(); });
  • 70 Gearman in CodeIgniter
  • 71 CodeIgniter Gearman Library https://github.com/appleboy/CodeIgniter-Gearman-Library
  • 72 Installation via Spark php tools/spark install -v1.0.1 codeigniter-gearman
  • 73 Setup Gearmand Server and Port config/gearman.php $config['gearman_server'] = array('localhost'); $config['gearman_port'] = array('4730');
  • 74 CodeIgniter Client
  • 75 Implement Image Resize Job create app controller
  • 76 class App extends CI_Controller { public function resize_sync() public function resize_async() }
  • 77 public function resize_sync() { $data = json_encode(array('file_name' => 'test.png')); $client = $this->lib_gearman->gearman_client(); do { $this->lib_gearman->do_job_foreground('resize', $data); switch($client->returnCode()) { //.... } } while ($client->returnCode() != GREARMAN_SUCCESS); }
  • 78 class App extends CI_Controller { public function resize_sync() public function resize_async() }
  • 79 public function resize_async() { $data = json_encode(array('file_name' => 'test.png')); $client = $this->lib_gearman->gearman_client(); $this->lib_gearman->do_job_background('resize', $data)); }
  • 80 CodeIgniter Worker create cli controller
  • 81 class Cli extends CI_Controller { public static function image_resize() public static function email_queue() public function worker() }
  • 82 public static function image_resize($job) { $data = json_decode($job->workload(), true); $ci =& get_instance(); $ci->image_lib->initialize($config); $ci->image_lib->resize(); $ci->image_lib->clear(); }
  • 83 Define Worker
  • 84 public function worker() { $worker = $this->lib_gearman->gearman_worker(); $this->lib_gearman ->add_worker_function('resize', 'Cli::image_resize'); $this->lib_gearman ->add_worker_function('queue', 'Cli::email_queue'); while ($this->lib_gearman->work()); }
  • 85 Start Worker $ php index.php cli worker
  • 86 Running workers via script #!/bin/sh script="index.php cli worker" folder="/script_path/" max_worker="5" php_bin="/usr/bin/php" count=0 current_count=$(ps aux | grep "index.php cli worker" | grep -v "grep" | wc -l) if test $current_count -lt $max_worker then cd $folder $php_bin $folder$script & else echo "The worker count has been limited." fi https://gist.github.com/appleboy/6806323
  • 87 About Job Data Don't pass 2GB object. Gearman is not data center.
  • 88 Keep your message data small Amazone SQS message siez: 256K http://goo.gl/0MBq8W
  • 89 Scalability
  • 90 Client Client Job Server Client ClientWorker Worker Worker Worker Job Server Client
  • 91 you can scale out your clients and workers as needed
  • 92 Client Setting Client1, Client2: Gearman1 Client3, Client4: Gearman2 …...
  • 93 Worker Setting Worker[1-4]:Server[1-2] …...
  • 94 Add new Job Server...
  • 95 Reconfigure all setting (client and worker program)
  • 96 Reconfigure all setting (client and worker program) even Deploy Tool
  • 97 如何控制易怒的情緒?
  • 98 How to improve your architecture?
  • 99 Client Client Job Server Client ClientWorker Worker Worker Worker Job Server Client Proxy (Haproxy ...)
  • 100 Live Demo Email Queue.
  • 101 Any Questions?
  • 102 Thanks 全體工作人員和聽眾