Your SlideShare is downloading. ×
0
Gearman work queue in php
用 Gearman 打造 PHP 排程
2013.10.05
@appleboy
2
Who am I
Bo-Yi Wu @appleboy
http://blog.wu-boy.com
https://github.com/appleboy
任職於瑞昱半導體 RealTek(IC Design House)
3
今日來聊聊排程系統
4
排程使用時機 ?
5
不需要即時回覆
都可以使用排程
6
哪些應用需要排程 ?
7
使用者註冊系統
8
使用者註冊
填寫表單完成
寄發認證信
使用者收到回覆
未使用排程
處理資料
透過第三方寄信
Amazone SES
9
缺點 Response time 過長
使用者不需要等伺服器寄信
( 來自知名插畫家彎彎 )
10
使用者註冊
填寫表單完成
寄發認證信
使用者收到回覆
使用排程
處理資料
寫入排程
11
Email Queue?
Github Social Coding
12
Application Server
Database Database Database
Work Queue
13
執行排程
Linux crontab
Ref: http://www.xenstreet.com/2013/03/choosing-the-right-linux/
14
<?php
$link = mysqli_connect("host", "user", "passw", "db");
$query = 'SELECT * FROM jobs where status = "on"';
$result...
15
每分鐘執行一次
*/1 * * * * root cron.sh
16
優點
●
容易瞭解
●
任何程式語言都可以實現
●
快速 Deploy 到任何機器
17
缺點
●
每分鐘只能執行一次
●
Race condition
●
浪費 Database 資源
18
換個寫法來試試看
19
每 10 秒執行一次
<?php
$link = mysqli_connect("myhost","myuser","mypassw","mybd");
$query = 'SELECT * FROM jobs where status ...
20
缺點
●
真的 10 秒執行一次 ?
●
Race condition
●
更浪費 Database 資源
21
如果沒有 Race Condition
22
#!/bin/sh
(sleep 5 && /path/script.sh) &
(sleep 15 && /path/script.sh) &
(sleep 25 && /path/script.sh) &
(sleep 35 && /...
23
如何分配處理這些問題 ?
24
Using Gearman
25
What is Gearman?
It allows you to do work in
parallel, to load balance
processing, and to call
functions between langua...
26
Client in
Perl,Python,PHP
Java,C#(.Net)
27
even
MySQL,PostgreSQL
28
How Does Gearman Work?
29
Gearman 優點
30
Work Queues
●
淺顯易懂
Got job do job repeat→ →
Application
Gearmand
Worker
Got job
Do job
31
Work Queues
●
淺顯易懂
Get job do job repeat→ →
●
跨語言
任何平台只要能存入字串即可
32
Multiple Language
Application
Gearmand
Application
Worker Worker
33
Work Queues
●
淺顯易懂
Get job do job repeat→ →
●
跨語言
任何平台只要能存入字串即可
●
跨平台
當 Client 寫入工作時, Worker 不需要啟動
34
跨平台
Application
Gearmand
Application
Gearmand
35
Work Queues
●
淺顯易懂
Get job do job repeat→ →
●
跨語言
任何平台只要能存入字串即可
●
跨平台
當 Client 寫入工作時, Worker 不需要啟動
●
即時性
可以為同步或非同步
36
Gearman Architecture
佈署架構
37
Application
Gearmand
Job
38
Application
Gearmand
Job
Worker
Job
39
Application
Gearmand
Job
Worker
Job Response
40
Application
Gearmand
Job
Worker
Job Response
Response
41
Application
Gearmand
Worker
Worker
Worker
42
Application
Gearmand
Worker
Worker
Gearmand
43
Application
Gearmand
Worker
Worker
Gearmand
Application
44
結論 :Gearman 擴充性佳
45
不論要幾台 gearmand
跟數個 worker 都可以
46
Application 只會跟
Gearmand 溝通
47
多個 Application 不管
丟出多少個 Jobs
48
最終由多個 Job Server
分配工作
49
Synchronous
vs
Asynchronous
50
Synchronous
Application
Gearmand
Job
Worker
Job Response
Response
51
Asynchronous
Application
Gearmand
Job
Worker
Job Done!
52
Installation
安裝方式
53
Installation
●
Job Server (C)
●
Client library (PHP,Python ...)
●
Worker library (PHP,Python ...)
54
Job Server
安裝方式
55
Ubuntu/Debian
aptitude -y install gearman gearman-job-server libgearman-dev libdrizzle0
56
CentOS/Fedora
yum -y install libgearman gearmand libgearman-devel libdrizzle
57
Tarball install
https://gist.github.com/appleboy/6774735
58
Gearmand enabled using
MySQL(Mariadb)
Postgresql
SQL Lite
Memcached
59
MySQL Example
/etc/default/gearman-job-server
-q mysql
--mysql-host=localhost
--mysql-user=xx
--mysql-password=xx
--mys...
60
Client Library
pecl install channel://pecl.php.net/gearman-1.1.2
echo 'extension=gearman.so' > /path/gearman.ini
61
Quick Test
php -i | grep 'gearman'
62
Gearman in PHP Client
63
PHP Client
# Create a client object
$gm = new GearmanClient();
# Add default server, default is localhost:4730
$gm->add...
64
Do background Job
$job = $gm->doBackground("reverse", "apple");
65
Gearman in PHP Worker
66
PHP Worker
# Define worker job function
function reverse_fn($job) { //.... }
# fetch message from client
function rever...
67
PHP Worker
# Create our worker object.
$gm= new GearmanWorker();
# Add default server (localhost).
$gm->addServer();
# ...
68
Multiple Jobs
$gm->addFunction("reverse", "reverse_fn");
$gm->addFunction("resize", "resize_fn");
$gm->addFunction("que...
69
Anonymous function
$gm->addFunction("reverse", function($job){
$data = $job->workload();
});
70
Gearman in CodeIgniter
71
CodeIgniter Gearman Library
https://github.com/appleboy/CodeIgniter-Gearman-Library
72
Installation via Spark
php tools/spark install -v1.0.1 codeigniter-gearman
73
Setup Gearmand Server and Port
config/gearman.php
$config['gearman_server'] = array('localhost');
$config['gearman_port...
74
CodeIgniter Client
75
Implement Image Resize Job
create app controller
76
class App extends CI_Controller
{
public function resize_sync()
public function resize_async()
}
77
public function resize_sync()
{
$data = json_encode(array('file_name' => 'test.png'));
$client = $this->lib_gearman->ge...
78
class App extends CI_Controller
{
public function resize_sync()
public function resize_async()
}
79
public function resize_async()
{
$data = json_encode(array('file_name' => 'test.png'));
$client = $this->lib_gearman->g...
80
CodeIgniter Worker
create cli controller
81
class Cli extends CI_Controller
{
public static function image_resize()
public static function email_queue()
public fun...
82
public static function image_resize($job)
{
$data = json_decode($job->workload(), true);
$ci =& get_instance();
$ci->im...
83
Define Worker
84
public function worker()
{
$worker = $this->lib_gearman->gearman_worker();
$this->lib_gearman
->add_worker_function('re...
85
Start Worker
$ php index.php cli worker
86
Running workers via script
#!/bin/sh
script="index.php cli worker"
folder="/script_path/"
max_worker="5"
php_bin="/usr/...
87
About Job Data
Don't pass 2GB object.
Gearman is not data center.
88
Keep your message data small
Amazone SQS message siez: 256K
http://goo.gl/0MBq8W
89
Scalability
90
Client Client
Job Server
Client
ClientWorker Worker Worker Worker
Job Server
Client
91
you can scale out your
clients and workers
as needed
92
Client Setting
Client1, Client2: Gearman1
Client3, Client4: Gearman2
…...
93
Worker Setting
Worker[1-4]:Server[1-2]
…...
94
Add new Job Server...
95
Reconfigure all setting
(client and worker program)
96
Reconfigure all setting
(client and worker program)
even
Deploy Tool
97
如何控制易怒的情緒?
98
How to improve your
architecture?
99
Client Client
Job Server
Client
ClientWorker Worker Worker Worker
Job Server
Client
Proxy (Haproxy ...)
100
Live Demo
Email Queue.
101
Any Questions?
102
Thanks
全體工作人員和聽眾
Upcoming SlideShare
Loading in...5
×

Gearman work queue in php

6,131

Published on

Gearman work queue in php
用 Gearman 打造 PHP 排程

Published in: Technology

Transcript of "Gearman work queue in php"

  1. 1. Gearman work queue in php 用 Gearman 打造 PHP 排程 2013.10.05 @appleboy
  2. 2. 2 Who am I Bo-Yi Wu @appleboy http://blog.wu-boy.com https://github.com/appleboy 任職於瑞昱半導體 RealTek(IC Design House)
  3. 3. 3 今日來聊聊排程系統
  4. 4. 4 排程使用時機 ?
  5. 5. 5 不需要即時回覆 都可以使用排程
  6. 6. 6 哪些應用需要排程 ?
  7. 7. 7 使用者註冊系統
  8. 8. 8 使用者註冊 填寫表單完成 寄發認證信 使用者收到回覆 未使用排程 處理資料 透過第三方寄信 Amazone SES
  9. 9. 9 缺點 Response time 過長 使用者不需要等伺服器寄信 ( 來自知名插畫家彎彎 )
  10. 10. 10 使用者註冊 填寫表單完成 寄發認證信 使用者收到回覆 使用排程 處理資料 寫入排程
  11. 11. 11 Email Queue? Github Social Coding
  12. 12. 12 Application Server Database Database Database Work Queue
  13. 13. 13 執行排程 Linux crontab Ref: http://www.xenstreet.com/2013/03/choosing-the-right-linux/
  14. 14. 14 <?php $link = mysqli_connect("host", "user", "passw", "db"); $query = 'SELECT * FROM jobs where status = "on"'; $result = $link->query($query); while($row = mysqli_fecth_array($result)) { // ….... // do work // ….... $sql = 'update jobs set status = "off" where id = {$row["id"]}'; $link->query($sql); }
  15. 15. 15 每分鐘執行一次 */1 * * * * root cron.sh
  16. 16. 16 優點 ● 容易瞭解 ● 任何程式語言都可以實現 ● 快速 Deploy 到任何機器
  17. 17. 17 缺點 ● 每分鐘只能執行一次 ● Race condition ● 浪費 Database 資源
  18. 18. 18 換個寫法來試試看
  19. 19. 19 每 10 秒執行一次 <?php $link = mysqli_connect("myhost","myuser","mypassw","mybd"); $query = 'SELECT * FROM jobs where status = "on"'; $done = false; while(!$done) { $result = $link->query($query); if ($result->num_rows > 0) { // do work $sql = 'update jobs set status = "off" where id = {$row["id"]}'; $link->query($sql); } else { sleep(10); } }
  20. 20. 20 缺點 ● 真的 10 秒執行一次 ? ● Race condition ● 更浪費 Database 資源
  21. 21. 21 如果沒有 Race Condition
  22. 22. 22 #!/bin/sh (sleep 5 && /path/script.sh) & (sleep 15 && /path/script.sh) & (sleep 25 && /path/script.sh) & (sleep 35 && /path/script.sh) & (sleep 45 && /path/script.sh) & (sleep 55 && /path/script.sh) & 每 10 秒執行一次
  23. 23. 23 如何分配處理這些問題 ?
  24. 24. 24 Using Gearman
  25. 25. 25 What is Gearman? It allows you to do work in parallel, to load balance processing, and to call functions between languages.
  26. 26. 26 Client in Perl,Python,PHP Java,C#(.Net)
  27. 27. 27 even MySQL,PostgreSQL
  28. 28. 28 How Does Gearman Work?
  29. 29. 29 Gearman 優點
  30. 30. 30 Work Queues ● 淺顯易懂 Got job do job repeat→ → Application Gearmand Worker Got job Do job
  31. 31. 31 Work Queues ● 淺顯易懂 Get job do job repeat→ → ● 跨語言 任何平台只要能存入字串即可
  32. 32. 32 Multiple Language Application Gearmand Application Worker Worker
  33. 33. 33 Work Queues ● 淺顯易懂 Get job do job repeat→ → ● 跨語言 任何平台只要能存入字串即可 ● 跨平台 當 Client 寫入工作時, Worker 不需要啟動
  34. 34. 34 跨平台 Application Gearmand Application Gearmand
  35. 35. 35 Work Queues ● 淺顯易懂 Get job do job repeat→ → ● 跨語言 任何平台只要能存入字串即可 ● 跨平台 當 Client 寫入工作時, Worker 不需要啟動 ● 即時性 可以為同步或非同步
  36. 36. 36 Gearman Architecture 佈署架構
  37. 37. 37 Application Gearmand Job
  38. 38. 38 Application Gearmand Job Worker Job
  39. 39. 39 Application Gearmand Job Worker Job Response
  40. 40. 40 Application Gearmand Job Worker Job Response Response
  41. 41. 41 Application Gearmand Worker Worker Worker
  42. 42. 42 Application Gearmand Worker Worker Gearmand
  43. 43. 43 Application Gearmand Worker Worker Gearmand Application
  44. 44. 44 結論 :Gearman 擴充性佳
  45. 45. 45 不論要幾台 gearmand 跟數個 worker 都可以
  46. 46. 46 Application 只會跟 Gearmand 溝通
  47. 47. 47 多個 Application 不管 丟出多少個 Jobs
  48. 48. 48 最終由多個 Job Server 分配工作
  49. 49. 49 Synchronous vs Asynchronous
  50. 50. 50 Synchronous Application Gearmand Job Worker Job Response Response
  51. 51. 51 Asynchronous Application Gearmand Job Worker Job Done!
  52. 52. 52 Installation 安裝方式
  53. 53. 53 Installation ● Job Server (C) ● Client library (PHP,Python ...) ● Worker library (PHP,Python ...)
  54. 54. 54 Job Server 安裝方式
  55. 55. 55 Ubuntu/Debian aptitude -y install gearman gearman-job-server libgearman-dev libdrizzle0
  56. 56. 56 CentOS/Fedora yum -y install libgearman gearmand libgearman-devel libdrizzle
  57. 57. 57 Tarball install https://gist.github.com/appleboy/6774735
  58. 58. 58 Gearmand enabled using MySQL(Mariadb) Postgresql SQL Lite Memcached
  59. 59. 59 MySQL Example /etc/default/gearman-job-server -q mysql --mysql-host=localhost --mysql-user=xx --mysql-password=xx --mysql-db=ovoq --mysql-table=gearman_queue
  60. 60. 60 Client Library pecl install channel://pecl.php.net/gearman-1.1.2 echo 'extension=gearman.so' > /path/gearman.ini
  61. 61. 61 Quick Test php -i | grep 'gearman'
  62. 62. 62 Gearman in PHP Client
  63. 63. 63 PHP Client # Create a client object $gm = new GearmanClient(); # Add default server, default is localhost:4730 $gm->addServer(); # Do Job $result = $gm->doNormal("reverse", "Hello!"); echo "Success: $resultn";
  64. 64. 64 Do background Job $job = $gm->doBackground("reverse", "apple");
  65. 65. 65 Gearman in PHP Worker
  66. 66. 66 PHP Worker # Define worker job function function reverse_fn($job) { //.... } # fetch message from client function reverse_fn($job) { $data = $job->workload(); }
  67. 67. 67 PHP Worker # Create our worker object. $gm= new GearmanWorker(); # Add default server (localhost). $gm->addServer(); # Register function "reverse" $gm->addFunction("reverse", "reverse_fn"); # Waiting for job... while($gm->work());
  68. 68. 68 Multiple Jobs $gm->addFunction("reverse", "reverse_fn"); $gm->addFunction("resize", "resize_fn"); $gm->addFunction("queue", "queue_fn"); …...
  69. 69. 69 Anonymous function $gm->addFunction("reverse", function($job){ $data = $job->workload(); });
  70. 70. 70 Gearman in CodeIgniter
  71. 71. 71 CodeIgniter Gearman Library https://github.com/appleboy/CodeIgniter-Gearman-Library
  72. 72. 72 Installation via Spark php tools/spark install -v1.0.1 codeigniter-gearman
  73. 73. 73 Setup Gearmand Server and Port config/gearman.php $config['gearman_server'] = array('localhost'); $config['gearman_port'] = array('4730');
  74. 74. 74 CodeIgniter Client
  75. 75. 75 Implement Image Resize Job create app controller
  76. 76. 76 class App extends CI_Controller { public function resize_sync() public function resize_async() }
  77. 77. 77 public function resize_sync() { $data = json_encode(array('file_name' => 'test.png')); $client = $this->lib_gearman->gearman_client(); do { $this->lib_gearman->do_job_foreground('resize', $data); switch($client->returnCode()) { //.... } } while ($client->returnCode() != GREARMAN_SUCCESS); }
  78. 78. 78 class App extends CI_Controller { public function resize_sync() public function resize_async() }
  79. 79. 79 public function resize_async() { $data = json_encode(array('file_name' => 'test.png')); $client = $this->lib_gearman->gearman_client(); $this->lib_gearman->do_job_background('resize', $data)); }
  80. 80. 80 CodeIgniter Worker create cli controller
  81. 81. 81 class Cli extends CI_Controller { public static function image_resize() public static function email_queue() public function worker() }
  82. 82. 82 public static function image_resize($job) { $data = json_decode($job->workload(), true); $ci =& get_instance(); $ci->image_lib->initialize($config); $ci->image_lib->resize(); $ci->image_lib->clear(); }
  83. 83. 83 Define Worker
  84. 84. 84 public function worker() { $worker = $this->lib_gearman->gearman_worker(); $this->lib_gearman ->add_worker_function('resize', 'Cli::image_resize'); $this->lib_gearman ->add_worker_function('queue', 'Cli::email_queue'); while ($this->lib_gearman->work()); }
  85. 85. 85 Start Worker $ php index.php cli worker
  86. 86. 86 Running workers via script #!/bin/sh script="index.php cli worker" folder="/script_path/" max_worker="5" php_bin="/usr/bin/php" count=0 current_count=$(ps aux | grep "index.php cli worker" | grep -v "grep" | wc -l) if test $current_count -lt $max_worker then cd $folder $php_bin $folder$script & else echo "The worker count has been limited." fi https://gist.github.com/appleboy/6806323
  87. 87. 87 About Job Data Don't pass 2GB object. Gearman is not data center.
  88. 88. 88 Keep your message data small Amazone SQS message siez: 256K http://goo.gl/0MBq8W
  89. 89. 89 Scalability
  90. 90. 90 Client Client Job Server Client ClientWorker Worker Worker Worker Job Server Client
  91. 91. 91 you can scale out your clients and workers as needed
  92. 92. 92 Client Setting Client1, Client2: Gearman1 Client3, Client4: Gearman2 …...
  93. 93. 93 Worker Setting Worker[1-4]:Server[1-2] …...
  94. 94. 94 Add new Job Server...
  95. 95. 95 Reconfigure all setting (client and worker program)
  96. 96. 96 Reconfigure all setting (client and worker program) even Deploy Tool
  97. 97. 97 如何控制易怒的情緒?
  98. 98. 98 How to improve your architecture?
  99. 99. 99 Client Client Job Server Client ClientWorker Worker Worker Worker Job Server Client Proxy (Haproxy ...)
  100. 100. 100 Live Demo Email Queue.
  101. 101. 101 Any Questions?
  102. 102. 102 Thanks 全體工作人員和聽眾
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×