Your SlideShare is downloading. ×
Gearman work queue in php
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Gearman work queue in php

5,953
views

Published on

Gearman work queue in php …

Gearman work queue in php
用 Gearman 打造 PHP 排程

Published in: Technology

0 Comments
22 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,953
On Slideshare
0
From Embeds
0
Number of Embeds
20
Actions
Shares
0
Downloads
73
Comments
0
Likes
22
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Gearman work queue in php 用 Gearman 打造 PHP 排程 2013.10.05 @appleboy
  • 2. 2 Who am I Bo-Yi Wu @appleboy http://blog.wu-boy.com https://github.com/appleboy 任職於瑞昱半導體 RealTek(IC Design House)
  • 3. 3 今日來聊聊排程系統
  • 4. 4 排程使用時機 ?
  • 5. 5 不需要即時回覆 都可以使用排程
  • 6. 6 哪些應用需要排程 ?
  • 7. 7 使用者註冊系統
  • 8. 8 使用者註冊 填寫表單完成 寄發認證信 使用者收到回覆 未使用排程 處理資料 透過第三方寄信 Amazone SES
  • 9. 9 缺點 Response time 過長 使用者不需要等伺服器寄信 ( 來自知名插畫家彎彎 )
  • 10. 10 使用者註冊 填寫表單完成 寄發認證信 使用者收到回覆 使用排程 處理資料 寫入排程
  • 11. 11 Email Queue? Github Social Coding
  • 12. 12 Application Server Database Database Database Work Queue
  • 13. 13 執行排程 Linux crontab Ref: http://www.xenstreet.com/2013/03/choosing-the-right-linux/
  • 14. 14 <?php $link = mysqli_connect("host", "user", "passw", "db"); $query = 'SELECT * FROM jobs where status = "on"'; $result = $link->query($query); while($row = mysqli_fecth_array($result)) { // ….... // do work // ….... $sql = 'update jobs set status = "off" where id = {$row["id"]}'; $link->query($sql); }
  • 15. 15 每分鐘執行一次 */1 * * * * root cron.sh
  • 16. 16 優點 ● 容易瞭解 ● 任何程式語言都可以實現 ● 快速 Deploy 到任何機器
  • 17. 17 缺點 ● 每分鐘只能執行一次 ● Race condition ● 浪費 Database 資源
  • 18. 18 換個寫法來試試看
  • 19. 19 每 10 秒執行一次 <?php $link = mysqli_connect("myhost","myuser","mypassw","mybd"); $query = 'SELECT * FROM jobs where status = "on"'; $done = false; while(!$done) { $result = $link->query($query); if ($result->num_rows > 0) { // do work $sql = 'update jobs set status = "off" where id = {$row["id"]}'; $link->query($sql); } else { sleep(10); } }
  • 20. 20 缺點 ● 真的 10 秒執行一次 ? ● Race condition ● 更浪費 Database 資源
  • 21. 21 如果沒有 Race Condition
  • 22. 22 #!/bin/sh (sleep 5 && /path/script.sh) & (sleep 15 && /path/script.sh) & (sleep 25 && /path/script.sh) & (sleep 35 && /path/script.sh) & (sleep 45 && /path/script.sh) & (sleep 55 && /path/script.sh) & 每 10 秒執行一次
  • 23. 23 如何分配處理這些問題 ?
  • 24. 24 Using Gearman
  • 25. 25 What is Gearman? It allows you to do work in parallel, to load balance processing, and to call functions between languages.
  • 26. 26 Client in Perl,Python,PHP Java,C#(.Net)
  • 27. 27 even MySQL,PostgreSQL
  • 28. 28 How Does Gearman Work?
  • 29. 29 Gearman 優點
  • 30. 30 Work Queues ● 淺顯易懂 Got job do job repeat→ → Application Gearmand Worker Got job Do job
  • 31. 31 Work Queues ● 淺顯易懂 Get job do job repeat→ → ● 跨語言 任何平台只要能存入字串即可
  • 32. 32 Multiple Language Application Gearmand Application Worker Worker
  • 33. 33 Work Queues ● 淺顯易懂 Get job do job repeat→ → ● 跨語言 任何平台只要能存入字串即可 ● 跨平台 當 Client 寫入工作時, Worker 不需要啟動
  • 34. 34 跨平台 Application Gearmand Application Gearmand
  • 35. 35 Work Queues ● 淺顯易懂 Get job do job repeat→ → ● 跨語言 任何平台只要能存入字串即可 ● 跨平台 當 Client 寫入工作時, Worker 不需要啟動 ● 即時性 可以為同步或非同步
  • 36. 36 Gearman Architecture 佈署架構
  • 37. 37 Application Gearmand Job
  • 38. 38 Application Gearmand Job Worker Job
  • 39. 39 Application Gearmand Job Worker Job Response
  • 40. 40 Application Gearmand Job Worker Job Response Response
  • 41. 41 Application Gearmand Worker Worker Worker
  • 42. 42 Application Gearmand Worker Worker Gearmand
  • 43. 43 Application Gearmand Worker Worker Gearmand Application
  • 44. 44 結論 :Gearman 擴充性佳
  • 45. 45 不論要幾台 gearmand 跟數個 worker 都可以
  • 46. 46 Application 只會跟 Gearmand 溝通
  • 47. 47 多個 Application 不管 丟出多少個 Jobs
  • 48. 48 最終由多個 Job Server 分配工作
  • 49. 49 Synchronous vs Asynchronous
  • 50. 50 Synchronous Application Gearmand Job Worker Job Response Response
  • 51. 51 Asynchronous Application Gearmand Job Worker Job Done!
  • 52. 52 Installation 安裝方式
  • 53. 53 Installation ● Job Server (C) ● Client library (PHP,Python ...) ● Worker library (PHP,Python ...)
  • 54. 54 Job Server 安裝方式
  • 55. 55 Ubuntu/Debian aptitude -y install gearman gearman-job-server libgearman-dev libdrizzle0
  • 56. 56 CentOS/Fedora yum -y install libgearman gearmand libgearman-devel libdrizzle
  • 57. 57 Tarball install https://gist.github.com/appleboy/6774735
  • 58. 58 Gearmand enabled using MySQL(Mariadb) Postgresql SQL Lite Memcached
  • 59. 59 MySQL Example /etc/default/gearman-job-server -q mysql --mysql-host=localhost --mysql-user=xx --mysql-password=xx --mysql-db=ovoq --mysql-table=gearman_queue
  • 60. 60 Client Library pecl install channel://pecl.php.net/gearman-1.1.2 echo 'extension=gearman.so' > /path/gearman.ini
  • 61. 61 Quick Test php -i | grep 'gearman'
  • 62. 62 Gearman in PHP Client
  • 63. 63 PHP Client # Create a client object $gm = new GearmanClient(); # Add default server, default is localhost:4730 $gm->addServer(); # Do Job $result = $gm->doNormal("reverse", "Hello!"); echo "Success: $resultn";
  • 64. 64 Do background Job $job = $gm->doBackground("reverse", "apple");
  • 65. 65 Gearman in PHP Worker
  • 66. 66 PHP Worker # Define worker job function function reverse_fn($job) { //.... } # fetch message from client function reverse_fn($job) { $data = $job->workload(); }
  • 67. 67 PHP Worker # Create our worker object. $gm= new GearmanWorker(); # Add default server (localhost). $gm->addServer(); # Register function "reverse" $gm->addFunction("reverse", "reverse_fn"); # Waiting for job... while($gm->work());
  • 68. 68 Multiple Jobs $gm->addFunction("reverse", "reverse_fn"); $gm->addFunction("resize", "resize_fn"); $gm->addFunction("queue", "queue_fn"); …...
  • 69. 69 Anonymous function $gm->addFunction("reverse", function($job){ $data = $job->workload(); });
  • 70. 70 Gearman in CodeIgniter
  • 71. 71 CodeIgniter Gearman Library https://github.com/appleboy/CodeIgniter-Gearman-Library
  • 72. 72 Installation via Spark php tools/spark install -v1.0.1 codeigniter-gearman
  • 73. 73 Setup Gearmand Server and Port config/gearman.php $config['gearman_server'] = array('localhost'); $config['gearman_port'] = array('4730');
  • 74. 74 CodeIgniter Client
  • 75. 75 Implement Image Resize Job create app controller
  • 76. 76 class App extends CI_Controller { public function resize_sync() public function resize_async() }
  • 77. 77 public function resize_sync() { $data = json_encode(array('file_name' => 'test.png')); $client = $this->lib_gearman->gearman_client(); do { $this->lib_gearman->do_job_foreground('resize', $data); switch($client->returnCode()) { //.... } } while ($client->returnCode() != GREARMAN_SUCCESS); }
  • 78. 78 class App extends CI_Controller { public function resize_sync() public function resize_async() }
  • 79. 79 public function resize_async() { $data = json_encode(array('file_name' => 'test.png')); $client = $this->lib_gearman->gearman_client(); $this->lib_gearman->do_job_background('resize', $data)); }
  • 80. 80 CodeIgniter Worker create cli controller
  • 81. 81 class Cli extends CI_Controller { public static function image_resize() public static function email_queue() public function worker() }
  • 82. 82 public static function image_resize($job) { $data = json_decode($job->workload(), true); $ci =& get_instance(); $ci->image_lib->initialize($config); $ci->image_lib->resize(); $ci->image_lib->clear(); }
  • 83. 83 Define Worker
  • 84. 84 public function worker() { $worker = $this->lib_gearman->gearman_worker(); $this->lib_gearman ->add_worker_function('resize', 'Cli::image_resize'); $this->lib_gearman ->add_worker_function('queue', 'Cli::email_queue'); while ($this->lib_gearman->work()); }
  • 85. 85 Start Worker $ php index.php cli worker
  • 86. 86 Running workers via script #!/bin/sh script="index.php cli worker" folder="/script_path/" max_worker="5" php_bin="/usr/bin/php" count=0 current_count=$(ps aux | grep "index.php cli worker" | grep -v "grep" | wc -l) if test $current_count -lt $max_worker then cd $folder $php_bin $folder$script & else echo "The worker count has been limited." fi https://gist.github.com/appleboy/6806323
  • 87. 87 About Job Data Don't pass 2GB object. Gearman is not data center.
  • 88. 88 Keep your message data small Amazone SQS message siez: 256K http://goo.gl/0MBq8W
  • 89. 89 Scalability
  • 90. 90 Client Client Job Server Client ClientWorker Worker Worker Worker Job Server Client
  • 91. 91 you can scale out your clients and workers as needed
  • 92. 92 Client Setting Client1, Client2: Gearman1 Client3, Client4: Gearman2 …...
  • 93. 93 Worker Setting Worker[1-4]:Server[1-2] …...
  • 94. 94 Add new Job Server...
  • 95. 95 Reconfigure all setting (client and worker program)
  • 96. 96 Reconfigure all setting (client and worker program) even Deploy Tool
  • 97. 97 如何控制易怒的情緒?
  • 98. 98 How to improve your architecture?
  • 99. 99 Client Client Job Server Client ClientWorker Worker Worker Worker Job Server Client Proxy (Haproxy ...)
  • 100. 100 Live Demo Email Queue.
  • 101. 101 Any Questions?
  • 102. 102 Thanks 全體工作人員和聽眾

×