Khanh-Nguyen - Gearman - distributed process solution

9,377 views
9,313 views

Published on

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. In other words, it is the nervous system for how distributed processing communicates.

Published in: Technology
1 Comment
11 Likes
Statistics
Notes
No Downloads
Views
Total views
9,377
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
118
Comments
1
Likes
11
Embeds 0
No embeds

No notes for slide

Khanh-Nguyen - Gearman - distributed process solution

  1. 1. Gearman Distributed Process Solution Created by Google Drive Presentation.
  2. 2. Introduction $ whoami ######################################################## #### Name : Nguyen Duy Khanh #### #### Job : Student of HCMUS, Part-timer at PNC #### #### My skills : PHP, MySQL, JS #### #### Love : New technology #### ######################################################## $|
  3. 3. Problems HEAVY WORKLOAD function main() { ... doSomeThing(); doSomeThingBig(); doSomeThingBigBig(); doSomeThingVeryBig(); … return result; }
  4. 4. Problems console.log(result);
  5. 5. Solution Give me a job, my workers will do it for you. Worker 1 Worker 2 Worker... Worker n
  6. 6. Gearman ❏ GEARMAN → MANAGER ❏ Open Source. ❏ Multi-language API. ❏ Multi-threaded ( 50k jobs / second ). ❏ Ultra fast ( written in C/C++ ). ❏ No limits on message size. ❏ Failover
  7. 7. Installation 1. 2. Download source code [ https://launchpad.net/gearmand ]. Compile and install from tarball. tar xzf gearmand-X.Y.tar.gz cd gearmand-X.Y ./configure make make install 3. Starting gearman. gearman -d
  8. 8. Gearman Architecture Customer Job Manager Worker
  9. 9. How does it work ? Client Hey, I can send email. Let me do it !!! Worker Python Worker Node.JS Job Manager Worker PHP
  10. 10. How does it work ? Client Worker Python (send_email) Worker Node.JS Job Manager Worker PHP
  11. 11. How does it work ? Client And I can push notification. Let me try !!! Worker Python Worker Node.JS Job Manager Worker PHP
  12. 12. How does it work ? Client Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP
  13. 13. How does it work ? Client Haha, I can do them all !!! Worker Python Worker Node.JS Job Manager Worker PHP
  14. 14. How does it work ? Client Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  15. 15. How does it work ? Client I need send an email to abc@xyz.com , please help me ! Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  16. 16. How does it work ? Client OK, Let me see. Aaah … We have 2 workers can do it for you. But, Worker Python is busy, let Worker PHP do it. Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  17. 17. How does it work ? Client Hey Worker PHP, please send an email to abc@xyz.com Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  18. 18. How does it work ? Client OK Boss, email has sent to abc@xyz.com Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  19. 19. How does it work ? Client Hi client, your email has sent to abc@xyz.com. Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  20. 20. Hello world !!! Install Node.js API $ npm install node-gearman Install PHP API $ pecl install gearman Install Python API $ pip install gearman or $ easy_install gearman
  21. 21. Hello world !!! worker1.js
  22. 22. Hello world !!! worker2.py
  23. 23. Hello world !!! client.php
  24. 24. Hello world !!! ● Python and C/C++ and a good choices for workers. ● PHP, Node.js should be client than worker. [ PHP Worker often crash after 1-2 hours because memory leak ]
  25. 25. Use-cases ❏ Scatter/Gather ❏ Map - Reduce ❏ Asynchronous Queues ❏ Pipeline Processing
  26. 26. Scatter / Gather ● Perform a number of task concurrently. ○ Ex : Db Query, Image Processing, Location Search, Full-text Search… ● Take advantage of many tier. ● Speed up your web applications. ● Tasks don’t need to be related.
  27. 27. Scatter / Gather Db Query Client Resize Image Search
  28. 28. Map / Reduce
  29. 29. Asynchronous Queues ● Most popular use-case. ● Do something needn’t immediate processing. ○ Ex : send emails, push notification, index data, crawling, ... ● Allows for batch operations.
  30. 30. Asynchronous Queues function tweet(msg) { // Insert to database db.insert(msg); // Background tasks gearman.push_notify(); // Return return true; }
  31. 31. Pipeline Processing ● Do related thing. ○ Ex : search engine, data analytic, ... ● Chain workers to send data to next step.
  32. 32. Pipeline Processing Main Client Worker Step 1 Client Worker Step 2 Client Worker Last Step Output
  33. 33. Examples GEARMAN Auto News Crawling System Reload Request MongoDB ● Manager : GEARMAN ● Workers “parse_rss” Gearman Client Webserver : Node.JS No Workers : Python ● RSS Sources ● Web Framework : Express ● Database : MongoDB ● Client Script : jQuery Have new items ? Crawl new items Workers “crawling” End
  34. 34. Persistent Queues ❏ By default, jobs are only stored in memory. ❏ Various contributions from community : ❏ MySQL / Drizzle ❏ PostgreSQL ❏ SQLite ❏ Tokyo Cabinet ❏ memcached ❏ Hope MongoDB (or an NOSQL DB) coming soon.
  35. 35. Gearman Admin GUI ● Command Line Tool : http://gearman.info/bin/gearadmin.html ● Gearman-Monitor [ PHP ] : https://github.com/yugene/Gearman-Monitor ● GearmanUI [ PHP ] : http://rripado.info/gearmanui/
  36. 36. THE N D THANK YOU !!!
  37. 37. About Us Author : Khanh Nguyen Duy Find me at : admin@ndksolution.com Presentation made for “Javascript HoChiMinh City Meetup” You can find us at : ● http://meetup.com/JavaScript-Ho-Chi-Minh-City/ ● https://www.facebook.com/JavaScriptHCMC ● https://plus.google.com/u/0/communities/116105314977285194967

×