0
Gearman
Distributed Process Solution
Created by Google Drive Presentation.
Introduction
$ whoami
########################################################
####

Name

: Nguyen Duy Khanh

####

####
...
Problems
HEAVY WORKLOAD
function main() {
...
doSomeThing();
doSomeThingBig();
doSomeThingBigBig();
doSomeThingVeryBig();
...
Problems

console.log(result);
Solution
Give me a job, my
workers will do it
for you.

Worker 1
Worker 2
Worker...
Worker n
Gearman
❏ GEARMAN → MANAGER
❏ Open Source.
❏ Multi-language API.
❏ Multi-threaded ( 50k jobs / second ).
❏ Ultra fast ( wr...
Installation
1.
2.

Download source code [ https://launchpad.net/gearmand ].
Compile and install from tarball.
tar xzf gea...
Gearman Architecture
Customer

Job Manager

Worker
How does it work ?
Client

Hey, I can send email.
Let me do it !!!

Worker Python

Worker Node.JS
Job Manager
Worker PHP
How does it work ?
Client

Worker Python
(send_email)

Worker Node.JS
Job Manager
Worker PHP
How does it work ?
Client

And I can push
notification. Let me try !!!

Worker Python

Worker Node.JS
Job Manager
Worker P...
How does it work ?
Client

Worker Python
(send_email)

Worker Node.JS
(push_notify)
Job Manager
Worker PHP
How does it work ?
Client

Haha, I can do them all !!!

Worker Python

Worker Node.JS
Job Manager
Worker PHP
How does it work ?
Client

Worker Python
(send_email)

Worker Node.JS
(push_notify)
Job Manager
Worker PHP
(send_email, pu...
How does it work ?
Client

I need send an email to
abc@xyz.com , please
help me !

Worker Python
(send_email)

Worker Node...
How does it work ?
Client

OK, Let me see. Aaah …
We have 2 workers can do
it for you. But, Worker
Python is busy, let Wor...
How does it work ?
Client

Hey Worker PHP, please
send an email to
abc@xyz.com

Worker Python
(send_email)

Worker Node.JS...
How does it work ?
Client
OK Boss, email has sent
to abc@xyz.com

Worker Python
(send_email)

Worker Node.JS
(push_notify)...
How does it work ?
Client
Hi client, your email has
sent to abc@xyz.com.

Worker Python
(send_email)

Worker Node.JS
(push...
Hello world !!!
Install Node.js API
$ npm install node-gearman
Install PHP API
$ pecl install gearman
Install Python API
$...
Hello world !!!
worker1.js
Hello world !!!
worker2.py
Hello world !!!
client.php
Hello world !!!
●

Python and C/C++ and a good choices for workers.

●

PHP, Node.js should be client than worker. [ PHP W...
Use-cases
❏ Scatter/Gather
❏ Map - Reduce
❏ Asynchronous Queues
❏ Pipeline Processing
Scatter / Gather
● Perform a number of task concurrently.
○ Ex : Db Query, Image Processing, Location Search,
Full-text Se...
Scatter / Gather
Db Query
Client

Resize Image
Search
Map / Reduce
Asynchronous Queues
● Most popular use-case.
● Do something needn’t immediate processing.
○ Ex : send emails, push notific...
Asynchronous Queues
function tweet(msg) {
// Insert to database
db.insert(msg);
// Background tasks
gearman.push_notify();...
Pipeline Processing
● Do related thing.
○ Ex : search engine, data analytic, ...
● Chain workers to send data to next step...
Pipeline Processing

Main
Client

Worker
Step 1

Client

Worker
Step 2

Client

Worker
Last Step

Output
Examples
GEARMAN Auto News Crawling System
Reload
Request

MongoDB

●

Manager : GEARMAN

●

Workers
“parse_rss”

Gearman ...
Persistent Queues
❏

By default, jobs are only stored in memory.

❏

Various contributions from community :
❏

MySQL / Dri...
Gearman Admin GUI
●

Command Line Tool : http://gearman.info/bin/gearadmin.html

●

Gearman-Monitor [ PHP ] : https://gith...
THE
N
D

THANK YOU !!!
About Us
Author : Khanh Nguyen Duy
Find me at : admin@ndksolution.com
Presentation made for “Javascript HoChiMinh City Mee...
Khanh-Nguyen - Gearman - distributed process solution
Khanh-Nguyen - Gearman - distributed process solution
Upcoming SlideShare
Loading in...5
×

Khanh-Nguyen - Gearman - distributed process solution

8,489

Published on

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. In other words, it is the nervous system for how distributed processing communicates.

Published in: Technology
1 Comment
8 Likes
Statistics
Notes
No Downloads
Views
Total Views
8,489
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
36
Comments
1
Likes
8
Embeds 0
No embeds

No notes for slide

Transcript of "Khanh-Nguyen - Gearman - distributed process solution"

  1. 1. Gearman Distributed Process Solution Created by Google Drive Presentation.
  2. 2. Introduction $ whoami ######################################################## #### Name : Nguyen Duy Khanh #### #### Job : Student of HCMUS, Part-timer at PNC #### #### My skills : PHP, MySQL, JS #### #### Love : New technology #### ######################################################## $|
  3. 3. Problems HEAVY WORKLOAD function main() { ... doSomeThing(); doSomeThingBig(); doSomeThingBigBig(); doSomeThingVeryBig(); … return result; }
  4. 4. Problems console.log(result);
  5. 5. Solution Give me a job, my workers will do it for you. Worker 1 Worker 2 Worker... Worker n
  6. 6. Gearman ❏ GEARMAN → MANAGER ❏ Open Source. ❏ Multi-language API. ❏ Multi-threaded ( 50k jobs / second ). ❏ Ultra fast ( written in C/C++ ). ❏ No limits on message size. ❏ Failover
  7. 7. Installation 1. 2. Download source code [ https://launchpad.net/gearmand ]. Compile and install from tarball. tar xzf gearmand-X.Y.tar.gz cd gearmand-X.Y ./configure make make install 3. Starting gearman. gearman -d
  8. 8. Gearman Architecture Customer Job Manager Worker
  9. 9. How does it work ? Client Hey, I can send email. Let me do it !!! Worker Python Worker Node.JS Job Manager Worker PHP
  10. 10. How does it work ? Client Worker Python (send_email) Worker Node.JS Job Manager Worker PHP
  11. 11. How does it work ? Client And I can push notification. Let me try !!! Worker Python Worker Node.JS Job Manager Worker PHP
  12. 12. How does it work ? Client Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP
  13. 13. How does it work ? Client Haha, I can do them all !!! Worker Python Worker Node.JS Job Manager Worker PHP
  14. 14. How does it work ? Client Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  15. 15. How does it work ? Client I need send an email to abc@xyz.com , please help me ! Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  16. 16. How does it work ? Client OK, Let me see. Aaah … We have 2 workers can do it for you. But, Worker Python is busy, let Worker PHP do it. Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  17. 17. How does it work ? Client Hey Worker PHP, please send an email to abc@xyz.com Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  18. 18. How does it work ? Client OK Boss, email has sent to abc@xyz.com Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  19. 19. How does it work ? Client Hi client, your email has sent to abc@xyz.com. Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  20. 20. Hello world !!! Install Node.js API $ npm install node-gearman Install PHP API $ pecl install gearman Install Python API $ pip install gearman or $ easy_install gearman
  21. 21. Hello world !!! worker1.js
  22. 22. Hello world !!! worker2.py
  23. 23. Hello world !!! client.php
  24. 24. Hello world !!! ● Python and C/C++ and a good choices for workers. ● PHP, Node.js should be client than worker. [ PHP Worker often crash after 1-2 hours because memory leak ]
  25. 25. Use-cases ❏ Scatter/Gather ❏ Map - Reduce ❏ Asynchronous Queues ❏ Pipeline Processing
  26. 26. Scatter / Gather ● Perform a number of task concurrently. ○ Ex : Db Query, Image Processing, Location Search, Full-text Search… ● Take advantage of many tier. ● Speed up your web applications. ● Tasks don’t need to be related.
  27. 27. Scatter / Gather Db Query Client Resize Image Search
  28. 28. Map / Reduce
  29. 29. Asynchronous Queues ● Most popular use-case. ● Do something needn’t immediate processing. ○ Ex : send emails, push notification, index data, crawling, ... ● Allows for batch operations.
  30. 30. Asynchronous Queues function tweet(msg) { // Insert to database db.insert(msg); // Background tasks gearman.push_notify(); // Return return true; }
  31. 31. Pipeline Processing ● Do related thing. ○ Ex : search engine, data analytic, ... ● Chain workers to send data to next step.
  32. 32. Pipeline Processing Main Client Worker Step 1 Client Worker Step 2 Client Worker Last Step Output
  33. 33. Examples GEARMAN Auto News Crawling System Reload Request MongoDB ● Manager : GEARMAN ● Workers “parse_rss” Gearman Client Webserver : Node.JS No Workers : Python ● RSS Sources ● Web Framework : Express ● Database : MongoDB ● Client Script : jQuery Have new items ? Crawl new items Workers “crawling” End
  34. 34. Persistent Queues ❏ By default, jobs are only stored in memory. ❏ Various contributions from community : ❏ MySQL / Drizzle ❏ PostgreSQL ❏ SQLite ❏ Tokyo Cabinet ❏ memcached ❏ Hope MongoDB (or an NOSQL DB) coming soon.
  35. 35. Gearman Admin GUI ● Command Line Tool : http://gearman.info/bin/gearadmin.html ● Gearman-Monitor [ PHP ] : https://github.com/yugene/Gearman-Monitor ● GearmanUI [ PHP ] : http://rripado.info/gearmanui/
  36. 36. THE N D THANK YOU !!!
  37. 37. About Us Author : Khanh Nguyen Duy Find me at : admin@ndksolution.com Presentation made for “Javascript HoChiMinh City Meetup” You can find us at : ● http://meetup.com/JavaScript-Ho-Chi-Minh-City/ ● https://www.facebook.com/JavaScriptHCMC ● https://plus.google.com/u/0/communities/116105314977285194967
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×