Khanh-Nguyen - Gearman - distributed process solution
Upcoming SlideShare
Loading in...5
×
 

Khanh-Nguyen - Gearman - distributed process solution

on

  • 7,403 views

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. In other words, it is the nervous system for how distributed ...

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. In other words, it is the nervous system for how distributed processing communicates.

Statistics

Views

Total Views
7,403
Views on SlideShare
7,227
Embed Views
176

Actions

Likes
8
Downloads
23
Comments
1

2 Embeds 176

http://blog.ndksolution.com 156
http://localhost 20

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Khanh-Nguyen - Gearman - distributed process solution Khanh-Nguyen - Gearman - distributed process solution Presentation Transcript

  • Gearman Distributed Process Solution Created by Google Drive Presentation.
  • Introduction $ whoami ######################################################## #### Name : Nguyen Duy Khanh #### #### Job : Student of HCMUS, Part-timer at PNC #### #### My skills : PHP, MySQL, JS #### #### Love : New technology #### ######################################################## $|
  • Problems HEAVY WORKLOAD function main() { ... doSomeThing(); doSomeThingBig(); doSomeThingBigBig(); doSomeThingVeryBig(); … return result; }
  • Problems console.log(result);
  • Solution Give me a job, my workers will do it for you. Worker 1 Worker 2 Worker... Worker n
  • Gearman ❏ GEARMAN → MANAGER ❏ Open Source. ❏ Multi-language API. ❏ Multi-threaded ( 50k jobs / second ). ❏ Ultra fast ( written in C/C++ ). ❏ No limits on message size. ❏ Failover
  • Installation 1. 2. Download source code [ https://launchpad.net/gearmand ]. Compile and install from tarball. tar xzf gearmand-X.Y.tar.gz cd gearmand-X.Y ./configure make make install 3. Starting gearman. gearman -d
  • Gearman Architecture Customer Job Manager Worker
  • How does it work ? Client Hey, I can send email. Let me do it !!! Worker Python Worker Node.JS Job Manager Worker PHP
  • How does it work ? Client Worker Python (send_email) Worker Node.JS Job Manager Worker PHP
  • How does it work ? Client And I can push notification. Let me try !!! Worker Python Worker Node.JS Job Manager Worker PHP
  • How does it work ? Client Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP
  • How does it work ? Client Haha, I can do them all !!! Worker Python Worker Node.JS Job Manager Worker PHP
  • How does it work ? Client Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  • How does it work ? Client I need send an email to abc@xyz.com , please help me ! Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  • How does it work ? Client OK, Let me see. Aaah … We have 2 workers can do it for you. But, Worker Python is busy, let Worker PHP do it. Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  • How does it work ? Client Hey Worker PHP, please send an email to abc@xyz.com Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  • How does it work ? Client OK Boss, email has sent to abc@xyz.com Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  • How does it work ? Client Hi client, your email has sent to abc@xyz.com. Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
  • Hello world !!! Install Node.js API $ npm install node-gearman Install PHP API $ pecl install gearman Install Python API $ pip install gearman or $ easy_install gearman
  • Hello world !!! worker1.js
  • Hello world !!! worker2.py
  • Hello world !!! client.php
  • Hello world !!! ● Python and C/C++ and a good choices for workers. ● PHP, Node.js should be client than worker. [ PHP Worker often crash after 1-2 hours because memory leak ]
  • Use-cases ❏ Scatter/Gather ❏ Map - Reduce ❏ Asynchronous Queues ❏ Pipeline Processing
  • Scatter / Gather ● Perform a number of task concurrently. ○ Ex : Db Query, Image Processing, Location Search, Full-text Search… ● Take advantage of many tier. ● Speed up your web applications. ● Tasks don’t need to be related.
  • Scatter / Gather Db Query Client Resize Image Search
  • Map / Reduce
  • Asynchronous Queues ● Most popular use-case. ● Do something needn’t immediate processing. ○ Ex : send emails, push notification, index data, crawling, ... ● Allows for batch operations.
  • Asynchronous Queues function tweet(msg) { // Insert to database db.insert(msg); // Background tasks gearman.push_notify(); // Return return true; }
  • Pipeline Processing ● Do related thing. ○ Ex : search engine, data analytic, ... ● Chain workers to send data to next step.
  • Pipeline Processing Main Client Worker Step 1 Client Worker Step 2 Client Worker Last Step Output
  • Examples GEARMAN Auto News Crawling System Reload Request MongoDB ● Manager : GEARMAN ● Workers “parse_rss” Gearman Client Webserver : Node.JS No Workers : Python ● RSS Sources ● Web Framework : Express ● Database : MongoDB ● Client Script : jQuery Have new items ? Crawl new items Workers “crawling” End
  • Persistent Queues ❏ By default, jobs are only stored in memory. ❏ Various contributions from community : ❏ MySQL / Drizzle ❏ PostgreSQL ❏ SQLite ❏ Tokyo Cabinet ❏ memcached ❏ Hope MongoDB (or an NOSQL DB) coming soon.
  • Gearman Admin GUI ● Command Line Tool : http://gearman.info/bin/gearadmin.html ● Gearman-Monitor [ PHP ] : https://github.com/yugene/Gearman-Monitor ● GearmanUI [ PHP ] : http://rripado.info/gearmanui/
  • THE N D THANK YOU !!!
  • About Us Author : Khanh Nguyen Duy Find me at : admin@ndksolution.com Presentation made for “Javascript HoChiMinh City Meetup” You can find us at : ● http://meetup.com/JavaScript-Ho-Chi-Minh-City/ ● https://www.facebook.com/JavaScriptHCMC ● https://plus.google.com/u/0/communities/116105314977285194967