Distributed Queue System using Gearman
Upcoming SlideShare
Loading in...5
×
 

Distributed Queue System using Gearman

on

  • 9,916 views

We use Gearman for managing queue system. This covers why we should use a queue in many situations on web-based interface as well as server-side application.

We use Gearman for managing queue system. This covers why we should use a queue in many situations on web-based interface as well as server-side application.

Statistics

Views

Total Views
9,916
Slideshare-icon Views on SlideShare
9,911
Embed Views
5

Actions

Likes
13
Downloads
63
Comments
0

3 Embeds 5

http://swazzy.com 2
https://twitter.com 2
http://www.pinterest.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Distributed Queue System using Gearman Distributed Queue System using Gearman Presentation Transcript

    • Distributed Queue System using Gearman Taehun Cho CTO @ IMcompany http://iamcompany.net/
    • What is a Queue? copyright@ www.mathworks.com
    • Multiple Queue With multiple teller, all customers must form multiple queue to be served
    • Some Situation on the WEB+You want to send an email, push notification,messages, import big file, etc...
    • Problem is that... DO NOT LET YOUR USERS WAIT
    • Run Asynchronously
    • Running Task in a Single Thread? Loading behavior may affect the main GUI
    • Run in Background!The user can do next action, and the task should be done in backend in a distributed manner
    • Job Queue SystemsCelery (http://www.celeryproject.org/)RabbitMQ (http://www.rabbitmq.com/)Zend Server Job Queue (http://www.zend.com/)ZeroMQ (http://www.zeromq.org/)BeanstalkdPeafowl, StarlingApache ActiveMQand many others..
    • Introducing Gearman LiveJournal
    • Introducing GearmanIn LiveJournal, many photos had uploadedevery day and it lead to a heavy load of imageprocessing, and this was a motivation to buildsuch a queue system.● Yahoo!: 120+ servers, 12M jobs/day● Digg: 45+ servers, 400K jobs/day● LiveJournal, SixApart, DealNews, Xing.com, and many others. - Expert PHP and MySQL - Andrew et al, (2010, Wrox)● Grooveshark, GoDaddy.com, IMcompany
    • Features of Gearman● Open Source● Simple & Fast (rewritten in C)● Support a variety of languages : build Worker in Python, Client in PHP● Flexible● Load Balance● Failover
    • Example of Architectures (from http://gearman.org/#what_is_gearman)
    • Architecture Acks the job, finds all sleeping workers Awake, asks for jobs to server Gearman Job ServerConnect, submit a job Sends a noop command to wake them up Client Worker
    • Installation● Compile (for PHP APIs)tar xzvf gearmand-X.Y.tar. ● Pecl Extensiongz sudo pecl install gearmancd gearmand-X.Y./configure ● Add below to php.inimake extension="gearman.so"make install● Start Server$ gearmand -d
    • Use Cases- Crawling a website- Image Manipulation- Push Notification- Sending Email/Messages- File verification/compressing- Fetching RSS Feeds- Indexing on Search Engine
    • Samples - Worker
    • Samples - Client
    • Samples - Monitoring A good tool for monitoring gearman, is available at https://github.com/yugene/Gearman-Monitor
    • Result Worker #1 Worker #2The incomplete job will re-queue to available workers for fault-tolerance
    • Motivation● At the beginning state, we run 3 computers for crawling each schools information. (articles, schedules of the school)● One job at a time, too much time to finish all of them, sometimes machines do the same job as the others do.● That was a motivation to make a job queue system that could do jobs in parallel. And weve found Gearman!
    • Gearman in IMcompanyBut there were some challenges!● How many workers should be up for a server? (How efficiently leverage the load?)● How can we handle unexpected termination of workers?● What if the servers resource is exhausted due to the jobs that given by workers? (Then the server would not respond to others requests/connections related to WEB, SVN, MySQL)
    • Exceptional Case #1
    • Reported bugs when using PHPBug #63041 "Failed to set exception option" onconnect when any gearman server is downhttps://bugs.php.net/bug.php?id=63041Bug #63648 Gearman worker stops withsegfault after 1-2 hour of workinghttps://bugs.php.net/bug.php?id=63648
    • Supervisord for sanity"PHP was not built for long running request""Sometimes it occurs memory leaks"Supervisord helps you in above cases!- Auto restart the processes based on customconfigurations* Installation guide - http://www.masnun.com/2011/11/02/gearman-php-and-supervisor-processing-background-jobs-with-sanity.html
    • Exceptional Case #2 PHP sometimes slows down after hundreds of executions, kill it off if you know this will happen. - Mike Willbanks, "Gearman: A Job Server made for Scale"
    • Server Seems Fine for Now
    • What We Learned● Gearmans queue list is unstable so persistent queueing was highly needed in our system● Integrating MySQL with Gearman was failed in both 1.0.2, 0.34● Tried SQLite, but performance was very poorDo NOT Reserve Too Much Jobs in a Queue
    • Also Weve Tried...● Firing queueing jobs over HTTP request is sometimes not working and may lead to freezing the server eventually● And doesnt support additional functions for the HTTP connection such as authentication● And is not customizableGearman Seems Too Young at This Moment
    • Limitations● Queue makes no guarantees - use MySQL, memcached, Redis, PostgreSQL, etc..● There are few administration tools● Jobs dont expire● If a job is dropped, the client is never be notified-from "http://inside.godaddy.com/cloud-processing-with-gearman/"
    • Join Community!http://gearman.org/http://groups.google.com/group/gearman/
    • Were hiring!● Work in Daejeon, Korea● Flexible, Small Company● Excellent Benefits● We Need Senior HackersFind more information at http://iamcompany.net/Thank you!Any questions?