0
Northeast PHP August 12, 2012A Job Server to Scale       By Mike Willbanks   Sr. Web Architect Manager       NOOK Developer
Housekeeping…    • Talk       Slides will be online later!    • Me       Sr. Web Architect Manager at NOOK Developer    ...
Agenda    • What is Gearman       A general introduction    • Main Concepts       Looking overall at how gearman works f...
What is Gearman?Official StatementWhat it meansVisual understandingPlatforms
Official Statement    “Gearman provides a generic application framework to farm       out work to other machines or proces...
What it Means    • Gearman consists of a daemon, client and worker       At the core, they are simply small programs.    ...
In Pictures7
Platforms    • OS       Linux       Windows (cygwin)    • API implementations available       PHP       Perl       Ja...
Main ConceptsClient -> Daemon -> Worker communicationDistributed Model
Client -> Daemon -> Worker communication10
Distributed Model11
Quick StartInstallationSimple Bash ExamplePHP Related (sorry, I’m all about the PHP)
Installation     • Head to gearman.org     • Click Download     • Click on the LaunchPad download     • Download the Binar...
Gearmand Usage     • gearmand        -d          Run as background daemon        -u [user]   Run as user        -L [hos...
Simple Bash Example     • Starting the Daemon        gearmand –d     • Worker – command line style        nohup gearman ...
Gearman Client Command Line Usage     • gearman        -w          Worker mode        -f [function] Function name to use...
Digging InPersistenceWorkersMonitoring
Persistence     • Gearman by default is an in-memory queue        Leaving this as the default is ideal; however, does not...
Getting Up and Running with Persistence     • Persistent queues require specific configuration during the       compilatio...
Configuration Options20
Clients     • Clients send work to the gearmand server        This is called the workload; it can be anything that can be...
Workers     • Workers are the dudes in the factory doing all the work     • Generally they will run as a daemon in the bac...
Workers – special notes     • Utilizing the Database        If you keep a database connection         • Must have the abi...
Keeping the Daemon Running     • Workers sometimes have issues and die, or you need to boot       them back up after a res...
Supervisord Example Add Proram25
Monitoring     • Gearman Status        telnet on port 4730        Write “STATUS”         • Gives you the registered func...
PHP IntegrationUsage (PEAR / PECL)Frameworks / IntegrationHandling ConditionsUse Cases
Usage     • Two Options        Net::Gearman (PEAR)        • Implemented through sockets with PHP.        • https://pear.p...
Frameworks and Integration     • GearmanManager - agnostic        https://github.com/brianlmoon/GearmanManager/     • Zen...
Conditions     • Watch for Memory Utilization        Check peak usage then kill and restart the worker     • Don’t execut...
Images     • If you resize images on your web server:        Web servers should serve, not process images.        Images...
Image Processing Example32
Image Processing Example33
Email     • Sending email and/or generating templates and processing       variables can take up time, time that is better...
Email Example35
Email Example36
Log Analysis / Aggregation     • Get all of your logs to a single place     • Process the logs to produce analytical data ...
Log Analysis / Aggregation38
Log Analysis / Aggregation39
Executable Processes     • You need to run an executable process…     • This process takes a given name and tells you how ...
Executable Process Example41
Executable Process Example42
Questions?These slides will be posted to SlideShare & SpeakerDeck.  Slideshare: http://www.slideshare.net/mwillbanks  Sp...
Upcoming SlideShare
Loading in...5
×

Gearman - Northeast PHP 2012

2,198

Published on

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,198
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
30
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Transcript of "Gearman - Northeast PHP 2012"

  1. 1. Northeast PHP August 12, 2012A Job Server to Scale By Mike Willbanks Sr. Web Architect Manager NOOK Developer
  2. 2. Housekeeping… • Talk   Slides will be online later! • Me   Sr. Web Architect Manager at NOOK Developer   Former MNPHP Organizer   Open Source Contributor (Zend Framework and various others)   Where you can find me: • Twitter: mwillbanks G+: Mike Willbanks • IRC (freenode): mwillbanks Blog: http://blog.digitalstruct.com • GitHub: https://github.com/mwillbanks2
  3. 3. Agenda • What is Gearman   A general introduction • Main Concepts   Looking overall at how gearman works for you. • Quick Start   Make it go do something. • Digging in   A detailed look into gearman. • PHP Integration   How you should work with it in PHP including use cases and samples. • Questions   Although you can bring them up at anytime!3
  4. 4. What is Gearman?Official StatementWhat it meansVisual understandingPlatforms
  5. 5. Official Statement “Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel, to load balance processing, and to call functions between languages.”5
  6. 6. What it Means • Gearman consists of a daemon, client and worker   At the core, they are simply small programs. • The daemon handles the negotiation of work   Workers and Clients • The worker does the work • The client requests work to be done6
  7. 7. In Pictures7
  8. 8. Platforms • OS   Linux   Windows (cygwin) • API implementations available   PHP   Perl   Java   Ruby   Python8
  9. 9. Main ConceptsClient -> Daemon -> Worker communicationDistributed Model
  10. 10. Client -> Daemon -> Worker communication10
  11. 11. Distributed Model11
  12. 12. Quick StartInstallationSimple Bash ExamplePHP Related (sorry, I’m all about the PHP)
  13. 13. Installation • Head to gearman.org • Click Download • Click on the LaunchPad download • Download the Binary • Unpack the binary • ./configure && make && make install • Bam! You’re off!   For more advanced configuration see ./configure –help • Starting   gearmand -d13
  14. 14. Gearmand Usage • gearmand   -d Run as background daemon   -u [user] Run as user   -L [host] Listen on host/ip   -p [port] Listen on port   -t [threads] Number of threads to use   -v[vv] Verbosity14
  15. 15. Simple Bash Example • Starting the Daemon   gearmand –d • Worker – command line style   nohup gearman -w -f wc -- wc –l & • Run the worker in the background. • Client – command line style   gearman -f wc < /etc/passwd • Outputs the number of lines.15
  16. 16. Gearman Client Command Line Usage • gearman   -w Worker mode   -f [function] Function name to use   -h [host] Job server host   -p [port] Job server port   -t [timeout] Timeout in milliseconds   -H Full options for both clients and workers.16
  17. 17. Digging InPersistenceWorkersMonitoring
  18. 18. Persistence • Gearman by default is an in-memory queue   Leaving this as the default is ideal; however, does not work in all environments. • Persistent Queues   Libdrizzle   Libsqlite3   Libmemcached   Postgres   TokyoCabinet   MySQL   Redis18
  19. 19. Getting Up and Running with Persistence • Persistent queues require specific configuration during the compilation of gearman. • Additionally, arguments to the gearman daemon need to be passed to talk to the specific persistence layer. • Each persistence layer is actually built as a plugin to gearmand   http://bazaar.launchpad.net/~tangent-org/gearmand/trunk/files/ head:/libgearman-server/plugins/queue/19
  20. 20. Configuration Options20
  21. 21. Clients • Clients send work to the gearmand server   This is called the workload; it can be anything that can become a string.   Utilize an open format; it will make life easier in the event you use multiple programming languages, are debugging or the like. • XML, JSON, etc. • Yes, you can serialize objects if you wanted to. –  I recommend against this.21
  22. 22. Workers • Workers are the dudes in the factory doing all the work • Generally they will run as a daemon in the background • Workers register a function that they perform   They should ONLY be doing a single task.   This makes them far easier to manage. • The worker does the work and “can” return results   If you are doing the work asynchronously you generally do not return the result.   Synchronous work you will return the result.22
  23. 23. Workers – special notes • Utilizing the Database   If you keep a database connection • Must have the ability to reconnect to the database. • Watch for connection timeouts • Handling Memory Leaks   Watch the amount of memory and detect leaks then kill the worker. • Request Languages   PHP for instance, sometimes slows down after hundreds of executions, kill it off if you know this will happen.23
  24. 24. Keeping the Daemon Running • Workers sometimes have issues and die, or you need to boot them back up after a restart   Utilizing a service to watch your workers and ensure they are always running is a GOOD thing. • Supervisord   Can watch processes, restart them if they die or get killed   Can manage multiple processes of the same program   Can start and stop your workers.   Running: supervisord –c myconfig.conf • When running workers, BE SURE to handle KILL signals such as SIGKILL.24
  25. 25. Supervisord Example Add Proram25
  26. 26. Monitoring • Gearman Status   telnet on port 4730   Write “STATUS” • Gives you the registered functions, number of workers and items in the queue. • Gearman Monitor – PHP Project   Basic monitoring; but works and it is open source so you can improve it!   https://github.com/yugene/Gearman-Monitor26
  27. 27. PHP IntegrationUsage (PEAR / PECL)Frameworks / IntegrationHandling ConditionsUse Cases
  28. 28. Usage • Two Options   Net::Gearman (PEAR) • Implemented through sockets with PHP. • https://pear.php.net/package/Net_Gearman/   Gearman Extension (PECL) • Implemented through the C API from libgearman • http://pecl.php.net/package/gearman28
  29. 29. Frameworks and Integration • GearmanManager - agnostic   https://github.com/brianlmoon/GearmanManager/ • Zend Framework 1: Zend_Gearman   https://github.com/mwillbanks/Zend_Gearman • Zend Framework 2: mwGearman   https://github.com/mwillbanks/mwGearman • Drupal   http://drupal.org/node/78329429
  30. 30. Conditions • Watch for Memory Utilization   Check peak usage then kill and restart the worker • Don’t execute too many times   PHP is not great at unlimited loops • Keep your memory free   Garbage collect when you can! • Databases   Implement a callback to ensure that you do not timeout; otherwise implement a reconnection.30
  31. 31. Images • If you resize images on your web server:   Web servers should serve, not process images.   Images require a lot of memory AND processing power • They are best to be processed on their own! • Processing in the Background   Generally will require a change to your workflow and checking the status with XHR to see if the job has been completed. • This allows you to process them as you have resources available. • Have enough workers to process them “quickly enough” • Or just do it synchronously31
  32. 32. Image Processing Example32
  33. 33. Image Processing Example33
  34. 34. Email • Sending email and/or generating templates and processing variables can take up time, time that is better spent getting the user to the next page. • The feedback on the mail doesn’t really make a difference so it is great to send it to the background.34
  35. 35. Email Example35
  36. 36. Email Example36
  37. 37. Log Analysis / Aggregation • Get all of your logs to a single place • Process the logs to produce analytical data • Impression / Click Tracking • Why run introspection over the log file itself?   Near real-time analysis is possible!37
  38. 38. Log Analysis / Aggregation38
  39. 39. Log Analysis / Aggregation39
  40. 40. Executable Processes • You need to run an executable process… • This process takes a given name and tells you how many processes are running on your worker machine.   Purely for example purposes; however, you might want to run SaaS against a CMS or something to that degree.40
  41. 41. Executable Process Example41
  42. 42. Executable Process Example42
  43. 43. Questions?These slides will be posted to SlideShare & SpeakerDeck.  Slideshare: http://www.slideshare.net/mwillbanks  SpeakerDeck: http://speakerdeck.com/u/mwillbanks  Twitter: mwillbanks  G+: Mike Willbanks  IRC (freenode): mwillbanks  Blog: http://blog.digitalstruct.com  GitHub: https://github.com/mwillbanks
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×