Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Robust worker pool for Jesque @moznion
@moznion
Resque / Jesque
Resque / Jesque Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and p...
Redis Client Resque Enqueue job(s) Poll any new jobs Obtain a job
Resque has one or more workers. Workers poll any new jobs with accessing to Redis. And a worker execute a retrieved job.
Resque . . . Poll any new jobs Obtain a job Execute an obtained job on a worker process (or thread) Workers
Worker Pooling
Resque Execute an obtained job on a worker process (or thread) Worker Poll any new jobs Obtain a job
Resque Execute an obtained job on a worker process (or thread) Worker Blocks!!!! Poll any new jobs Obtain a job
Resque . . . Poll any new jobs Obtain a job Execute an obtained job on a worker process (or thread) Workers
Resque . . . Poll any new jobs Obtain a job Execute an obtained job on a worker process (or thread) Workers Concurrently
Worker pooling is needed to execute jobs concurrently
Jesque
Jesque has a mechanism of worker pooling, but…
Jesque Poll any new jobs Obtain a job Execute an obtained job on a worker process (or thread) Workers
Jesque Workers
Jesque DIE FOREVER!!!Workers
Jesque Workers
Jesque Workers
Jesque And Then There Were None… Workers
Not robust…
RobustWorkerPool for Jesque https://github.com/moznion/jesque-robust-worker-pool http://theartmad.com/wp-content/uploads/2...
Jesque Poll any new jobs Obtain a job Workers
Jesque Workers Detects WORKER_STOP
Jesque Workers Remove from worker pool
Jesque Workers Create a new worker based on a died worker
Jesque Poll any new jobs Obtain a job Workers And join to pool
RobustWorkerPool monitors status of workers, alive or not.
And if any worker is died, manager make a new workers (Rarely case; if excess workers exist then manager removes excesses)
Upcoming SlideShare
Loading in …5
×

Jesque robust-worker-pool

2,416 views

Published on

Introduce about `RobustWorkerPool` that is an implementation of worker pooling for Jesque.

Published in: Technology
no profile picture user

  • Be the first to comment

Jesque robust-worker-pool

  1. 1. Robust worker pool for Jesque @moznion
  2. 2. @moznion
  3. 3. Resque / Jesque
  4. 4. Resque / Jesque Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later. (Jesque is a Java port of Resque)
  5. 5. Redis Client Resque Enqueue job(s) Poll any new jobs Obtain a job
  6. 6. Resque has one or more workers. Workers poll any new jobs with accessing to Redis. And a worker execute a retrieved job.
  7. 7. Resque . . . Poll any new jobs Obtain a job Execute an obtained job on a worker process (or thread) Workers
  8. 8. Worker Pooling
  9. 9. Resque Execute an obtained job on a worker process (or thread) Worker Poll any new jobs Obtain a job
  10. 10. Resque Execute an obtained job on a worker process (or thread) Worker Blocks!!!! Poll any new jobs Obtain a job
  11. 11. Resque . . . Poll any new jobs Obtain a job Execute an obtained job on a worker process (or thread) Workers
  12. 12. Resque . . . Poll any new jobs Obtain a job Execute an obtained job on a worker process (or thread) Workers Concurrently
  13. 13. Worker pooling is needed to execute jobs concurrently
  14. 14. Jesque
  15. 15. Jesque has a mechanism of worker pooling, but…
  16. 16. Jesque Poll any new jobs Obtain a job Execute an obtained job on a worker process (or thread) Workers
  17. 17. Jesque Workers
  18. 18. Jesque DIE FOREVER!!!Workers
  19. 19. Jesque Workers
  20. 20. Jesque Workers
  21. 21. Jesque And Then There Were None… Workers
  22. 22. Not robust…
  23. 23. RobustWorkerPool for Jesque https://github.com/moznion/jesque-robust-worker-pool http://theartmad.com/wp-content/uploads/2015/06/Arnold-Schwarzenegger-18.jpg
  24. 24. Jesque Poll any new jobs Obtain a job Workers
  25. 25. Jesque Workers Detects WORKER_STOP
  26. 26. Jesque Workers Remove from worker pool
  27. 27. Jesque Workers Create a new worker based on a died worker
  28. 28. Jesque Poll any new jobs Obtain a job Workers And join to pool
  29. 29. RobustWorkerPool monitors status of workers, alive or not.
  30. 30. And if any worker is died, manager make a new workers (Rarely case; if excess workers exist then manager removes excesses)

×