Successfully reported this slideshow.

Jesque robust-worker-pool

1

Share

1 of 30
1 of 30

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

Jesque robust-worker-pool

  1. 1. Robust worker pool for Jesque @moznion
  2. 2. @moznion
  3. 3. Resque / Jesque
  4. 4. Resque / Jesque Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later. (Jesque is a Java port of Resque)
  5. 5. Redis Client Resque Enqueue job(s) Poll any new jobs Obtain a job
  6. 6. Resque has one or more workers. Workers poll any new jobs with accessing to Redis. And a worker execute a retrieved job.
  7. 7. Resque . . . Poll any new jobs Obtain a job Execute an obtained job on a worker process (or thread) Workers
  8. 8. Worker Pooling
  9. 9. Resque Execute an obtained job on a worker process (or thread) Worker Poll any new jobs Obtain a job
  10. 10. Resque Execute an obtained job on a worker process (or thread) Worker Blocks!!!! Poll any new jobs Obtain a job
  11. 11. Resque . . . Poll any new jobs Obtain a job Execute an obtained job on a worker process (or thread) Workers
  12. 12. Resque . . . Poll any new jobs Obtain a job Execute an obtained job on a worker process (or thread) Workers Concurrently
  13. 13. Worker pooling is needed to execute jobs concurrently
  14. 14. Jesque
  15. 15. Jesque has a mechanism of worker pooling, but…
  16. 16. Jesque Poll any new jobs Obtain a job Execute an obtained job on a worker process (or thread) Workers
  17. 17. Jesque Workers
  18. 18. Jesque DIE FOREVER!!!Workers
  19. 19. Jesque Workers
  20. 20. Jesque Workers
  21. 21. Jesque And Then There Were None… Workers
  22. 22. Not robust…
  23. 23. RobustWorkerPool for Jesque https://github.com/moznion/jesque-robust-worker-pool http://theartmad.com/wp-content/uploads/2015/06/Arnold-Schwarzenegger-18.jpg
  24. 24. Jesque Poll any new jobs Obtain a job Workers
  25. 25. Jesque Workers Detects WORKER_STOP
  26. 26. Jesque Workers Remove from worker pool
  27. 27. Jesque Workers Create a new worker based on a died worker
  28. 28. Jesque Poll any new jobs Obtain a job Workers And join to pool
  29. 29. RobustWorkerPool monitors status of workers, alive or not.
  30. 30. And if any worker is died, manager make a new workers (Rarely case; if excess workers exist then manager removes excesses)

×