Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Robust worker pool
for Jesque
@moznion
@moznion
Resque / Jesque
Resque / Jesque
Resque is a Redis-backed Ruby library for
creating background jobs, placing them on
multiple queues, and p...
Redis
Client
Resque
Enqueue job(s)
Poll any new jobs
Obtain a job
Resque has one or more workers.
Workers poll any new jobs
with accessing to Redis.
And a worker execute a retrieved job.
Resque
.
.
.
Poll any new jobs
Obtain a job
Execute an obtained job
on a worker process
(or thread)
Workers
Worker Pooling
Resque
Execute an obtained job
on a worker process
(or thread)
Worker
Poll any new jobs
Obtain a job
Resque
Execute an obtained job
on a worker process
(or thread)
Worker
Blocks!!!!
Poll any new jobs
Obtain a job
Resque
.
.
.
Poll any new jobs
Obtain a job
Execute an obtained job
on a worker process
(or thread)
Workers
Resque
.
.
.
Poll any new jobs
Obtain a job
Execute an obtained job
on a worker process
(or thread)
Workers
Concurrently
Worker pooling is needed
to execute jobs concurrently
Jesque
Jesque has a mechanism
of worker pooling, but…
Jesque
Poll any new jobs
Obtain a job
Execute an obtained job
on a worker process
(or thread)
Workers
Jesque
Workers
Jesque
DIE FOREVER!!!Workers
Jesque
Workers
Jesque
Workers
Jesque
And Then There Were None…
Workers
Not robust…
RobustWorkerPool
for Jesque
https://github.com/moznion/jesque-robust-worker-pool
http://theartmad.com/wp-content/uploads/2...
Jesque
Poll any new jobs
Obtain a job
Workers
Jesque
Workers
Detects WORKER_STOP
Jesque
Workers
Remove from worker pool
Jesque
Workers
Create a new worker
based on a died worker
Jesque
Poll any new jobs
Obtain a job
Workers
And join to pool
RobustWorkerPool
monitors status of workers,
alive or not.
And if any worker is died,
manager make a new
workers
(Rarely case; if excess
workers exist then manager
removes excesses)
Upcoming SlideShare
Loading in …5
×

Jesque robust-worker-pool

2,416 views

Published on

Introduce about `RobustWorkerPool` that is an implementation of worker pooling for Jesque.

Published in: Technology
  • Be the first to comment

Jesque robust-worker-pool

  1. 1. Robust worker pool for Jesque @moznion
  2. 2. @moznion
  3. 3. Resque / Jesque
  4. 4. Resque / Jesque Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later. (Jesque is a Java port of Resque)
  5. 5. Redis Client Resque Enqueue job(s) Poll any new jobs Obtain a job
  6. 6. Resque has one or more workers. Workers poll any new jobs with accessing to Redis. And a worker execute a retrieved job.
  7. 7. Resque . . . Poll any new jobs Obtain a job Execute an obtained job on a worker process (or thread) Workers
  8. 8. Worker Pooling
  9. 9. Resque Execute an obtained job on a worker process (or thread) Worker Poll any new jobs Obtain a job
  10. 10. Resque Execute an obtained job on a worker process (or thread) Worker Blocks!!!! Poll any new jobs Obtain a job
  11. 11. Resque . . . Poll any new jobs Obtain a job Execute an obtained job on a worker process (or thread) Workers
  12. 12. Resque . . . Poll any new jobs Obtain a job Execute an obtained job on a worker process (or thread) Workers Concurrently
  13. 13. Worker pooling is needed to execute jobs concurrently
  14. 14. Jesque
  15. 15. Jesque has a mechanism of worker pooling, but…
  16. 16. Jesque Poll any new jobs Obtain a job Execute an obtained job on a worker process (or thread) Workers
  17. 17. Jesque Workers
  18. 18. Jesque DIE FOREVER!!!Workers
  19. 19. Jesque Workers
  20. 20. Jesque Workers
  21. 21. Jesque And Then There Were None… Workers
  22. 22. Not robust…
  23. 23. RobustWorkerPool for Jesque https://github.com/moznion/jesque-robust-worker-pool http://theartmad.com/wp-content/uploads/2015/06/Arnold-Schwarzenegger-18.jpg
  24. 24. Jesque Poll any new jobs Obtain a job Workers
  25. 25. Jesque Workers Detects WORKER_STOP
  26. 26. Jesque Workers Remove from worker pool
  27. 27. Jesque Workers Create a new worker based on a died worker
  28. 28. Jesque Poll any new jobs Obtain a job Workers And join to pool
  29. 29. RobustWorkerPool monitors status of workers, alive or not.
  30. 30. And if any worker is died, manager make a new workers (Rarely case; if excess workers exist then manager removes excesses)

×