1.
Robust worker pool
for Jesque
@moznion

2.
@moznion

3.
Resque / Jesque

4.
Resque / Jesque
Resque is a Redis-backed Ruby library for
creating background jobs, placing them on
multiple queues, and processing them later.
(Jesque is a Java port of Resque)

5.
Redis
Client
Resque
Enqueue job(s)
Poll any new jobs
Obtain a job

6.
Resque has one or more workers.
Workers poll any new jobs
with accessing to Redis.
And a worker execute a retrieved job.

7.
Resque
.
.
.
Poll any new jobs
Obtain a job
Execute an obtained job
on a worker process
(or thread)
Workers

8.
Worker Pooling

9.
Resque
Execute an obtained job
on a worker process
(or thread)
Worker
Poll any new jobs
Obtain a job

10.
Resque
Execute an obtained job
on a worker process
(or thread)
Worker
Blocks!!!!
Poll any new jobs
Obtain a job

11.
Resque
.
.
.
Poll any new jobs
Obtain a job
Execute an obtained job
on a worker process
(or thread)
Workers

12.
Resque
.
.
.
Poll any new jobs
Obtain a job
Execute an obtained job
on a worker process
(or thread)
Workers
Concurrently

13.
Worker pooling is needed
to execute jobs concurrently

14.
Jesque

15.
Jesque has a mechanism
of worker pooling, but…

16.
Jesque
Poll any new jobs
Obtain a job
Execute an obtained job
on a worker process
(or thread)
Workers

17.
Jesque
Workers

18.
Jesque
DIE FOREVER!!!Workers

19.
Jesque
Workers

20.
Jesque
Workers

21.
Jesque
And Then There Were None…
Workers

22.
Not robust…

23.
RobustWorkerPool
for Jesque
https://github.com/moznion/jesque-robust-worker-pool
http://theartmad.com/wp-content/uploads/2015/06/Arnold-Schwarzenegger-18.jpg

24.
Jesque
Poll any new jobs
Obtain a job
Workers

25.
Jesque
Workers
Detects WORKER_STOP

26.
Jesque
Workers
Remove from worker pool

27.
Jesque
Workers
Create a new worker
based on a died worker

28.
Jesque
Poll any new jobs
Obtain a job
Workers
And join to pool

29.
RobustWorkerPool
monitors status of workers,
alive or not.

30.
And if any worker is died,
manager make a new
workers
(Rarely case; if excess
workers exist then manager
removes excesses)