MinionPool

A worker pool for NodeJS
MinionPool - Overview
●
●
●
●
●
●

●
●

Easily create worker pools in seconds
Based on callbacks (hooks) into the lifecycl...
MinionPool - Proven in the field
●
●
●
●
●

Log files processing
Large mysql tables processing
ElasticSearch reindexing
S3...
MinionPool - Installing
●
●
●

npm install minionpool
npm install mysql_minionpool
npm install rabbitmq_minionpool
MinionPool - As dependency
{
"dependencies": {
"minionpool": "*",
"rabbitmq_minionpool": "*",
"mysql_minionpool": "*"
}
}
MinionPool - Concepts
●
●

TaskSource: Used to produce and introduce tasks into the pool. A task
source will be initialize...
MinionPool - How to use it
var minionpoolMod = require('minionpool');
var minionPool = new minionpoolMod.MinionPool(option...
MinionPool - Options
var options = {
name: 'test',
debug: true,
concurrency: 5,
logger: console.log,
continueOnError: true...
MinionPool - ArrayMinionPool
var options = {
name: 'test',
concurrency: 5,
minionTaskHandler: function(task, state, callba...
MinionPool - MysqlMinionPool
var pool = new mysqlMinionPoolMod.MysqlMinionPool({
mysqlConfig: {
host: '127.0.0.1',
user: '...
MinionPool - MysqlMinionPool
taskSourceNext: function(state, callback) {
var query = "SELECT * FROM `db`.`table` LIMIT ?,?...
MinionPool - RabbitMQMinionPool
var options = {
name: 'test',
concurrency: 5,
mqOptions: {
exchangeName: 'workers', // Wil...
MinionPool - RabbitMQMinionPool
minionTaskHandler: function(msg, state, callback) {
var payload = msg.payload;
var headers...
MinionPool - RabbitMQMinionPool
●
●
●
●

Dead Letter eXchanges to support retrying failed (rejected) operations
Channels i...
MinionPool - MultiCore
taskset: "used to set or retrieve the CPU affinity of a running process given its
PID or to launch ...
MinionPool - MultiCore
For mysql minion pools: you can divide the total number of rows per the
number of cores available, ...
MinionPool - Links

●
●
●

https://github.com/marcelog/minionpool
https://github.com/marcelog/mysql_minionpool
https://git...
MinionPool

Thank you, Inakos!
Upcoming SlideShare
Loading in...5
×

Minion pool - a worker pool for nodejs

1,773

Published on

Internal talk at Inaka. An overview of the minionpool set of projects. Essentially a worker pool for nodejs that can consume tasks from rabbitmq, mysql, an array, very suitable to work with a huge number of tasks, most useful for idempotent operations.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,773
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
5
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Minion pool - a worker pool for nodejs

  1. 1. MinionPool A worker pool for NodeJS
  2. 2. MinionPool - Overview ● ● ● ● ● ● ● ● Easily create worker pools in seconds Based on callbacks (hooks) into the lifecycle of the pool Let's your do any number and type of tasks in parallel ONLY if those tasks do not rely in the order of processing Best results when doing idempotent operations Actually a family of pools: ○ MinionPool ○ ArrayMinionPool ○ MysqlMinionPool ○ RabbitMQMinionPool Can poll for tasks or accept injected tasks Open Source
  3. 3. MinionPool - Proven in the field ● ● ● ● ● Log files processing Large mysql tables processing ElasticSearch reindexing S3 uploaders Web crawlers
  4. 4. MinionPool - Installing ● ● ● npm install minionpool npm install mysql_minionpool npm install rabbitmq_minionpool
  5. 5. MinionPool - As dependency { "dependencies": { "minionpool": "*", "rabbitmq_minionpool": "*", "mysql_minionpool": "*" } }
  6. 6. MinionPool - Concepts ● ● TaskSource: Used to produce and introduce tasks into the pool. A task source will be initialized, used, and then shutdown Minion: Each pool consists of N minions. Tasks will be assigned to the first available minion. Each minion will be initialized, used, and then shutdown when the TaskSource has reported that no more tasks are available. MySQL RabbitMQ W1 Poll Inject Task Source Pool W2
  7. 7. MinionPool - How to use it var minionpoolMod = require('minionpool'); var minionPool = new minionpoolMod.MinionPool(options); minionPool.start();
  8. 8. MinionPool - Options var options = { name: 'test', debug: true, concurrency: 5, logger: console.log, continueOnError: true, taskSourceStart: function(callback) { … callback(err, state); } taskSourceNext: function(state, callback) { callback(err, task); return state; }, taskSourceEnd: function(state, callback) { callback(); }, minionTaskHandler: function(task, state, callback) { callback(err, state); }, minionStart: function(callback) { callback(err, state); }, minionEnd: function(state, callback) { callback(); }, poolEnd: function() { process.exit(0); } };
  9. 9. MinionPool - ArrayMinionPool var options = { name: 'test', concurrency: 5, minionTaskHandler: function(task, state, callback) { setTimeout(function() { callback(undefined, state); }, Math.floor(Math.random() * 500)); }, poolEnd: function() { process.exit(0); } }; var data = [.., .., .., .., ...]; var minionPool = new minionsMod.ArrayMinionPool(options, data); minionPool.start();
  10. 10. MinionPool - MysqlMinionPool var pool = new mysqlMinionPoolMod.MysqlMinionPool({ mysqlConfig: { host: '127.0.0.1', user: 'root', password: 'pass', database: 'db', port: 3306 }, concurrency: 5, // How many pages to get concurrently... rowConcurrency: 1, // ... and how many concurrent rows processed PER query taskSourceStart: function(callback) { callback(undefined, {page: 0, pageSize: 10}); }, minionTaskHandler: function(task, state, callback) { callback(undefined, state); } }); pool.start();
  11. 11. MinionPool - MysqlMinionPool taskSourceNext: function(state, callback) { var query = "SELECT * FROM `db`.`table` LIMIT ?,?"; state.mysqlPool.getConnection(function(err, mysqlConnection) { if(err) { callback(err, undefined); } else { mysqlConnection.query( query, [state.page * state.pageSize, state.pageSize], function(err, rows) { mysqlConnection.release(); if(err) { callback(err, undefined); } else if(rows.length === 0) { callback(undefined, undefined); } else { callback(undefined, rows); } } ); } }); state.page++; return state; },
  12. 12. MinionPool - RabbitMQMinionPool var options = { name: 'test', concurrency: 5, mqOptions: { exchangeName: 'workers', // Will also create workers.retry queueName: 'myWorkers', // Will also create myWorkers.retry routingKey: 'myWorkers', // Optional. Equals to queueName if missing } }; var pool = new minionsMod.RabbitMqMinionPool(options); process.on('SIGINT', function() { pool.end(); }); pool.start();
  13. 13. MinionPool - RabbitMQMinionPool minionTaskHandler: function(msg, state, callback) { var payload = msg.payload; var headers = msg.headers; var deliveryInfo = msg.deliveryInfo; var message = msg.message; var queue = msg.queue; console.log('got task: %s', util.inspect(payload)); // See the node-amqp doc for more info. message.reject(); // or message.acknowledge(); callback(undefined, state); },
  14. 14. MinionPool - RabbitMQMinionPool ● ● ● ● Dead Letter eXchanges to support retrying failed (rejected) operations Channels in confirmation mode, so failed publishes will be notified Messages published as persistent Queues and exchanges marked as durable, autodelete = false
  15. 15. MinionPool - MultiCore taskset: "used to set or retrieve the CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity" ● ● http://linux.die.net/man/1/taskset https://github.com/karelzak/util-linux $ taskset -c N /usr/node/bin/node bin/myworker # 0 =< N < number of cores
  16. 16. MinionPool - MultiCore For mysql minion pools: you can divide the total number of rows per the number of cores available, and launch one pool per core that will process then given range of id's. For rabbitmq minion pools: Just spawn as many pools per core needed, they will compete to consume tasks, rabbitmq will handle the complex part :)
  17. 17. MinionPool - Links ● ● ● https://github.com/marcelog/minionpool https://github.com/marcelog/mysql_minionpool https://github.com/marcelog/rabbitmq_minionpool
  18. 18. MinionPool Thank you, Inakos!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×