Arbyte - A modular, flexible, scalable job queing and execution system

ARBYTE ETYBRA Alistair N. MacLeod anm@lokku.com ltd

Arbyte Alistair N. MacLeod Motivation 1 Motivation Problem Requirements Existing Systems Arbyte Architecture 2 Architecture Component Diagram Design and Implementa- tion 3 Design and Implementation Objects Processes IPC Practicalities 4 Practicalities Deployment Project Status

Introduction Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Arbyte - Job queuing and execution framework Architecture Component Required system to run jobs Diagram Design and Considered gearman, TheSchwartz . . . Implementa- tion Decided to create wrapper - Arbyte Objects Processes IPC Practicalities Deployment Project Status

Requirements Arbyte Alistair N. MacLeod Motivation Problem Requirements Fully scalable Existing Systems Arbyte Modular Architecture Component Logging Diagram Design and Good reliability Implementa- tion Thor Compliance Objects Processes IPC Batching Practicalities Deployment Project Status

Batching Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Job speciﬁc optimisations Component Diagram In main queue Design and Implementa- tion Objects Processes IPC Practicalities Deployment Project Status

Distributed Computing Models Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Cluster Architecture Component Diagram Grid Design and Implementa- MapReduce tion Objects Processes IPC Practicalities Deployment Project Status

Gearman Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Limitations Features Arbyte Architecture Not reliable Has multiple manager / Component queuing daemons Diagram No retries Design and Scalable Implementa- It didn’t work when I tried No single point of tion Objects it failure Processes IPC Practicalities Deployment Project Status

The Schwartz Arbyte Alistair N. MacLeod Motivation Problem Requirements Limitations Features Existing Systems Arbyte Single DB store - not Reliability Architecture Component easily scalable Diagram Design and No batching after Implementa- tion submission. Objects Processes Relational DB overhead IPC Practicalities Deployment Project Status

Helios Arbyte Alistair N. MacLeod Motivation Layer over TheSchwartz Problem Requirements Existing Systems Arbyte Architecture Component Diagram Limitations Features Design and Implementa- Same as TheSchwartz Manages worker processes tion Objects Doesn’t add batching or Adds XML Job submission Processes IPC change the fundamental format and web interface Practicalities architecture Deployment Project Status

Non-Perl Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Possible but not as good for hacking on, integrating Component Diagram components. Design and We mostly have perl skills. Implementa- tion Objects Processes IPC Practicalities Deployment Project Status

Considered Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Torque Architecture Component Diagram Hadoop Design and Implementa- Dr. Queue tion Objects Processes IPC Practicalities Deployment Project Status

Back to Arbyte Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Modular framework for job queuing and execution Architecture Component Flexible, Customisable Diagram Design and Can be used with many other systems Implementa- tion e.g. Gearman, with batching, reliability and retries Objects Processes IPC Practicalities Deployment Project Status

Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor

Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Responsibilities Storing Jobs Job Specific Optimisations Batching Priorities Notes Currently have JobBuffer::Simple

Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Responsibilities Logging Retries Basic load balancing Notes Only “active” component

Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Responsibilities Arrange for Job Execution Notes Consistent interface

Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Notes JobRunner::Simple is implemented Forks a helper process Others are examples (todo)

Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Responsibilities Run Job code Report success / failure Notes Classes correspond to Job classes

Object Implementation: Options Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Homemade Component Diagram Moose Design and Implementa- tion Objects Processes IPC Practicalities Deployment Project Status

Object Implementation: Choice Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Using homemade objects Architecture Component Diagram All hashes Design and Implementa- AUTOLOADed get and set methods tion Objects Processes IPC Practicalities Deployment Project Status

Processes Arbyte Alistair N. MacLeod Motivation Problem Requirements No threads Existing Systems Arbyte JobBuﬀer Architecture Component JobRunner Diagram Will likely have own processes Design and Implementa- e.g. JobRunnerHelper tion Objects Manager Processes IPC StatusAccepter Practicalities Deployment Project Status

IPC Requirements Arbyte Alistair N. MacLeod Motivation Wanted something with: Problem Requirements Easy way to serverify an object Existing Systems Arbyte Stub generation Architecture Component Diagram Parameter passing Design and Exceptions Implementa- tion Objects Timeouts Processes IPC Security Practicalities Deployment Garbage collection Project Status

Remote Object System Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Object Oriented Design RMI like system Architecture Component Assumed RMI on CPAN (Ruby has it, DRb) but no Diagram Feel like ﬁxing this? Design and Implementa- tion Had to make do Objects Processes IPC Practicalities Deployment Project Status

IPC: Implementation Options Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Considered Arbyte Architecture GRID::Machine Component Diagram Distributed::Process Design and Implementa- RPC::Serialized tion Objects RCGI - RPC with CGI server Processes IPC Practicalities Deployment Project Status

IPC: Implementation Choice Arbyte Alistair N. MacLeod Motivation Problem Event::RPC Requirements Existing Systems Closest to RMI Arbyte Architecture Maintained Component Diagram Has (some) timeouts Design and Implementa- Propagates Exceptions tion Objects Processes Confusing - capabilities not clear IPC Using some hackery to make it Good Enough Practicalities Deployment Project Status

Deployment Hardware Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Own Servers Component Diagram Design and Cloud Implementa- tion Objects Processes IPC Practicalities Deployment Project Status

Grid Management Software Arbyte Alistair N. MacLeod Motivation To Manage Problem Requirements Booting Existing Systems Arbyte Package distribution Architecture Component Diagram Conﬁguration Design and For example Implementa- tion Objects RPMs Processes IPC Puppet Practicalities Deployment Wigwam Project Status

Project Status Arbyte Alistair N. MacLeod Motivation Problem Now Requirements Existing Systems Running in parallel with production system Arbyte Architecture Todo Component Diagram Better JobBuﬀers Design and Implementa- Better JobRunners tion Objects Processes Worker capabilities? IPC Optimise Practicalities Deployment Project Status

The Route to CPAN Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Object system Architecture Component Conﬁg system Diagram Design and High level documentation Implementa- tion More tests Objects Processes IPC Practicalities Deployment Project Status

Questions Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Component Diagram Questions Design and Implementa- tion Objects Processes IPC Practicalities Deployment Project Status

Arbyte - A modular, flexible, scalable job queing and execution system

0 comments

Notes on slide 1

1 Favorite, 1 Group & 1 Event