Arbyte - A modular, flexible, scalable job queing and execution system
Report
•3 likes•1,385 views
Arbyte - A modular, flexible, scalable job queing and execution system
•3 likes•1,385 views
Download to read offline
Report
Talk given at the London Perl Workshop 2008.
Recommended
More Related Content
Similar to Arbyte - A modular, flexible, scalable job queing and execution system
Similar to Arbyte - A modular, flexible, scalable job queing and execution system(20)
More from lokku
A living hell - lessons learned in eight years of parsing real estate data lokku
5.7K views•28 slides
More from lokku(20)
Recently uploaded
Recently uploaded(20)
Arbyte - A modular, flexible, scalable job queing and execution system
- 1. ARBYTE ETYBRA Alistair N. MacLeod anm@lokku.com ltd
- 2. Arbyte Alistair N. MacLeod Motivation 1 Motivation Problem Requirements Existing Systems Arbyte Architecture 2 Architecture Component Diagram Design and Implementa- tion 3 Design and Implementation Objects Processes IPC Practicalities 4 Practicalities Deployment Project Status
- 3. Introduction Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Arbyte - Job queuing and execution framework Architecture Component Required system to run jobs Diagram Design and Considered gearman, TheSchwartz . . . Implementa- tion Decided to create wrapper - Arbyte Objects Processes IPC Practicalities Deployment Project Status
- 4. Requirements Arbyte Alistair N. MacLeod Motivation Problem Requirements Fully scalable Existing Systems Arbyte Modular Architecture Component Logging Diagram Design and Good reliability Implementa- tion Thor Compliance Objects Processes IPC Batching Practicalities Deployment Project Status
- 5. Batching Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Job specific optimisations Component Diagram In main queue Design and Implementa- tion Objects Processes IPC Practicalities Deployment Project Status
- 6. Distributed Computing Models Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Cluster Architecture Component Diagram Grid Design and Implementa- MapReduce tion Objects Processes IPC Practicalities Deployment Project Status
- 7. Distributed Computing Models Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Cluster Architecture Component Diagram Grid Design and Implementa- MapReduce tion Objects Processes IPC Practicalities Deployment Project Status
- 8. Gearman Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Limitations Features Arbyte Architecture Not reliable Has multiple manager / Component queuing daemons Diagram No retries Design and Scalable Implementa- It didn’t work when I tried No single point of tion Objects it failure Processes IPC Practicalities Deployment Project Status
- 9. The Schwartz Arbyte Alistair N. MacLeod Motivation Problem Requirements Limitations Features Existing Systems Arbyte Single DB store - not Reliability Architecture Component easily scalable Diagram Design and No batching after Implementa- tion submission. Objects Processes Relational DB overhead IPC Practicalities Deployment Project Status
- 10. Helios Arbyte Alistair N. MacLeod Motivation Layer over TheSchwartz Problem Requirements Existing Systems Arbyte Architecture Component Diagram Limitations Features Design and Implementa- Same as TheSchwartz Manages worker processes tion Objects Doesn’t add batching or Adds XML Job submission Processes IPC change the fundamental format and web interface Practicalities architecture Deployment Project Status
- 11. Non-Perl Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Possible but not as good for hacking on, integrating Component Diagram components. Design and We mostly have perl skills. Implementa- tion Objects Processes IPC Practicalities Deployment Project Status
- 12. Considered Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Torque Architecture Component Diagram Hadoop Design and Implementa- Dr. Queue tion Objects Processes IPC Practicalities Deployment Project Status
- 13. Back to Arbyte Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Modular framework for job queuing and execution Architecture Component Flexible, Customisable Diagram Design and Can be used with many other systems Implementa- tion e.g. Gearman, with batching, reliability and retries Objects Processes IPC Practicalities Deployment Project Status
- 14. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor
- 15. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Responsibilities Storing Jobs Job Specific Optimisations Batching Priorities Notes Currently have JobBuffer::Simple
- 16. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Responsibilities Logging Retries Basic load balancing Notes Only “active” component
- 17. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Responsibilities Arrange for Job Execution Notes Consistent interface
- 18. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Notes JobRunner::Simple is implemented Forks a helper process Others are examples (todo)
- 19. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Responsibilities Run Job code Report success / failure Notes Classes correspond to Job classes
- 20. Object Implementation: Options Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Homemade Component Diagram Moose Design and Implementa- tion Objects Processes IPC Practicalities Deployment Project Status
- 21. Object Implementation: Choice Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Using homemade objects Architecture Component Diagram All hashes Design and Implementa- AUTOLOADed get and set methods tion Objects Processes IPC Practicalities Deployment Project Status
- 22. Processes Arbyte Alistair N. MacLeod Motivation Problem Requirements No threads Existing Systems Arbyte JobBuffer Architecture Component JobRunner Diagram Will likely have own processes Design and Implementa- e.g. JobRunnerHelper tion Objects Manager Processes IPC StatusAccepter Practicalities Deployment Project Status
- 23. IPC Requirements Arbyte Alistair N. MacLeod Motivation Wanted something with: Problem Requirements Easy way to serverify an object Existing Systems Arbyte Stub generation Architecture Component Diagram Parameter passing Design and Exceptions Implementa- tion Objects Timeouts Processes IPC Security Practicalities Deployment Garbage collection Project Status
- 24. Remote Object System Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Object Oriented Design RMI like system Architecture Component Assumed RMI on CPAN (Ruby has it, DRb) but no Diagram Feel like fixing this? Design and Implementa- tion Had to make do Objects Processes IPC Practicalities Deployment Project Status
- 25. IPC: Implementation Options Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Considered Arbyte Architecture GRID::Machine Component Diagram Distributed::Process Design and Implementa- RPC::Serialized tion Objects RCGI - RPC with CGI server Processes IPC Practicalities Deployment Project Status
- 26. IPC: Implementation Choice Arbyte Alistair N. MacLeod Motivation Problem Event::RPC Requirements Existing Systems Closest to RMI Arbyte Architecture Maintained Component Diagram Has (some) timeouts Design and Implementa- Propagates Exceptions tion Objects Processes Confusing - capabilities not clear IPC Using some hackery to make it Good Enough Practicalities Deployment Project Status
- 27. Deployment Hardware Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Own Servers Component Diagram Design and Cloud Implementa- tion Objects Processes IPC Practicalities Deployment Project Status
- 28. Grid Management Software Arbyte Alistair N. MacLeod Motivation To Manage Problem Requirements Booting Existing Systems Arbyte Package distribution Architecture Component Diagram Configuration Design and For example Implementa- tion Objects RPMs Processes IPC Puppet Practicalities Deployment Wigwam Project Status
- 29. Project Status Arbyte Alistair N. MacLeod Motivation Problem Now Requirements Existing Systems Running in parallel with production system Arbyte Architecture Todo Component Diagram Better JobBuffers Design and Implementa- Better JobRunners tion Objects Processes Worker capabilities? IPC Optimise Practicalities Deployment Project Status
- 30. The Route to CPAN Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Object system Architecture Component Config system Diagram Design and High level documentation Implementa- tion More tests Objects Processes IPC Practicalities Deployment Project Status
- 31. Questions Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Component Diagram Questions Design and Implementa- tion Objects Processes IPC Practicalities Deployment Project Status