ARBYTE
      ETYBRA
       Alistair N. MacLeod
        anm@lokku.com




ltd
Arbyte

  Alistair N.
  MacLeod

Motivation         1 Motivation
Problem
Requirements
Existing Systems
Arbyte

Architectur...
Introduction

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems
Arbyte                ...
Requirements

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements           Fully scalable
Existing Syste...
Batching

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems
Arbyte

Architecture      ...
Distributed Computing Models

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems
Arbyte...
Distributed Computing Models

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems
Arbyte...
Gearman

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems   Limitations              ...
The Schwartz

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements       Limitations                  Feat...
Helios

    Arbyte

  Alistair N.
  MacLeod

Motivation         Layer over TheSchwartz
Problem
Requirements
Existing Syste...
Non-Perl

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems
Arbyte

Architecture
     ...
Considered

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems
Arbyte
                 ...
Back to Arbyte

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems
Arbyte              ...
Job Producers   JobBuffer                  JobRunner     Helios
                           Manager
Job Producers   JobBuffer...
Job Producers   JobBuffer                  JobRunner     Helios
                              Manager
   Job Producers   Jo...
Job Producers   JobBuffer                  JobRunner     Helios
                              Manager
   Job Producers   Jo...
Job Producers   JobBuffer                    JobRunner     Helios
                                Manager
   Job Producers ...
Job Producers   JobBuffer                       JobRunner     Helios
                                  Manager
  Job Produc...
Job Producers   JobBuffer                    JobRunner     Helios
                               Manager
   Job Producers  ...
Object Implementation: Options

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems
Arby...
Object Implementation: Choice

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems
Arbyt...
Processes

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
                       No threads
Existing...
IPC Requirements

    Arbyte

  Alistair N.
  MacLeod

Motivation         Wanted something with:
Problem
Requirements     ...
Remote Object System

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems
Arbyte        ...
IPC: Implementation Options

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems
       ...
IPC: Implementation Choice

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
                   Event::RPC
Requirem...
Deployment Hardware

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems
Arbyte

Archite...
Grid Management Software

    Arbyte

  Alistair N.
  MacLeod

Motivation         To Manage
Problem
Requirements          ...
Project Status

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
                   Now
Requirements
Existing Syste...
The Route to CPAN

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems
Arbyte           ...
Questions

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems
Arbyte

Architecture
Comp...
Upcoming SlideShare
Loading in...5
×

Arbyte - A modular, flexible, scalable job queing and execution system

4,123

Published on

Talk given at the London Perl Workshop 2008.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,123
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
40
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Arbyte - A modular, flexible, scalable job queing and execution system

  1. 1. ARBYTE ETYBRA Alistair N. MacLeod anm@lokku.com ltd
  2. 2. Arbyte Alistair N. MacLeod Motivation 1 Motivation Problem Requirements Existing Systems Arbyte Architecture 2 Architecture Component Diagram Design and Implementa- tion 3 Design and Implementation Objects Processes IPC Practicalities 4 Practicalities Deployment Project Status
  3. 3. Introduction Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Arbyte - Job queuing and execution framework Architecture Component Required system to run jobs Diagram Design and Considered gearman, TheSchwartz . . . Implementa- tion Decided to create wrapper - Arbyte Objects Processes IPC Practicalities Deployment Project Status
  4. 4. Requirements Arbyte Alistair N. MacLeod Motivation Problem Requirements Fully scalable Existing Systems Arbyte Modular Architecture Component Logging Diagram Design and Good reliability Implementa- tion Thor Compliance Objects Processes IPC Batching Practicalities Deployment Project Status
  5. 5. Batching Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Job specific optimisations Component Diagram In main queue Design and Implementa- tion Objects Processes IPC Practicalities Deployment Project Status
  6. 6. Distributed Computing Models Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Cluster Architecture Component Diagram Grid Design and Implementa- MapReduce tion Objects Processes IPC Practicalities Deployment Project Status
  7. 7. Distributed Computing Models Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Cluster Architecture Component Diagram Grid Design and Implementa- MapReduce tion Objects Processes IPC Practicalities Deployment Project Status
  8. 8. Gearman Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Limitations Features Arbyte Architecture Not reliable Has multiple manager / Component queuing daemons Diagram No retries Design and Scalable Implementa- It didn’t work when I tried No single point of tion Objects it failure Processes IPC Practicalities Deployment Project Status
  9. 9. The Schwartz Arbyte Alistair N. MacLeod Motivation Problem Requirements Limitations Features Existing Systems Arbyte Single DB store - not Reliability Architecture Component easily scalable Diagram Design and No batching after Implementa- tion submission. Objects Processes Relational DB overhead IPC Practicalities Deployment Project Status
  10. 10. Helios Arbyte Alistair N. MacLeod Motivation Layer over TheSchwartz Problem Requirements Existing Systems Arbyte Architecture Component Diagram Limitations Features Design and Implementa- Same as TheSchwartz Manages worker processes tion Objects Doesn’t add batching or Adds XML Job submission Processes IPC change the fundamental format and web interface Practicalities architecture Deployment Project Status
  11. 11. Non-Perl Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Possible but not as good for hacking on, integrating Component Diagram components. Design and We mostly have perl skills. Implementa- tion Objects Processes IPC Practicalities Deployment Project Status
  12. 12. Considered Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Torque Architecture Component Diagram Hadoop Design and Implementa- Dr. Queue tion Objects Processes IPC Practicalities Deployment Project Status
  13. 13. Back to Arbyte Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Modular framework for job queuing and execution Architecture Component Flexible, Customisable Diagram Design and Can be used with many other systems Implementa- tion e.g. Gearman, with batching, reliability and retries Objects Processes IPC Practicalities Deployment Project Status
  14. 14. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor
  15. 15. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Responsibilities Storing Jobs Job Specific Optimisations Batching Priorities Notes Currently have JobBuffer::Simple
  16. 16. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Responsibilities Logging Retries Basic load balancing Notes Only “active” component
  17. 17. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Responsibilities Arrange for Job Execution Notes Consistent interface
  18. 18. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Notes JobRunner::Simple is implemented Forks a helper process Others are examples (todo)
  19. 19. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Responsibilities Run Job code Report success / failure Notes Classes correspond to Job classes
  20. 20. Object Implementation: Options Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Homemade Component Diagram Moose Design and Implementa- tion Objects Processes IPC Practicalities Deployment Project Status
  21. 21. Object Implementation: Choice Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Using homemade objects Architecture Component Diagram All hashes Design and Implementa- AUTOLOADed get and set methods tion Objects Processes IPC Practicalities Deployment Project Status
  22. 22. Processes Arbyte Alistair N. MacLeod Motivation Problem Requirements No threads Existing Systems Arbyte JobBuffer Architecture Component JobRunner Diagram Will likely have own processes Design and Implementa- e.g. JobRunnerHelper tion Objects Manager Processes IPC StatusAccepter Practicalities Deployment Project Status
  23. 23. IPC Requirements Arbyte Alistair N. MacLeod Motivation Wanted something with: Problem Requirements Easy way to serverify an object Existing Systems Arbyte Stub generation Architecture Component Diagram Parameter passing Design and Exceptions Implementa- tion Objects Timeouts Processes IPC Security Practicalities Deployment Garbage collection Project Status
  24. 24. Remote Object System Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Object Oriented Design RMI like system Architecture Component Assumed RMI on CPAN (Ruby has it, DRb) but no Diagram Feel like fixing this? Design and Implementa- tion Had to make do Objects Processes IPC Practicalities Deployment Project Status
  25. 25. IPC: Implementation Options Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Considered Arbyte Architecture GRID::Machine Component Diagram Distributed::Process Design and Implementa- RPC::Serialized tion Objects RCGI - RPC with CGI server Processes IPC Practicalities Deployment Project Status
  26. 26. IPC: Implementation Choice Arbyte Alistair N. MacLeod Motivation Problem Event::RPC Requirements Existing Systems Closest to RMI Arbyte Architecture Maintained Component Diagram Has (some) timeouts Design and Implementa- Propagates Exceptions tion Objects Processes Confusing - capabilities not clear IPC Using some hackery to make it Good Enough Practicalities Deployment Project Status
  27. 27. Deployment Hardware Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Own Servers Component Diagram Design and Cloud Implementa- tion Objects Processes IPC Practicalities Deployment Project Status
  28. 28. Grid Management Software Arbyte Alistair N. MacLeod Motivation To Manage Problem Requirements Booting Existing Systems Arbyte Package distribution Architecture Component Diagram Configuration Design and For example Implementa- tion Objects RPMs Processes IPC Puppet Practicalities Deployment Wigwam Project Status
  29. 29. Project Status Arbyte Alistair N. MacLeod Motivation Problem Now Requirements Existing Systems Running in parallel with production system Arbyte Architecture Todo Component Diagram Better JobBuffers Design and Implementa- Better JobRunners tion Objects Processes Worker capabilities? IPC Optimise Practicalities Deployment Project Status
  30. 30. The Route to CPAN Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Object system Architecture Component Config system Diagram Design and High level documentation Implementa- tion More tests Objects Processes IPC Practicalities Deployment Project Status
  31. 31. Questions Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Component Diagram Questions Design and Implementa- tion Objects Processes IPC Practicalities Deployment Project Status
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×