SlideShare a Scribd company logo

Arbyte - A modular, flexible, scalable job queing and execution system

lokku
lokku

Talk given at the London Perl Workshop 2008.

1 of 31
Download to read offline
ARBYTE
      ETYBRA
       Alistair N. MacLeod
        anm@lokku.com




ltd
Arbyte

  Alistair N.
  MacLeod

Motivation         1 Motivation
Problem
Requirements
Existing Systems
Arbyte

Architecture       2 Architecture
Component
Diagram

Design and
Implementa-
tion
                   3 Design and Implementation
Objects
Processes
IPC

Practicalities     4 Practicalities
Deployment
Project Status
Introduction

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems
Arbyte                 Arbyte - Job queuing and execution framework
Architecture
Component
                       Required system to run jobs
Diagram

Design and
                       Considered gearman, TheSchwartz . . .
Implementa-
tion                   Decided to create wrapper - Arbyte
Objects
Processes
IPC

Practicalities
Deployment
Project Status
Requirements

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements           Fully scalable
Existing Systems
Arbyte                 Modular
Architecture
Component              Logging
Diagram

Design and             Good reliability
Implementa-
tion                   Thor Compliance
Objects
Processes
IPC
                       Batching
Practicalities
Deployment
Project Status
Batching

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems
Arbyte

Architecture           Job specific optimisations
Component
Diagram
                       In main queue
Design and
Implementa-
tion
Objects
Processes
IPC

Practicalities
Deployment
Project Status
Distributed Computing Models

    Arbyte

  Alistair N.
  MacLeod

Motivation
Problem
Requirements
Existing Systems
Arbyte
                       Cluster
Architecture
Component
Diagram
                       Grid
Design and
Implementa-
                       MapReduce
tion
Objects
Processes
IPC

Practicalities
Deployment
Project Status

Recommended

Introducing SMART Plug-Ins (SPIs)
Introducing SMART Plug-Ins (SPIs)Introducing SMART Plug-Ins (SPIs)
Introducing SMART Plug-Ins (SPIs)ITDogadjaji.com
 
2012 regina TC 102 kanban
2012 regina TC 102 kanban2012 regina TC 102 kanban
2012 regina TC 102 kanbanAmir Barylko
 
Agile requirements
Agile requirementsAgile requirements
Agile requirementsAmir Barylko
 
YEG-Agile-planning
YEG-Agile-planningYEG-Agile-planning
YEG-Agile-planningAmir Barylko
 
Ralph jocham agile portfolio based release trains
Ralph jocham agile portfolio based release trainsRalph jocham agile portfolio based release trains
Ralph jocham agile portfolio based release trainsAgora Group
 

More Related Content

Similar to Arbyte - A modular, flexible, scalable job queing and execution system

Alfresco day madrid jeff potts - activiti
Alfresco day madrid   jeff potts - activitiAlfresco day madrid   jeff potts - activiti
Alfresco day madrid jeff potts - activitiAlfresco Software
 
Alfresco Day Madrid - Jeff Potts - Activiti
Alfresco Day Madrid - Jeff Potts - ActivitiAlfresco Day Madrid - Jeff Potts - Activiti
Alfresco Day Madrid - Jeff Potts - ActivitiToni de la Fuente
 
Activiti in Action (Devoxx 2010)
Activiti in Action (Devoxx 2010)Activiti in Action (Devoxx 2010)
Activiti in Action (Devoxx 2010)Joram Barrez
 
Developing for Remote Bamboo Agents, AtlasCamp US 2012
Developing for Remote Bamboo Agents, AtlasCamp US 2012Developing for Remote Bamboo Agents, AtlasCamp US 2012
Developing for Remote Bamboo Agents, AtlasCamp US 2012Atlassian
 
Workflow bis17
Workflow bis17Workflow bis17
Workflow bis17sakpob
 
Agile labs 2011
Agile labs   2011Agile labs   2011
Agile labs 2011vpraghu
 
XPages is Workflow's new best friend
XPages is Workflow's new best friendXPages is Workflow's new best friend
XPages is Workflow's new best friendStephan H. Wissel
 
Improving UX through Application Lifecycle Management
Improving UX through Application Lifecycle ManagementImproving UX through Application Lifecycle Management
Improving UX through Application Lifecycle Managementgoodfriday
 
Are good SharePoint solutions only a myth?
Are good SharePoint solutions only a myth?Are good SharePoint solutions only a myth?
Are good SharePoint solutions only a myth?Adis Jugo
 
Hadoop World 2011: Proven Tools to Manage Hadoop Environments - Joey Jablonsk...
Hadoop World 2011: Proven Tools to Manage Hadoop Environments - Joey Jablonsk...Hadoop World 2011: Proven Tools to Manage Hadoop Environments - Joey Jablonsk...
Hadoop World 2011: Proven Tools to Manage Hadoop Environments - Joey Jablonsk...Cloudera, Inc.
 
The Web Development Eco-system with VSTS, ASP.NET 2.0 & Microsoft Ajax
The Web Development Eco-system with VSTS, ASP.NET 2.0 & Microsoft AjaxThe Web Development Eco-system with VSTS, ASP.NET 2.0 & Microsoft Ajax
The Web Development Eco-system with VSTS, ASP.NET 2.0 & Microsoft AjaxDarren Sim
 
2009 11-04 mm (carson, california - csu-dh) bpm introduction
2009 11-04 mm (carson, california - csu-dh) bpm introduction2009 11-04 mm (carson, california - csu-dh) bpm introduction
2009 11-04 mm (carson, california - csu-dh) bpm introductionMike Marin
 
Att lyckas med integration av arbetet från flera scrum team - Christophe Acho...
Att lyckas med integration av arbetet från flera scrum team - Christophe Acho...Att lyckas med integration av arbetet från flera scrum team - Christophe Acho...
Att lyckas med integration av arbetet från flera scrum team - Christophe Acho...manssandstrom
 
Introduction to Ember.js and how we used it at FlowPro.io
Introduction to Ember.js and how we used it at FlowPro.ioIntroduction to Ember.js and how we used it at FlowPro.io
Introduction to Ember.js and how we used it at FlowPro.ioPaul Knittel
 
RESTful Work Items: Opening up Collaborative ALM
RESTful Work Items: Opening up Collaborative ALMRESTful Work Items: Opening up Collaborative ALM
RESTful Work Items: Opening up Collaborative ALMoslc
 
Intro to Drools - St Louis Gateway JUG
Intro to Drools - St Louis Gateway JUGIntro to Drools - St Louis Gateway JUG
Intro to Drools - St Louis Gateway JUGRay Ploski
 
Batching and Java EE (jdk.io)
Batching and Java EE (jdk.io)Batching and Java EE (jdk.io)
Batching and Java EE (jdk.io)Ryan Cuprak
 
Simon brown archsummit2012 the-frustrated-architect
Simon brown archsummit2012 the-frustrated-architectSimon brown archsummit2012 the-frustrated-architect
Simon brown archsummit2012 the-frustrated-architectdrewz lin
 
Ejb course in-mumbai
Ejb course in-mumbaiEjb course in-mumbai
Ejb course in-mumbaivibrantuser
 

Similar to Arbyte - A modular, flexible, scalable job queing and execution system (20)

Alfresco day madrid jeff potts - activiti
Alfresco day madrid   jeff potts - activitiAlfresco day madrid   jeff potts - activiti
Alfresco day madrid jeff potts - activiti
 
Alfresco Day Madrid - Jeff Potts - Activiti
Alfresco Day Madrid - Jeff Potts - ActivitiAlfresco Day Madrid - Jeff Potts - Activiti
Alfresco Day Madrid - Jeff Potts - Activiti
 
Activiti in Action (Devoxx 2010)
Activiti in Action (Devoxx 2010)Activiti in Action (Devoxx 2010)
Activiti in Action (Devoxx 2010)
 
Developing for Remote Bamboo Agents, AtlasCamp US 2012
Developing for Remote Bamboo Agents, AtlasCamp US 2012Developing for Remote Bamboo Agents, AtlasCamp US 2012
Developing for Remote Bamboo Agents, AtlasCamp US 2012
 
Workflow bis17
Workflow bis17Workflow bis17
Workflow bis17
 
Agile labs 2011
Agile labs   2011Agile labs   2011
Agile labs 2011
 
XPages is Workflow's new best friend
XPages is Workflow's new best friendXPages is Workflow's new best friend
XPages is Workflow's new best friend
 
Improving UX through Application Lifecycle Management
Improving UX through Application Lifecycle ManagementImproving UX through Application Lifecycle Management
Improving UX through Application Lifecycle Management
 
Introducing spring
Introducing springIntroducing spring
Introducing spring
 
Are good SharePoint solutions only a myth?
Are good SharePoint solutions only a myth?Are good SharePoint solutions only a myth?
Are good SharePoint solutions only a myth?
 
Hadoop World 2011: Proven Tools to Manage Hadoop Environments - Joey Jablonsk...
Hadoop World 2011: Proven Tools to Manage Hadoop Environments - Joey Jablonsk...Hadoop World 2011: Proven Tools to Manage Hadoop Environments - Joey Jablonsk...
Hadoop World 2011: Proven Tools to Manage Hadoop Environments - Joey Jablonsk...
 
The Web Development Eco-system with VSTS, ASP.NET 2.0 & Microsoft Ajax
The Web Development Eco-system with VSTS, ASP.NET 2.0 & Microsoft AjaxThe Web Development Eco-system with VSTS, ASP.NET 2.0 & Microsoft Ajax
The Web Development Eco-system with VSTS, ASP.NET 2.0 & Microsoft Ajax
 
2009 11-04 mm (carson, california - csu-dh) bpm introduction
2009 11-04 mm (carson, california - csu-dh) bpm introduction2009 11-04 mm (carson, california - csu-dh) bpm introduction
2009 11-04 mm (carson, california - csu-dh) bpm introduction
 
Att lyckas med integration av arbetet från flera scrum team - Christophe Acho...
Att lyckas med integration av arbetet från flera scrum team - Christophe Acho...Att lyckas med integration av arbetet från flera scrum team - Christophe Acho...
Att lyckas med integration av arbetet från flera scrum team - Christophe Acho...
 
Introduction to Ember.js and how we used it at FlowPro.io
Introduction to Ember.js and how we used it at FlowPro.ioIntroduction to Ember.js and how we used it at FlowPro.io
Introduction to Ember.js and how we used it at FlowPro.io
 
RESTful Work Items: Opening up Collaborative ALM
RESTful Work Items: Opening up Collaborative ALMRESTful Work Items: Opening up Collaborative ALM
RESTful Work Items: Opening up Collaborative ALM
 
Intro to Drools - St Louis Gateway JUG
Intro to Drools - St Louis Gateway JUGIntro to Drools - St Louis Gateway JUG
Intro to Drools - St Louis Gateway JUG
 
Batching and Java EE (jdk.io)
Batching and Java EE (jdk.io)Batching and Java EE (jdk.io)
Batching and Java EE (jdk.io)
 
Simon brown archsummit2012 the-frustrated-architect
Simon brown archsummit2012 the-frustrated-architectSimon brown archsummit2012 the-frustrated-architect
Simon brown archsummit2012 the-frustrated-architect
 
Ejb course in-mumbai
Ejb course in-mumbaiEjb course in-mumbai
Ejb course in-mumbai
 

More from lokku

Geocoding Overview
Geocoding OverviewGeocoding Overview
Geocoding Overviewlokku
 
OpenCage Data and sustainable business models for open data
OpenCage Data and sustainable business models for open data OpenCage Data and sustainable business models for open data
OpenCage Data and sustainable business models for open data lokku
 
Presenting the OpenCage Geocoder at #londonapi 17 Sept 2014
Presenting the OpenCage Geocoder at #londonapi 17 Sept 2014Presenting the OpenCage Geocoder at #londonapi 17 Sept 2014
Presenting the OpenCage Geocoder at #londonapi 17 Sept 2014lokku
 
A living hell - lessons learned in eight years of parsing real estate data
A living hell - lessons learned in eight years of parsing real estate data  A living hell - lessons learned in eight years of parsing real estate data
A living hell - lessons learned in eight years of parsing real estate data lokku
 
Geo-search-location-based-results-for-site-search
Geo-search-location-based-results-for-site-searchGeo-search-location-based-results-for-site-search
Geo-search-location-based-results-for-site-searchlokku
 
Geocoding India - talk delivered on 31 Jan 2014 at the Bangalore goeBLR event
Geocoding India - talk delivered on 31 Jan 2014 at the Bangalore goeBLR eventGeocoding India - talk delivered on 31 Jan 2014 at the Bangalore goeBLR event
Geocoding India - talk delivered on 31 Jan 2014 at the Bangalore goeBLR eventlokku
 
Nestoria new design
Nestoria new designNestoria new design
Nestoria new designlokku
 
CSS::SpriteMaker in action!
CSS::SpriteMaker in action!CSS::SpriteMaker in action!
CSS::SpriteMaker in action!lokku
 
Reducing the technical hurdle - why we started OpenCage Data
Reducing the technical hurdle - why we started OpenCage DataReducing the technical hurdle - why we started OpenCage Data
Reducing the technical hurdle - why we started OpenCage Datalokku
 
Css sprite_maker-1
Css  sprite_maker-1Css  sprite_maker-1
Css sprite_maker-1lokku
 
Nestoria case study - The effective use of geo-data for search marketing
Nestoria case study - The effective use of geo-data for search marketingNestoria case study - The effective use of geo-data for search marketing
Nestoria case study - The effective use of geo-data for search marketinglokku
 
The Nestoria GeoChallenge
The Nestoria GeoChallengeThe Nestoria GeoChallenge
The Nestoria GeoChallengelokku
 
Geo-Data for Search Marketing SEM & SEO
Geo-Data for Search Marketing SEM & SEOGeo-Data for Search Marketing SEM & SEO
Geo-Data for Search Marketing SEM & SEOlokku
 
Making using OSM data simpler - OpenCage Data
Making using OSM data simpler - OpenCage Data Making using OSM data simpler - OpenCage Data
Making using OSM data simpler - OpenCage Data lokku
 
What’s next in mapping for portals? ppw2012
What’s next in mapping for portals? ppw2012What’s next in mapping for portals? ppw2012
What’s next in mapping for portals? ppw2012lokku
 
How Nestoria switched to OpenStreetMap maps
How Nestoria switched to OpenStreetMap mapsHow Nestoria switched to OpenStreetMap maps
How Nestoria switched to OpenStreetMap mapslokku
 
Remote Geocoding
Remote GeocodingRemote Geocoding
Remote Geocodinglokku
 
Lessons learned in doing lots with few people
Lessons learned in  doing lots with few peopleLessons learned in  doing lots with few people
Lessons learned in doing lots with few peoplelokku
 
Mapstraction
MapstractionMapstraction
Mapstractionlokku
 
Bar Camp London 7
Bar Camp London 7Bar Camp London 7
Bar Camp London 7lokku
 

More from lokku (20)

Geocoding Overview
Geocoding OverviewGeocoding Overview
Geocoding Overview
 
OpenCage Data and sustainable business models for open data
OpenCage Data and sustainable business models for open data OpenCage Data and sustainable business models for open data
OpenCage Data and sustainable business models for open data
 
Presenting the OpenCage Geocoder at #londonapi 17 Sept 2014
Presenting the OpenCage Geocoder at #londonapi 17 Sept 2014Presenting the OpenCage Geocoder at #londonapi 17 Sept 2014
Presenting the OpenCage Geocoder at #londonapi 17 Sept 2014
 
A living hell - lessons learned in eight years of parsing real estate data
A living hell - lessons learned in eight years of parsing real estate data  A living hell - lessons learned in eight years of parsing real estate data
A living hell - lessons learned in eight years of parsing real estate data
 
Geo-search-location-based-results-for-site-search
Geo-search-location-based-results-for-site-searchGeo-search-location-based-results-for-site-search
Geo-search-location-based-results-for-site-search
 
Geocoding India - talk delivered on 31 Jan 2014 at the Bangalore goeBLR event
Geocoding India - talk delivered on 31 Jan 2014 at the Bangalore goeBLR eventGeocoding India - talk delivered on 31 Jan 2014 at the Bangalore goeBLR event
Geocoding India - talk delivered on 31 Jan 2014 at the Bangalore goeBLR event
 
Nestoria new design
Nestoria new designNestoria new design
Nestoria new design
 
CSS::SpriteMaker in action!
CSS::SpriteMaker in action!CSS::SpriteMaker in action!
CSS::SpriteMaker in action!
 
Reducing the technical hurdle - why we started OpenCage Data
Reducing the technical hurdle - why we started OpenCage DataReducing the technical hurdle - why we started OpenCage Data
Reducing the technical hurdle - why we started OpenCage Data
 
Css sprite_maker-1
Css  sprite_maker-1Css  sprite_maker-1
Css sprite_maker-1
 
Nestoria case study - The effective use of geo-data for search marketing
Nestoria case study - The effective use of geo-data for search marketingNestoria case study - The effective use of geo-data for search marketing
Nestoria case study - The effective use of geo-data for search marketing
 
The Nestoria GeoChallenge
The Nestoria GeoChallengeThe Nestoria GeoChallenge
The Nestoria GeoChallenge
 
Geo-Data for Search Marketing SEM & SEO
Geo-Data for Search Marketing SEM & SEOGeo-Data for Search Marketing SEM & SEO
Geo-Data for Search Marketing SEM & SEO
 
Making using OSM data simpler - OpenCage Data
Making using OSM data simpler - OpenCage Data Making using OSM data simpler - OpenCage Data
Making using OSM data simpler - OpenCage Data
 
What’s next in mapping for portals? ppw2012
What’s next in mapping for portals? ppw2012What’s next in mapping for portals? ppw2012
What’s next in mapping for portals? ppw2012
 
How Nestoria switched to OpenStreetMap maps
How Nestoria switched to OpenStreetMap mapsHow Nestoria switched to OpenStreetMap maps
How Nestoria switched to OpenStreetMap maps
 
Remote Geocoding
Remote GeocodingRemote Geocoding
Remote Geocoding
 
Lessons learned in doing lots with few people
Lessons learned in  doing lots with few peopleLessons learned in  doing lots with few people
Lessons learned in doing lots with few people
 
Mapstraction
MapstractionMapstraction
Mapstraction
 
Bar Camp London 7
Bar Camp London 7Bar Camp London 7
Bar Camp London 7
 

Recently uploaded

From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsFrom Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsInflectra
 
The Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolThe Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolProduct School
 
How to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanHow to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanDatabarracks
 
Empowering Net-Zero: Digital Insights and Funding Opportunities for Industria...
Empowering Net-Zero: Digital Insights and Funding Opportunities for Industria...Empowering Net-Zero: Digital Insights and Funding Opportunities for Industria...
Empowering Net-Zero: Digital Insights and Funding Opportunities for Industria...IES VE
 
Enterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewEnterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewAshraf Fouad
 
Artificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeArtificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeJosh Gellers
 
Pragmatic UI testing with Compose Semantics.pdf
Pragmatic UI testing with Compose Semantics.pdfPragmatic UI testing with Compose Semantics.pdf
Pragmatic UI testing with Compose Semantics.pdfinfogdgmi
 
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxThe Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxNeo4j
 
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...ISPMAIndia
 
Building Bridges: Merging RPA Processes, UiPath Apps, and Data Service to bu...
Building Bridges:  Merging RPA Processes, UiPath Apps, and Data Service to bu...Building Bridges:  Merging RPA Processes, UiPath Apps, and Data Service to bu...
Building Bridges: Merging RPA Processes, UiPath Apps, and Data Service to bu...DianaGray10
 
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, GoogleISPMAIndia
 
"Testing of Helm Charts or There and Back Again", Yura Rochniak
"Testing of Helm Charts or There and Back Again", Yura Rochniak"Testing of Helm Charts or There and Back Again", Yura Rochniak
"Testing of Helm Charts or There and Back Again", Yura RochniakFwdays
 
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoRevolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoProduct School
 
Mind your App Footprint 🐾⚡️🌱 (@FlutterHeroes 2024)
Mind your App Footprint 🐾⚡️🌱 (@FlutterHeroes 2024)Mind your App Footprint 🐾⚡️🌱 (@FlutterHeroes 2024)
Mind your App Footprint 🐾⚡️🌱 (@FlutterHeroes 2024)François
 
Battle of React State Managers in frontend applications
Battle of React State Managers in frontend applicationsBattle of React State Managers in frontend applications
Battle of React State Managers in frontend applicationsEvangelia Mitsopoulou
 
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Product School
 
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...Product School
 
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...Product School
 
"AIRe - AI Reliability Engineering", Denys Vasyliev
"AIRe - AI Reliability Engineering", Denys Vasyliev"AIRe - AI Reliability Engineering", Denys Vasyliev
"AIRe - AI Reliability Engineering", Denys VasylievFwdays
 
"How we created an SRE team in Temabit as a part of FOZZY Group in conditions...
"How we created an SRE team in Temabit as a part of FOZZY Group in conditions..."How we created an SRE team in Temabit as a part of FOZZY Group in conditions...
"How we created an SRE team in Temabit as a part of FOZZY Group in conditions...Fwdays
 

Recently uploaded (20)

From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsFrom Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
 
The Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolThe Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product School
 
How to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanHow to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response Plan
 
Empowering Net-Zero: Digital Insights and Funding Opportunities for Industria...
Empowering Net-Zero: Digital Insights and Funding Opportunities for Industria...Empowering Net-Zero: Digital Insights and Funding Opportunities for Industria...
Empowering Net-Zero: Digital Insights and Funding Opportunities for Industria...
 
Enterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewEnterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book Review
 
Artificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeArtificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human Justice
 
Pragmatic UI testing with Compose Semantics.pdf
Pragmatic UI testing with Compose Semantics.pdfPragmatic UI testing with Compose Semantics.pdf
Pragmatic UI testing with Compose Semantics.pdf
 
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxThe Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
 
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
 
Building Bridges: Merging RPA Processes, UiPath Apps, and Data Service to bu...
Building Bridges:  Merging RPA Processes, UiPath Apps, and Data Service to bu...Building Bridges:  Merging RPA Processes, UiPath Apps, and Data Service to bu...
Building Bridges: Merging RPA Processes, UiPath Apps, and Data Service to bu...
 
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
 
"Testing of Helm Charts or There and Back Again", Yura Rochniak
"Testing of Helm Charts or There and Back Again", Yura Rochniak"Testing of Helm Charts or There and Back Again", Yura Rochniak
"Testing of Helm Charts or There and Back Again", Yura Rochniak
 
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoRevolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
 
Mind your App Footprint 🐾⚡️🌱 (@FlutterHeroes 2024)
Mind your App Footprint 🐾⚡️🌱 (@FlutterHeroes 2024)Mind your App Footprint 🐾⚡️🌱 (@FlutterHeroes 2024)
Mind your App Footprint 🐾⚡️🌱 (@FlutterHeroes 2024)
 
Battle of React State Managers in frontend applications
Battle of React State Managers in frontend applicationsBattle of React State Managers in frontend applications
Battle of React State Managers in frontend applications
 
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
 
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...
 
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
 
"AIRe - AI Reliability Engineering", Denys Vasyliev
"AIRe - AI Reliability Engineering", Denys Vasyliev"AIRe - AI Reliability Engineering", Denys Vasyliev
"AIRe - AI Reliability Engineering", Denys Vasyliev
 
"How we created an SRE team in Temabit as a part of FOZZY Group in conditions...
"How we created an SRE team in Temabit as a part of FOZZY Group in conditions..."How we created an SRE team in Temabit as a part of FOZZY Group in conditions...
"How we created an SRE team in Temabit as a part of FOZZY Group in conditions...
 

Arbyte - A modular, flexible, scalable job queing and execution system

  • 1. ARBYTE ETYBRA Alistair N. MacLeod anm@lokku.com ltd
  • 2. Arbyte Alistair N. MacLeod Motivation 1 Motivation Problem Requirements Existing Systems Arbyte Architecture 2 Architecture Component Diagram Design and Implementa- tion 3 Design and Implementation Objects Processes IPC Practicalities 4 Practicalities Deployment Project Status
  • 3. Introduction Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Arbyte - Job queuing and execution framework Architecture Component Required system to run jobs Diagram Design and Considered gearman, TheSchwartz . . . Implementa- tion Decided to create wrapper - Arbyte Objects Processes IPC Practicalities Deployment Project Status
  • 4. Requirements Arbyte Alistair N. MacLeod Motivation Problem Requirements Fully scalable Existing Systems Arbyte Modular Architecture Component Logging Diagram Design and Good reliability Implementa- tion Thor Compliance Objects Processes IPC Batching Practicalities Deployment Project Status
  • 5. Batching Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Job specific optimisations Component Diagram In main queue Design and Implementa- tion Objects Processes IPC Practicalities Deployment Project Status
  • 6. Distributed Computing Models Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Cluster Architecture Component Diagram Grid Design and Implementa- MapReduce tion Objects Processes IPC Practicalities Deployment Project Status
  • 7. Distributed Computing Models Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Cluster Architecture Component Diagram Grid Design and Implementa- MapReduce tion Objects Processes IPC Practicalities Deployment Project Status
  • 8. Gearman Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Limitations Features Arbyte Architecture Not reliable Has multiple manager / Component queuing daemons Diagram No retries Design and Scalable Implementa- It didn’t work when I tried No single point of tion Objects it failure Processes IPC Practicalities Deployment Project Status
  • 9. The Schwartz Arbyte Alistair N. MacLeod Motivation Problem Requirements Limitations Features Existing Systems Arbyte Single DB store - not Reliability Architecture Component easily scalable Diagram Design and No batching after Implementa- tion submission. Objects Processes Relational DB overhead IPC Practicalities Deployment Project Status
  • 10. Helios Arbyte Alistair N. MacLeod Motivation Layer over TheSchwartz Problem Requirements Existing Systems Arbyte Architecture Component Diagram Limitations Features Design and Implementa- Same as TheSchwartz Manages worker processes tion Objects Doesn’t add batching or Adds XML Job submission Processes IPC change the fundamental format and web interface Practicalities architecture Deployment Project Status
  • 11. Non-Perl Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Possible but not as good for hacking on, integrating Component Diagram components. Design and We mostly have perl skills. Implementa- tion Objects Processes IPC Practicalities Deployment Project Status
  • 12. Considered Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Torque Architecture Component Diagram Hadoop Design and Implementa- Dr. Queue tion Objects Processes IPC Practicalities Deployment Project Status
  • 13. Back to Arbyte Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Modular framework for job queuing and execution Architecture Component Flexible, Customisable Diagram Design and Can be used with many other systems Implementa- tion e.g. Gearman, with batching, reliability and retries Objects Processes IPC Practicalities Deployment Project Status
  • 14. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor
  • 15. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Responsibilities Storing Jobs Job Specific Optimisations Batching Priorities Notes Currently have JobBuffer::Simple
  • 16. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Responsibilities Logging Retries Basic load balancing Notes Only “active” component
  • 17. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Responsibilities Arrange for Job Execution Notes Consistent interface
  • 18. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Notes JobRunner::Simple is implemented Forks a helper process Others are examples (todo)
  • 19. Job Producers JobBuffer JobRunner Helios Manager Job Producers JobBuffer JobRunner Gearman Manager Job Producers JobBuffer JobRunner Simple Arbyte Boundary JobExecutor JobExecutor JobExecutor Responsibilities Run Job code Report success / failure Notes Classes correspond to Job classes
  • 20. Object Implementation: Options Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Homemade Component Diagram Moose Design and Implementa- tion Objects Processes IPC Practicalities Deployment Project Status
  • 21. Object Implementation: Choice Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Using homemade objects Architecture Component Diagram All hashes Design and Implementa- AUTOLOADed get and set methods tion Objects Processes IPC Practicalities Deployment Project Status
  • 22. Processes Arbyte Alistair N. MacLeod Motivation Problem Requirements No threads Existing Systems Arbyte JobBuffer Architecture Component JobRunner Diagram Will likely have own processes Design and Implementa- e.g. JobRunnerHelper tion Objects Manager Processes IPC StatusAccepter Practicalities Deployment Project Status
  • 23. IPC Requirements Arbyte Alistair N. MacLeod Motivation Wanted something with: Problem Requirements Easy way to serverify an object Existing Systems Arbyte Stub generation Architecture Component Diagram Parameter passing Design and Exceptions Implementa- tion Objects Timeouts Processes IPC Security Practicalities Deployment Garbage collection Project Status
  • 24. Remote Object System Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Object Oriented Design RMI like system Architecture Component Assumed RMI on CPAN (Ruby has it, DRb) but no Diagram Feel like fixing this? Design and Implementa- tion Had to make do Objects Processes IPC Practicalities Deployment Project Status
  • 25. IPC: Implementation Options Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Considered Arbyte Architecture GRID::Machine Component Diagram Distributed::Process Design and Implementa- RPC::Serialized tion Objects RCGI - RPC with CGI server Processes IPC Practicalities Deployment Project Status
  • 26. IPC: Implementation Choice Arbyte Alistair N. MacLeod Motivation Problem Event::RPC Requirements Existing Systems Closest to RMI Arbyte Architecture Maintained Component Diagram Has (some) timeouts Design and Implementa- Propagates Exceptions tion Objects Processes Confusing - capabilities not clear IPC Using some hackery to make it Good Enough Practicalities Deployment Project Status
  • 27. Deployment Hardware Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Own Servers Component Diagram Design and Cloud Implementa- tion Objects Processes IPC Practicalities Deployment Project Status
  • 28. Grid Management Software Arbyte Alistair N. MacLeod Motivation To Manage Problem Requirements Booting Existing Systems Arbyte Package distribution Architecture Component Diagram Configuration Design and For example Implementa- tion Objects RPMs Processes IPC Puppet Practicalities Deployment Wigwam Project Status
  • 29. Project Status Arbyte Alistair N. MacLeod Motivation Problem Now Requirements Existing Systems Running in parallel with production system Arbyte Architecture Todo Component Diagram Better JobBuffers Design and Implementa- Better JobRunners tion Objects Processes Worker capabilities? IPC Optimise Practicalities Deployment Project Status
  • 30. The Route to CPAN Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Object system Architecture Component Config system Diagram Design and High level documentation Implementa- tion More tests Objects Processes IPC Practicalities Deployment Project Status
  • 31. Questions Arbyte Alistair N. MacLeod Motivation Problem Requirements Existing Systems Arbyte Architecture Component Diagram Questions Design and Implementa- tion Objects Processes IPC Practicalities Deployment Project Status