SlideShare a Scribd company logo
Hierarchical Cluster Engine
Distributed Remote Command Execution
(DRCE)

v1.1 general description
and architecture basics

IOIX Ukraine 2014
DRCE subsystem
The DCRE functionality represent the way of distributed data
processing that includes the algorithms code populated to the
target low level nodes for execution.
Any kind of supported by target node OS and platform executable
binary, byte or source code can be used. Each request
represents task that can be scheduled different way for parallel
execution in load-balancing or shard mode.
Depends on cluster structural schema the shard or replica manager
node will handle the request task to one or to several down layer
nodes. The cluster architecture defines the planning of
distributed task request execution target node.
Heterogeneous HCE cluster architecture can be built to get a
complex planning strategy for tasks execution planning.
DRCE application
The Distributed Remote Common Execution as a service have
several main application areas for target end-user project
system. The concrete cluster architecture schema and DRCE
usage depends on task timing and size of income and outgoing
data for processing.
Small tasks execution effective parallelism. Usable for one host or
several physical hosts in pool. Because HCE node handlers are
single-threaded and connected with ZMQ socket that used in
buffered asynchronous mode effectiveness of small tasks with
time from 1 to 50 ms is near to physical platform possibility.
Middle tasks execution also effective parallelism and results data
reducing.
In simple mode the results data reducing includes only collecting the
data from several nodes in pool that is connected and managed by
one shard manager.
In more complex mode the resulted data reducing can to include some
custom data processing after collecting. The reducer's data
processing executable can be specified with request task parameters
for each level of cluster hierarchical structure. In result the outgoing
data will be transformed from set of parts from down level nodes
processing results to one item that is result of processing on next
level of cluster hierarchical structure.
HCE key functional principles
●

●

●

●

●

Free network cluster structure architecture. Target applied project can
construct specific schema of network relations that succeeds on
hardware, load-balancing, file-over, fault-tolerance and another
principles on the basis of one simple Lego-like engine.
Stable based on ZMQ sockets reversed client-server networking
protocol with connection heart-beating, automated restoration and
messages buffering.
Easy asynchronous connections handling with NUMA oriented
architecture of messages handlers.
Unified I/O messages based on json format.
Ready to have client APIs bindings for many programmer languages
covered by ZMQ library. Can be easily integrated and deployed.
HCE key functional principles
●

●

●

●

●

Free network cluster structure architecture. Target applied project can
construct specific schema of network relations that succeeds on
hardware, load-balancing, file-over, fault-tolerance and another
principles on the basis of one simple Lego-like engine.
Stable based on ZMQ sockets reversed client-server networking
protocol with connection heart-beating, automated restoration and
messages buffering.
Easy asynchronous connections handling with NUMA oriented
architecture of messages handlers.
Unified I/O messages based on json format.
Ready to have client APIs bindings for many programmer languages
covered by ZMQ library. Can be easily integrated and deployed.

More Related Content

What's hot

Logisim Ethernet MAC Address Reader(Final)
Logisim Ethernet MAC Address Reader(Final)Logisim Ethernet MAC Address Reader(Final)
Logisim Ethernet MAC Address Reader(Final)
Old Dominion University
 
Session 7 Tp 7
Session 7 Tp 7Session 7 Tp 7
Session 7 Tp 7
githe26200
 

What's hot (12)

Software defined network
Software defined network Software defined network
Software defined network
 
Scope of parallelism
Scope of parallelismScope of parallelism
Scope of parallelism
 
Dynamic Load balancing Linux private Cloud (DRS)
Dynamic Load balancing Linux private Cloud (DRS)Dynamic Load balancing Linux private Cloud (DRS)
Dynamic Load balancing Linux private Cloud (DRS)
 
Cloud computing
Cloud computingCloud computing
Cloud computing
 
Sqlnet
SqlnetSqlnet
Sqlnet
 
Sdn 소개서
Sdn 소개서Sdn 소개서
Sdn 소개서
 
Designing distributed systems
Designing distributed systemsDesigning distributed systems
Designing distributed systems
 
RPC in Smalltalk
 RPC in Smalltalk RPC in Smalltalk
RPC in Smalltalk
 
Logisim Ethernet MAC Address Reader(Final)
Logisim Ethernet MAC Address Reader(Final)Logisim Ethernet MAC Address Reader(Final)
Logisim Ethernet MAC Address Reader(Final)
 
Session 7 Tp 7
Session 7 Tp 7Session 7 Tp 7
Session 7 Tp 7
 
FP7 PACE PCE Tutorial
FP7 PACE PCE TutorialFP7 PACE PCE Tutorial
FP7 PACE PCE Tutorial
 
Cluster computings
Cluster computingsCluster computings
Cluster computings
 

Similar to HCE project Distributed Remote Command Execution

Performance evaluation of ecc in single and multi( eliptic curve)
Performance evaluation of ecc in single and multi( eliptic curve)Performance evaluation of ecc in single and multi( eliptic curve)
Performance evaluation of ecc in single and multi( eliptic curve)
Danilo Calle
 
Microx - A Unix like kernel for Embedded Systems written from scratch.
Microx - A Unix like kernel for Embedded Systems written from scratch.Microx - A Unix like kernel for Embedded Systems written from scratch.
Microx - A Unix like kernel for Embedded Systems written from scratch.
Waqar Sheikh
 
37248136-Nano-Technology.pdf
37248136-Nano-Technology.pdf37248136-Nano-Technology.pdf
37248136-Nano-Technology.pdf
TB107thippeswamyM
 

Similar to HCE project Distributed Remote Command Execution (20)

HCE project brief
HCE project briefHCE project brief
HCE project brief
 
Performance evaluation of ecc in single and multi( eliptic curve)
Performance evaluation of ecc in single and multi( eliptic curve)Performance evaluation of ecc in single and multi( eliptic curve)
Performance evaluation of ecc in single and multi( eliptic curve)
 
Public Cloud Workshop
Public Cloud WorkshopPublic Cloud Workshop
Public Cloud Workshop
 
Microx - A Unix like kernel for Embedded Systems written from scratch.
Microx - A Unix like kernel for Embedded Systems written from scratch.Microx - A Unix like kernel for Embedded Systems written from scratch.
Microx - A Unix like kernel for Embedded Systems written from scratch.
 
SDN, OpenFlow, NFV, and Virtual Network
SDN, OpenFlow, NFV, and Virtual NetworkSDN, OpenFlow, NFV, and Virtual Network
SDN, OpenFlow, NFV, and Virtual Network
 
Software Defined Networking: A Concept and Related Issues
Software Defined Networking: A Concept and Related IssuesSoftware Defined Networking: A Concept and Related Issues
Software Defined Networking: A Concept and Related Issues
 
IRJET- ALPYNE - A Grid Computing Framework
IRJET- ALPYNE - A Grid Computing FrameworkIRJET- ALPYNE - A Grid Computing Framework
IRJET- ALPYNE - A Grid Computing Framework
 
SDN Multi-Controller Domain.pptx
SDN Multi-Controller Domain.pptxSDN Multi-Controller Domain.pptx
SDN Multi-Controller Domain.pptx
 
journal of mathematics research
journal of mathematics researchjournal of mathematics research
journal of mathematics research
 
journalism research paper
journalism research paperjournalism research paper
journalism research paper
 
journal in research
journal in researchjournal in research
journal in research
 
journal to publish research paper
journal to publish research paperjournal to publish research paper
journal to publish research paper
 
research on journaling
research on journalingresearch on journaling
research on journaling
 
37248136-Nano-Technology.pdf
37248136-Nano-Technology.pdf37248136-Nano-Technology.pdf
37248136-Nano-Technology.pdf
 
Distributed Crawler Service architecture presentation
Distributed Crawler Service architecture presentationDistributed Crawler Service architecture presentation
Distributed Crawler Service architecture presentation
 
ONP 2.1 platforms maximize VNF interoperability
ONP 2.1 platforms maximize VNF interoperabilityONP 2.1 platforms maximize VNF interoperability
ONP 2.1 platforms maximize VNF interoperability
 
Edge Computing Platforms and Protocols - Ph.D. thesis
Edge Computing Platforms and Protocols - Ph.D. thesisEdge Computing Platforms and Protocols - Ph.D. thesis
Edge Computing Platforms and Protocols - Ph.D. thesis
 
TERM PAPER
TERM PAPERTERM PAPER
TERM PAPER
 
Architecture evolution for automation and network programmability
Architecture evolution for automation and network programmabilityArchitecture evolution for automation and network programmability
Architecture evolution for automation and network programmability
 
1.multicore processors
1.multicore processors1.multicore processors
1.multicore processors
 

Recently uploaded

Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
UXDXConf
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Peter Udo Diehl
 

Recently uploaded (20)

Server-Driven User Interface (SDUI) at Priceline
Server-Driven User Interface (SDUI) at PricelineServer-Driven User Interface (SDUI) at Priceline
Server-Driven User Interface (SDUI) at Priceline
 
Transforming The New York Times: Empowering Evolution through UX
Transforming The New York Times: Empowering Evolution through UXTransforming The New York Times: Empowering Evolution through UX
Transforming The New York Times: Empowering Evolution through UX
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
 
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCustom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
 
Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdf
 
The architecture of Generative AI for enterprises.pdf
The architecture of Generative AI for enterprises.pdfThe architecture of Generative AI for enterprises.pdf
The architecture of Generative AI for enterprises.pdf
 
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMsTo Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
Intelligent Gimbal FINAL PAPER Engineering.pdf
Intelligent Gimbal FINAL PAPER Engineering.pdfIntelligent Gimbal FINAL PAPER Engineering.pdf
Intelligent Gimbal FINAL PAPER Engineering.pdf
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
 

HCE project Distributed Remote Command Execution

  • 1. Hierarchical Cluster Engine Distributed Remote Command Execution (DRCE) v1.1 general description and architecture basics IOIX Ukraine 2014
  • 2. DRCE subsystem The DCRE functionality represent the way of distributed data processing that includes the algorithms code populated to the target low level nodes for execution. Any kind of supported by target node OS and platform executable binary, byte or source code can be used. Each request represents task that can be scheduled different way for parallel execution in load-balancing or shard mode. Depends on cluster structural schema the shard or replica manager node will handle the request task to one or to several down layer nodes. The cluster architecture defines the planning of distributed task request execution target node. Heterogeneous HCE cluster architecture can be built to get a complex planning strategy for tasks execution planning.
  • 3. DRCE application The Distributed Remote Common Execution as a service have several main application areas for target end-user project system. The concrete cluster architecture schema and DRCE usage depends on task timing and size of income and outgoing data for processing. Small tasks execution effective parallelism. Usable for one host or several physical hosts in pool. Because HCE node handlers are single-threaded and connected with ZMQ socket that used in buffered asynchronous mode effectiveness of small tasks with time from 1 to 50 ms is near to physical platform possibility.
  • 4. Middle tasks execution also effective parallelism and results data reducing. In simple mode the results data reducing includes only collecting the data from several nodes in pool that is connected and managed by one shard manager. In more complex mode the resulted data reducing can to include some custom data processing after collecting. The reducer's data processing executable can be specified with request task parameters for each level of cluster hierarchical structure. In result the outgoing data will be transformed from set of parts from down level nodes processing results to one item that is result of processing on next level of cluster hierarchical structure.
  • 5. HCE key functional principles ● ● ● ● ● Free network cluster structure architecture. Target applied project can construct specific schema of network relations that succeeds on hardware, load-balancing, file-over, fault-tolerance and another principles on the basis of one simple Lego-like engine. Stable based on ZMQ sockets reversed client-server networking protocol with connection heart-beating, automated restoration and messages buffering. Easy asynchronous connections handling with NUMA oriented architecture of messages handlers. Unified I/O messages based on json format. Ready to have client APIs bindings for many programmer languages covered by ZMQ library. Can be easily integrated and deployed.
  • 6. HCE key functional principles ● ● ● ● ● Free network cluster structure architecture. Target applied project can construct specific schema of network relations that succeeds on hardware, load-balancing, file-over, fault-tolerance and another principles on the basis of one simple Lego-like engine. Stable based on ZMQ sockets reversed client-server networking protocol with connection heart-beating, automated restoration and messages buffering. Easy asynchronous connections handling with NUMA oriented architecture of messages handlers. Unified I/O messages based on json format. Ready to have client APIs bindings for many programmer languages covered by ZMQ library. Can be easily integrated and deployed.