SlideShare a Scribd company logo
Hierarchical Cluster Engine
(HCE) project
v1.0 general description
and architecture basics

IOIX Ukraine 2013
HCE project aim and main idea
This project became the successor of Associative Search Machine
(ASM) full-text web search engine project that was developed
from 2006 to 2012 by IOIX Ukraine.
The main idea of this new project – to implement the solution that
can be used to: construct custom network mesh or distributed
network cluster structure with several relations types between
nodes, formalize the data flow processing goes from upper node
level central source point to down nodes and backward,
formalize the management requests handling from multiple
source points, support native reducing of multiple nodes results
(aggregation, duplicates elimination, sorting and so on), internally
support powerful full-text search engine and data storage,
provide transactions-less and transactional requests processing,
support flexible run-time changes of cluster infrastructure, have
many languages bindings for client-side integration APIs in one
product build on C++ language...
HCE application area
As a network infrastructure and messages transport layer provider –
the HCE can be used in any big-data solution that needs some
custom network structure to build distributed high-performance
easy scalable vertically and horizontally data processing or datamining architecture.
As a native internally supported full text search engine interface
provider – the HCE can be used in web or corporate network
solutions that needs smoothly integrated with usage of natural
target project specific languages, fast and powerful full text
search and NOSQL distributed data storage. Now the Sphinx (c)
search engine with extended data model internally supported.
AS a Distributed Remote Command Execution service provider –
the HCE can be used for automation of administration of many
host servers in ensemble mode for OS and services deployment,
maintenance and support tasks.
Hierarchical Cluster as engine
●

●

●

●

●

Provides hierarchical cluster infrastructure – nodes
connection schema, relations between nodes, roles of
nodes, requests typification and data processing
sequences algorithms, data sharding modes, and so on.
Provides network transport layer for data of client
application and administration management messages.
Manages native supported integrated NOSQL data storage
(Sphinx (c) search index and Distributed Remote Command Execution).
Collect, reduce and sort results of native and custom data
processing.
Ready to support transactional messages processing.
HCE key functional principles
●

●

●

●

●

Free network cluster structure architecture. Target applied project can
construct specific schema of network relations that succeeds on
hardware, load-balancing, file-over, fault-tolerance and another
principles on the basis of one simple Lego-like engine.
Stable based on ZMQ sockets reversed client-server networking
protocol with connection heart-beating, automated restoration and
messages buffering.
Easy asynchronous connections handling with NUMA oriented
architecture of messages handlers.
Unified I/O messages based on json format.
Ready to have client APIs bindings for many programmer languages
covered by ZMQ library. Can be easily integrated and deployed.
HCE-node application
The heart and main component of the HCE project it is hce-node
application. This application integrates complete set of base
functionality to support network infrastructure, hierarchical cluster
construction, full-text search system integration and so on, see
“Hierarchical Cluster as engine” main points.
Implemented for Linux OS environment and distributed in form of
source code tarball archive and Debian Linux binary package with
dependencies packages.
Supports single instance configuration-less start or requires set of
options that used to build correspondent network cluster
architecture.
Supposes usage with client-side applications or integrated IPI.
First implementation of client-side API and cli utilities bind on PHP.
Hce-node roles in the cluster
structure
Internally HCE-node application contains seven basic handler threads. Each handler
acts as special black-box messages processor/dispatcher and used in
combination with other to work in one of five different roles of node:
●

●

●

●

Router – upper end-point of cluster hierarchy. Has three server-type connections.
Handles client API, any kind of another node roles instances (typically, shard or
replica managers) and admin connections.
Shard manager – intermediate-point of cluster hierarchy. Routes messages between
upper and down layers. Uses data sharding and messages multicast dispatching
algorithms. Has two server-type and one client connections.
Replica manager – the same as shard manager. Routes messages between upper and
down layers uses data balancing and messages round-robin algorithms.
Replica – down end-point of cluster hierarchy. Data node, interacts with data storage
and/or process data with target algorithm(s), provides interface with fill-text search
engine, target host for Distributed Remote Commands Execution. Has one serverand one client-side connections used for cluster infrastructure Also can to have
several data storage-dependent connections.
hce-node typical connection points
Hce-node internal architecture
seven handlers objects in relations
hce-node in cluster structure relations
Simple cluster structures comparison
Hierarchical cluster structure example
This is just one
example of four
level cluster
structure. This
structure can be
used to create four
hosts distributed
data model
solution to split
one huge
documents storage
on three physical
host servers and
utilize all CPU and
I/O resources in
effective way.
Each level uses different data distribution modes (shard or replica). Two replica nodes of
each shard balances requests and have complete mirrored data. Shards are multicasted and
requests results accumulated and reduced by shard manager node.
Extended Sphinx index support
- Smooth transparent aggregation of several different indexes and search
algorithms in federated search.
- Native support by file operation the run-time indexes attach, copy,
replace, swap, mirror, merge.
- Native support of indexes sharding and mirroring according different data
schema.
- Extended full text search queries including full match, pattern match, set
of fields match, logical operators, proximity limits, stop-words lists and
many other provided by regular Sphinx search (c) engine.
- Filters for numeric fields including exact match and range comparisons.
- Multi-value numeric fields, bit-set and logical operations simulation in
search query.
- Custom order including Sphinx full text search weight and custom
weightier calculation algorithms definitions like SQL ORDER BY
prioritized fields list or more complex logic.
HCE Sphinx Index Extension - SIE
hce-node application provides solution named HCE Sphinx Index Extension
(HCE SIE) that add to the Sphinx index nature more functionality and two
high level structures:

Index – local FS folder contains complete set of sub-folders and files
including native Sphinx indexes (trunk and branches), configuration
files for searchd, source xml data files of branches, prepared xml
data files of branches, schema xml file and other...
Index supports operations: create, copy, rename, delete, start, stop,
rebuild, merge branches, delete document, connect, and other...
Branch – local FS files like source xml data file, prepared xml data file
and native Sphinx Index files created from prepared xml data file. All
branches of index uses the same schema and represents some part
of index. Set of branches can be merged in trunk that is used in
search process. Branch supports operations: create, delete, rebuild,
insert document, delete document and other...
HCE SIE folder structure
HCE technologies basics
●

●

●

●

●

ZMQ (c) sockets for inter-process and in-process interactions.
POCO C++ framework (c) for application architecture and
algorithms including json and xml parsers, objects
serialization, configuration settings and many other.
Sphinx search (c) engine client native integration with
extensions.
Mutex-less multithread application architecture and messages
handling.
Strict C++ OOP design, including elements of 11 standard.
HCE distribution and deployment
●

●

Open source distribution in source code of tarball archive.
Binary Debian Linux native packages separated on HCE-node
and utility packages as well as complete full meta-package.
Dependencies libraries packaging and maintenance to
complete cover dependencies and make installation smooth.

●

Public repositories accessible for contributors community.

●

Upgrade dependency packages and libraries as part of product.

●

●

Box ready to start distributive with Demo Test Suit (DTS) with
examples of simple cluster structure to start full text Sphinxbased search right after installation.
Open documentation.
HCE project todo...
- Additions and extensions to support transparent
configurable transactional model for all kind of actions.
- Extension of data node with native support of another index
types and data storage like SQLite.
- Extensions with set of API and utility tools for creating
typical cluster structures and sharding models.
- Extensions with set of API for general management and
tracking of statistics, including web-UI for state
visualization and visual management.
- Extensions for statistics and logging data analysis and
visualization.

More Related Content

Viewers also liked

p selling 1st Place Corporate Written
p selling 1st Place Corporate Written p selling 1st Place Corporate Written
p selling 1st Place Corporate Written
Jordan Idleman
 
Market Insertion and Corporate Selling
Market Insertion and Corporate SellingMarket Insertion and Corporate Selling
Market Insertion and Corporate Selling
moonchild1
 
Corporate Personal Selling
Corporate Personal SellingCorporate Personal Selling
Corporate Personal Selling
Anis Zuberi
 
“Corporate selling and feedback a project report on “Corporate selling and fe...
“Corporate selling and feedback a project report on “Corporate selling and fe...“Corporate selling and feedback a project report on “Corporate selling and fe...
“Corporate selling and feedback a project report on “Corporate selling and fe...
Babasab Patil
 
SLC_Selling Offline Airlines to Corporate - SAA_France
SLC_Selling Offline Airlines to Corporate - SAA_FranceSLC_Selling Offline Airlines to Corporate - SAA_France
SLC_Selling Offline Airlines to Corporate - SAA_France
Natalia Ignatenko
 
A Corporate approach to Social selling
A Corporate approach to Social sellingA Corporate approach to Social selling
A Corporate approach to Social selling
Jordi Gili
 
The 8 step personal selling process. process of selling
The 8 step personal selling process. process of sellingThe 8 step personal selling process. process of selling
The 8 step personal selling process. process of selling
Aaron Dlamini
 
A presentation on personal selling
A presentation on personal sellingA presentation on personal selling
A presentation on personal selling
Pranav Kumar Ojha
 
PPT on Personal selling
PPT on Personal sellingPPT on Personal selling
PPT on Personal selling
Prateek Mishra
 

Viewers also liked (9)

p selling 1st Place Corporate Written
p selling 1st Place Corporate Written p selling 1st Place Corporate Written
p selling 1st Place Corporate Written
 
Market Insertion and Corporate Selling
Market Insertion and Corporate SellingMarket Insertion and Corporate Selling
Market Insertion and Corporate Selling
 
Corporate Personal Selling
Corporate Personal SellingCorporate Personal Selling
Corporate Personal Selling
 
“Corporate selling and feedback a project report on “Corporate selling and fe...
“Corporate selling and feedback a project report on “Corporate selling and fe...“Corporate selling and feedback a project report on “Corporate selling and fe...
“Corporate selling and feedback a project report on “Corporate selling and fe...
 
SLC_Selling Offline Airlines to Corporate - SAA_France
SLC_Selling Offline Airlines to Corporate - SAA_FranceSLC_Selling Offline Airlines to Corporate - SAA_France
SLC_Selling Offline Airlines to Corporate - SAA_France
 
A Corporate approach to Social selling
A Corporate approach to Social sellingA Corporate approach to Social selling
A Corporate approach to Social selling
 
The 8 step personal selling process. process of selling
The 8 step personal selling process. process of sellingThe 8 step personal selling process. process of selling
The 8 step personal selling process. process of selling
 
A presentation on personal selling
A presentation on personal sellingA presentation on personal selling
A presentation on personal selling
 
PPT on Personal selling
PPT on Personal sellingPPT on Personal selling
PPT on Personal selling
 

Similar to HCE project brief

Distributed Crawler Service architecture presentation
Distributed Crawler Service architecture presentationDistributed Crawler Service architecture presentation
Distributed Crawler Service architecture presentation
Gennady Baranov
 
Building Killer RESTful APIs with NodeJs
Building Killer RESTful APIs with NodeJsBuilding Killer RESTful APIs with NodeJs
Building Killer RESTful APIs with NodeJs
Srdjan Strbanovic
 
HCE project Distributed Remote Command Execution
HCE project Distributed Remote Command ExecutionHCE project Distributed Remote Command Execution
HCE project Distributed Remote Command Execution
Gennady Baranov
 
Hpcc
HpccHpcc
TOLL MANAGEMENT SYSTEM
TOLL MANAGEMENT SYSTEMTOLL MANAGEMENT SYSTEM
TOLL MANAGEMENT SYSTEM
vishnuRajan20
 
Toll management system (1) (1)
Toll management system (1) (1)Toll management system (1) (1)
Toll management system (1) (1)
vishnuRajan20
 
Ravi Namboori 's Open stack framework introduction
Ravi Namboori 's Open stack framework introductionRavi Namboori 's Open stack framework introduction
Ravi Namboori 's Open stack framework introduction
Ravi namboori
 
A Quick Survey of Open Source Software for PH Organizations, a paper by Massi...
A Quick Survey of Open Source Software for PH Organizations, a paper by Massi...A Quick Survey of Open Source Software for PH Organizations, a paper by Massi...
A Quick Survey of Open Source Software for PH Organizations, a paper by Massi...
InSTEDD
 
Adopting AnswerModules ModuleSuite
Adopting AnswerModules ModuleSuiteAdopting AnswerModules ModuleSuite
Adopting AnswerModules ModuleSuite
AnswerModules
 
Public Cloud Workshop
Public Cloud WorkshopPublic Cloud Workshop
Public Cloud Workshop
Amer Ather
 
Hadoop Integration with Microstrategy
Hadoop Integration with Microstrategy Hadoop Integration with Microstrategy
Hadoop Integration with Microstrategy
snehal parikh
 
Net Fundamentals
Net FundamentalsNet Fundamentals
Net Fundamentals
Ali Taki
 
Oslo bekk2014
Oslo bekk2014Oslo bekk2014
Oslo bekk2014
Max Neunhöffer
 
In15orlesss hadoop
In15orlesss hadoopIn15orlesss hadoop
In15orlesss hadoop
Worapol Alex Pongpech, PhD
 
Big data processing using HPCC Systems Above and Beyond Hadoop
Big data processing using HPCC Systems Above and Beyond HadoopBig data processing using HPCC Systems Above and Beyond Hadoop
Big data processing using HPCC Systems Above and Beyond Hadoop
HPCC Systems
 
The Why and How of HPC-Cloud Hybrids with OpenStack - Lev Lafayette, Universi...
The Why and How of HPC-Cloud Hybrids with OpenStack - Lev Lafayette, Universi...The Why and How of HPC-Cloud Hybrids with OpenStack - Lev Lafayette, Universi...
The Why and How of HPC-Cloud Hybrids with OpenStack - Lev Lafayette, Universi...
OpenStack
 
Fyp presentation 2 (SQL Converter)
Fyp presentation 2 (SQL Converter)Fyp presentation 2 (SQL Converter)
Fyp presentation 2 (SQL Converter)
Muhammad Shafiq
 
Tycs sem 5 asp.net notes unit 1 2 3 4 (2017)
Tycs sem 5 asp.net notes unit 1 2 3 4 (2017)Tycs sem 5 asp.net notes unit 1 2 3 4 (2017)
Tycs sem 5 asp.net notes unit 1 2 3 4 (2017)
WE-IT TUTORIALS
 
Deploying and Managing Artificial Intelligence Services using the Open Data H...
Deploying and Managing Artificial Intelligence Services using the Open Data H...Deploying and Managing Artificial Intelligence Services using the Open Data H...
Deploying and Managing Artificial Intelligence Services using the Open Data H...
Orgad Kimchi
 
Apache frameworks for Big and Fast Data
Apache frameworks for Big and Fast DataApache frameworks for Big and Fast Data
Apache frameworks for Big and Fast Data
Naveen Korakoppa
 

Similar to HCE project brief (20)

Distributed Crawler Service architecture presentation
Distributed Crawler Service architecture presentationDistributed Crawler Service architecture presentation
Distributed Crawler Service architecture presentation
 
Building Killer RESTful APIs with NodeJs
Building Killer RESTful APIs with NodeJsBuilding Killer RESTful APIs with NodeJs
Building Killer RESTful APIs with NodeJs
 
HCE project Distributed Remote Command Execution
HCE project Distributed Remote Command ExecutionHCE project Distributed Remote Command Execution
HCE project Distributed Remote Command Execution
 
Hpcc
HpccHpcc
Hpcc
 
TOLL MANAGEMENT SYSTEM
TOLL MANAGEMENT SYSTEMTOLL MANAGEMENT SYSTEM
TOLL MANAGEMENT SYSTEM
 
Toll management system (1) (1)
Toll management system (1) (1)Toll management system (1) (1)
Toll management system (1) (1)
 
Ravi Namboori 's Open stack framework introduction
Ravi Namboori 's Open stack framework introductionRavi Namboori 's Open stack framework introduction
Ravi Namboori 's Open stack framework introduction
 
A Quick Survey of Open Source Software for PH Organizations, a paper by Massi...
A Quick Survey of Open Source Software for PH Organizations, a paper by Massi...A Quick Survey of Open Source Software for PH Organizations, a paper by Massi...
A Quick Survey of Open Source Software for PH Organizations, a paper by Massi...
 
Adopting AnswerModules ModuleSuite
Adopting AnswerModules ModuleSuiteAdopting AnswerModules ModuleSuite
Adopting AnswerModules ModuleSuite
 
Public Cloud Workshop
Public Cloud WorkshopPublic Cloud Workshop
Public Cloud Workshop
 
Hadoop Integration with Microstrategy
Hadoop Integration with Microstrategy Hadoop Integration with Microstrategy
Hadoop Integration with Microstrategy
 
Net Fundamentals
Net FundamentalsNet Fundamentals
Net Fundamentals
 
Oslo bekk2014
Oslo bekk2014Oslo bekk2014
Oslo bekk2014
 
In15orlesss hadoop
In15orlesss hadoopIn15orlesss hadoop
In15orlesss hadoop
 
Big data processing using HPCC Systems Above and Beyond Hadoop
Big data processing using HPCC Systems Above and Beyond HadoopBig data processing using HPCC Systems Above and Beyond Hadoop
Big data processing using HPCC Systems Above and Beyond Hadoop
 
The Why and How of HPC-Cloud Hybrids with OpenStack - Lev Lafayette, Universi...
The Why and How of HPC-Cloud Hybrids with OpenStack - Lev Lafayette, Universi...The Why and How of HPC-Cloud Hybrids with OpenStack - Lev Lafayette, Universi...
The Why and How of HPC-Cloud Hybrids with OpenStack - Lev Lafayette, Universi...
 
Fyp presentation 2 (SQL Converter)
Fyp presentation 2 (SQL Converter)Fyp presentation 2 (SQL Converter)
Fyp presentation 2 (SQL Converter)
 
Tycs sem 5 asp.net notes unit 1 2 3 4 (2017)
Tycs sem 5 asp.net notes unit 1 2 3 4 (2017)Tycs sem 5 asp.net notes unit 1 2 3 4 (2017)
Tycs sem 5 asp.net notes unit 1 2 3 4 (2017)
 
Deploying and Managing Artificial Intelligence Services using the Open Data H...
Deploying and Managing Artificial Intelligence Services using the Open Data H...Deploying and Managing Artificial Intelligence Services using the Open Data H...
Deploying and Managing Artificial Intelligence Services using the Open Data H...
 
Apache frameworks for Big and Fast Data
Apache frameworks for Big and Fast DataApache frameworks for Big and Fast Data
Apache frameworks for Big and Fast Data
 

Recently uploaded

Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 
Discover the Unseen: Tailored Recommendation of Unwatched Content
Discover the Unseen: Tailored Recommendation of Unwatched ContentDiscover the Unseen: Tailored Recommendation of Unwatched Content
Discover the Unseen: Tailored Recommendation of Unwatched Content
ScyllaDB
 
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
AlexanderRichford
 
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
Safe Software
 
GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...
GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...
GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...
GlobalLogic Ukraine
 
A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
ScyllaDB
 
Christine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptxChristine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptx
christinelarrosa
 
Containers & AI - Beauty and the Beast!?!
Containers & AI - Beauty and the Beast!?!Containers & AI - Beauty and the Beast!?!
Containers & AI - Beauty and the Beast!?!
Tobias Schneck
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
Fwdays
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
Fwdays
 
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham HillinQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
LizaNolte
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
DianaGray10
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
DanBrown980551
 
Introducing BoxLang : A new JVM language for productivity and modularity!
Introducing BoxLang : A new JVM language for productivity and modularity!Introducing BoxLang : A new JVM language for productivity and modularity!
Introducing BoxLang : A new JVM language for productivity and modularity!
Ortus Solutions, Corp
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
AstuteBusiness
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving
 
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptxPrinciple of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
BibashShahi
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
c5vrf27qcz
 
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during MigrationMust Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
Mydbops
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Neo4j
 

Recently uploaded (20)

Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 
Discover the Unseen: Tailored Recommendation of Unwatched Content
Discover the Unseen: Tailored Recommendation of Unwatched ContentDiscover the Unseen: Tailored Recommendation of Unwatched Content
Discover the Unseen: Tailored Recommendation of Unwatched Content
 
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
 
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
 
GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...
GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...
GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...
 
A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
 
Christine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptxChristine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptx
 
Containers & AI - Beauty and the Beast!?!
Containers & AI - Beauty and the Beast!?!Containers & AI - Beauty and the Beast!?!
Containers & AI - Beauty and the Beast!?!
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
 
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham HillinQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
 
Introducing BoxLang : A new JVM language for productivity and modularity!
Introducing BoxLang : A new JVM language for productivity and modularity!Introducing BoxLang : A new JVM language for productivity and modularity!
Introducing BoxLang : A new JVM language for productivity and modularity!
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
 
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptxPrinciple of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
 
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during MigrationMust Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
 

HCE project brief

  • 1. Hierarchical Cluster Engine (HCE) project v1.0 general description and architecture basics IOIX Ukraine 2013
  • 2. HCE project aim and main idea This project became the successor of Associative Search Machine (ASM) full-text web search engine project that was developed from 2006 to 2012 by IOIX Ukraine. The main idea of this new project – to implement the solution that can be used to: construct custom network mesh or distributed network cluster structure with several relations types between nodes, formalize the data flow processing goes from upper node level central source point to down nodes and backward, formalize the management requests handling from multiple source points, support native reducing of multiple nodes results (aggregation, duplicates elimination, sorting and so on), internally support powerful full-text search engine and data storage, provide transactions-less and transactional requests processing, support flexible run-time changes of cluster infrastructure, have many languages bindings for client-side integration APIs in one product build on C++ language...
  • 3. HCE application area As a network infrastructure and messages transport layer provider – the HCE can be used in any big-data solution that needs some custom network structure to build distributed high-performance easy scalable vertically and horizontally data processing or datamining architecture. As a native internally supported full text search engine interface provider – the HCE can be used in web or corporate network solutions that needs smoothly integrated with usage of natural target project specific languages, fast and powerful full text search and NOSQL distributed data storage. Now the Sphinx (c) search engine with extended data model internally supported. AS a Distributed Remote Command Execution service provider – the HCE can be used for automation of administration of many host servers in ensemble mode for OS and services deployment, maintenance and support tasks.
  • 4. Hierarchical Cluster as engine ● ● ● ● ● Provides hierarchical cluster infrastructure – nodes connection schema, relations between nodes, roles of nodes, requests typification and data processing sequences algorithms, data sharding modes, and so on. Provides network transport layer for data of client application and administration management messages. Manages native supported integrated NOSQL data storage (Sphinx (c) search index and Distributed Remote Command Execution). Collect, reduce and sort results of native and custom data processing. Ready to support transactional messages processing.
  • 5. HCE key functional principles ● ● ● ● ● Free network cluster structure architecture. Target applied project can construct specific schema of network relations that succeeds on hardware, load-balancing, file-over, fault-tolerance and another principles on the basis of one simple Lego-like engine. Stable based on ZMQ sockets reversed client-server networking protocol with connection heart-beating, automated restoration and messages buffering. Easy asynchronous connections handling with NUMA oriented architecture of messages handlers. Unified I/O messages based on json format. Ready to have client APIs bindings for many programmer languages covered by ZMQ library. Can be easily integrated and deployed.
  • 6. HCE-node application The heart and main component of the HCE project it is hce-node application. This application integrates complete set of base functionality to support network infrastructure, hierarchical cluster construction, full-text search system integration and so on, see “Hierarchical Cluster as engine” main points. Implemented for Linux OS environment and distributed in form of source code tarball archive and Debian Linux binary package with dependencies packages. Supports single instance configuration-less start or requires set of options that used to build correspondent network cluster architecture. Supposes usage with client-side applications or integrated IPI. First implementation of client-side API and cli utilities bind on PHP.
  • 7. Hce-node roles in the cluster structure Internally HCE-node application contains seven basic handler threads. Each handler acts as special black-box messages processor/dispatcher and used in combination with other to work in one of five different roles of node: ● ● ● ● Router – upper end-point of cluster hierarchy. Has three server-type connections. Handles client API, any kind of another node roles instances (typically, shard or replica managers) and admin connections. Shard manager – intermediate-point of cluster hierarchy. Routes messages between upper and down layers. Uses data sharding and messages multicast dispatching algorithms. Has two server-type and one client connections. Replica manager – the same as shard manager. Routes messages between upper and down layers uses data balancing and messages round-robin algorithms. Replica – down end-point of cluster hierarchy. Data node, interacts with data storage and/or process data with target algorithm(s), provides interface with fill-text search engine, target host for Distributed Remote Commands Execution. Has one serverand one client-side connections used for cluster infrastructure Also can to have several data storage-dependent connections.
  • 9. Hce-node internal architecture seven handlers objects in relations
  • 10. hce-node in cluster structure relations
  • 12. Hierarchical cluster structure example This is just one example of four level cluster structure. This structure can be used to create four hosts distributed data model solution to split one huge documents storage on three physical host servers and utilize all CPU and I/O resources in effective way. Each level uses different data distribution modes (shard or replica). Two replica nodes of each shard balances requests and have complete mirrored data. Shards are multicasted and requests results accumulated and reduced by shard manager node.
  • 13. Extended Sphinx index support - Smooth transparent aggregation of several different indexes and search algorithms in federated search. - Native support by file operation the run-time indexes attach, copy, replace, swap, mirror, merge. - Native support of indexes sharding and mirroring according different data schema. - Extended full text search queries including full match, pattern match, set of fields match, logical operators, proximity limits, stop-words lists and many other provided by regular Sphinx search (c) engine. - Filters for numeric fields including exact match and range comparisons. - Multi-value numeric fields, bit-set and logical operations simulation in search query. - Custom order including Sphinx full text search weight and custom weightier calculation algorithms definitions like SQL ORDER BY prioritized fields list or more complex logic.
  • 14. HCE Sphinx Index Extension - SIE hce-node application provides solution named HCE Sphinx Index Extension (HCE SIE) that add to the Sphinx index nature more functionality and two high level structures: Index – local FS folder contains complete set of sub-folders and files including native Sphinx indexes (trunk and branches), configuration files for searchd, source xml data files of branches, prepared xml data files of branches, schema xml file and other... Index supports operations: create, copy, rename, delete, start, stop, rebuild, merge branches, delete document, connect, and other... Branch – local FS files like source xml data file, prepared xml data file and native Sphinx Index files created from prepared xml data file. All branches of index uses the same schema and represents some part of index. Set of branches can be merged in trunk that is used in search process. Branch supports operations: create, delete, rebuild, insert document, delete document and other...
  • 15. HCE SIE folder structure
  • 16. HCE technologies basics ● ● ● ● ● ZMQ (c) sockets for inter-process and in-process interactions. POCO C++ framework (c) for application architecture and algorithms including json and xml parsers, objects serialization, configuration settings and many other. Sphinx search (c) engine client native integration with extensions. Mutex-less multithread application architecture and messages handling. Strict C++ OOP design, including elements of 11 standard.
  • 17. HCE distribution and deployment ● ● Open source distribution in source code of tarball archive. Binary Debian Linux native packages separated on HCE-node and utility packages as well as complete full meta-package. Dependencies libraries packaging and maintenance to complete cover dependencies and make installation smooth. ● Public repositories accessible for contributors community. ● Upgrade dependency packages and libraries as part of product. ● ● Box ready to start distributive with Demo Test Suit (DTS) with examples of simple cluster structure to start full text Sphinxbased search right after installation. Open documentation.
  • 18. HCE project todo... - Additions and extensions to support transparent configurable transactional model for all kind of actions. - Extension of data node with native support of another index types and data storage like SQLite. - Extensions with set of API and utility tools for creating typical cluster structures and sharding models. - Extensions with set of API for general management and tracking of statistics, including web-UI for state visualization and visual management. - Extensions for statistics and logging data analysis and visualization.