SlideShare a Scribd company logo
1 of 25
Download to read offline
GaianDB
A dynamic distributed
federated database
Dale Lane
@dalelane
A massively over-simplified view of
data-warehousing...
The “Internet of Things”
GaianDB
a
dynamic
distributed
federated
database
Federated data
Network of distributed databases
A dynamic network
A dynamic network
Biologically-Inspired Self-Organisation
Exploit natural selection in nature to
build better networks
Robust self-organizing network
architectures
Frameworks and algorithms for robust
fault-tolerant information dissemination
Robust communications with minimal
complexity or human control
Gaian database
N0
N3
N11
N4
N5
N1
N2
N6
N7
N8
N10
N9
SQL Query
N0
N3
N11
N4
N5
N1
N2
N6
N7
N8
N10
N9
SQL Query
N0
N3
N11
N4
N5
N1
N2
N6
N7
N8
N10
N9
SQL Query
N0
N3
N11
N4
N5
N1
N2
N6
N7
N8
N10
N9
SQL Queries
Queries routed to all database nodes – a
flood query, but retrieving only the data
required to satisfy a query
Exchanges query traffic in the network for
data traffic – aiming to minimize total traffic
Predicated on a concept of ‘store
data locally - read data from
anywhere’ paradigm
Architecture
GaianDB
Derby Engine: Parsing, Compilation, Execution
GaianPStmtNode VTI:
Executes queries on physical leaf nodes +
Propagates the original SQL (+ queryID & steps state info) to linked Gaian nodes
Instantiates Invokes costing
methods
Pushes columns
and ‘where’ clause
in a structure
MQ(tt)
Stream Data
Original SQL
DB2 Oracle MS
SQLServer
Sybase MySQL Flat files
In-memory
tables
Derby
GaianDB
GaianDB
GaianDB
propagate
Text Index
Derby
tables
N0
N3
N11
N4
N5
N1
N2
N6
N7
N8
N10
N9
SQL Query
N0
N3
N11
N4
N5
N1
N2
N6
N7
N8
N10
N9
SQL Query
Expanded Node
Multithreaded, breadth-first query propagation
Loop detection/handling – no duplicates
Performance – with 1,250 nodes
Query time for 1025 nodes, fetching up to 1025 rows from each
y = 4.217x + 349.251
0
1000
2000
3000
4000
5000
6000
0 200 400 600 800 1000 1200
Row s fetched per node
Time(milliseconds)
Query Execute Time
Total Query Time
Linear (Total Query Time)
Query Performance
0.0
53.9
107.8
161.7
215.6
269.5
323.4
377.3
431.2
485.1
539.0
0 200 400 600 800 1000 1200
Number of Nodes
QueryTime(milliseconds)
Average Query Time
Predicted Max (Layers)
Predicted Min (Layers)
Performance questions
The time to propagate a query to all of
the nodes in the database, as a function
of the number of database nodes (N);
The time to fetch data from across the
nodes of the database to a single node,
as a function of the volume of data;
The time to fetch data from across the
database to multiple nodes concurrently
querying, as a function of the number
of nodes concurrently querying.
Graph metrics
The eccentricity ε(νi) of a graph
vertex νi is the maximum graph
distance between νi and any other
vertex νj of G i.e. the "longest
shortest path" between any two
graph vertices (νi , νj) of the graph.
The maximum eccentricity is the
graph diameter Gd. The minimum
graph eccentricity is the graph
radius Gr. We define the size of G as
the number of vertices N and the
number of connections at each
vertex as the vertex degree δi
(1 < i ≤ N).
Biologically inspired self-organisation
0
1
2
3
4
5
6
7
8
9
10
0 200 400 600 800 1000
Number of Nodes (N)
GraphDimension(edges)
Radius
Diameter
(1+e)ln(N)
(1-e)ln(N)
Network growth by
preferential attachment
Using a fitness function at
each node
Limit maximum vertex degree =10
Gd = nint [ (1+e) * ln(N) ]
Gr = nint [ (1-e) * ln(N) ]
e = 0.24
Query propagation time
The predicted maximum (Tmax) and
minimum times (Tmin) to execute the
flood query are:
TL = link latency
Tp = processor delay
Tmax = (Gd + 1)(TL + Tp)
Tmin = (Gr + 1)(TL + Tp)
with the predicted execute query time
from any node (Tν) being:
Tν = (ε(ν) + 1)(TL + Tp)
Hence substituting for ε(ν)
Tν = nint[1 + B * ln(N) * (TL + Tp)]
Measured query propagation
IndividualQueryTimeScalability
0.0
53.9
107.8
161.7
215.6
269.5
323.4
377.3
431.2
485.1
539.0
592.9
0 200 400 600 800 1000 1200
Number of Nodes
QueryTime(ms) AverageQueryTime
PredictedMax(Diameter+1)
PredictedMin(Radius+1)
Queriednodeeccentricity+1
Individual Query Time Scalability
0
53.9
107.8
161.7
215.6
269.5
323.4
0 50 100
Number ofNodes
QueryTime(ms)
Individual Query Times
Average Query Time
Queried node eccentricity+1
Measured data fetch
Query time to fetch 1 million rows
y = 4.217x + 349.251
y = 1.7383x + 678.141
0
1000
2000
3000
4000
5000
6000
0 200000 400000 600000 800000 1000000 1200000
Total Rows fetched
Time(milliseconds)
Total Query Time 1025 nodes
Total Query Time 1 node
Total Query Time 1 node indexed
Linear (Total Query Time 1025 nodes)
Linear (Total Query Time 1 node)
Example uses
Smart Metering
centralised
write
Smart Metering
centralised
read
Smart Metering
distributed federated
write
Smart Metering
distributed federated
read
Other uses...
http://www.alphaworks.ibm.com/tech/gaiandb
Image credits
Background: YouTube video “The Internet of Things”, IBM
http://www.youtube.com/watch?v=sfEbMV295Kk
Icons: DB and envelope icons, Tim Morgan
http://flickr.com/photos/timothymorgan/sets/1615269
Microsoft Excel icon, Vincent Garnier (courtesy of IconArchive)
http://iconarchive.com/show/softdimension-icons-by-benjigarner/Excel-icon.html
Photo of car mechanics, Tomas
http://flickr.com/photos/tma/2264878
All other images original from GaianDB work

More Related Content

What's hot

Talend ESB : Monitoring, Repartition de Charge et Authentification
Talend ESB : Monitoring, Repartition de Charge et AuthentificationTalend ESB : Monitoring, Repartition de Charge et Authentification
Talend ESB : Monitoring, Repartition de Charge et AuthentificationLilia Sfaxi
 
Système Information - ETL et EAI - Décisionnel et Opérationnel
Système Information - ETL et EAI - Décisionnel et OpérationnelSystème Information - ETL et EAI - Décisionnel et Opérationnel
Système Information - ETL et EAI - Décisionnel et OpérationnelFrédéric FAURE
 
Cours d'algorithmique distribuée (2010-2011)
Cours d'algorithmique distribuée (2010-2011)Cours d'algorithmique distribuée (2010-2011)
Cours d'algorithmique distribuée (2010-2011)Élodie Descharmes
 
La gestion des archives avec Alfresco
La gestion des archives avec AlfrescoLa gestion des archives avec Alfresco
La gestion des archives avec AlfrescoSavoir-faire Linux
 
alphorm.com - Formation SQL Server 2012 (70-462)
alphorm.com - Formation SQL Server 2012 (70-462)alphorm.com - Formation SQL Server 2012 (70-462)
alphorm.com - Formation SQL Server 2012 (70-462)Alphorm
 
Un exemple élémentaire d'application MVC en PHP
Un exemple élémentaire d'application MVC en PHPUn exemple élémentaire d'application MVC en PHP
Un exemple élémentaire d'application MVC en PHPKristen Le Liboux
 
Les architectures client serveur
Les architectures client serveurLes architectures client serveur
Les architectures client serveurAmeni Ouertani
 
Introduction to JCR and Apache Jackrabbi
Introduction to JCR and Apache JackrabbiIntroduction to JCR and Apache Jackrabbi
Introduction to JCR and Apache JackrabbiJukka Zitting
 
Architectures distribuées
Architectures distribuéesArchitectures distribuées
Architectures distribuéesFranck SIMON
 
BigData_TP5 : Neo4J
BigData_TP5 : Neo4JBigData_TP5 : Neo4J
BigData_TP5 : Neo4JLilia Sfaxi
 
cours javascript.pptx
cours javascript.pptxcours javascript.pptx
cours javascript.pptxYaminaGh1
 
Diaporamas-Bases de données.pdf
Diaporamas-Bases de données.pdfDiaporamas-Bases de données.pdf
Diaporamas-Bases de données.pdfEST-UMI de Meknès
 
introduction à la modélisation objet
introduction à la modélisation objetintroduction à la modélisation objet
introduction à la modélisation objetAmir Souissi
 
Database/ Bases de données
Database/ Bases de donnéesDatabase/ Bases de données
Database/ Bases de donnéeszied kallel
 
Uml: Diagrammes de classes -- Concepts avances --- 27
Uml: Diagrammes de classes -- Concepts avances --- 27Uml: Diagrammes de classes -- Concepts avances --- 27
Uml: Diagrammes de classes -- Concepts avances --- 27megaplanet20
 

What's hot (20)

Préservation par la numérisation
Préservation par la numérisationPréservation par la numérisation
Préservation par la numérisation
 
Igl cours 3 - introduction à uml
Igl   cours 3 - introduction à umlIgl   cours 3 - introduction à uml
Igl cours 3 - introduction à uml
 
Talend ESB : Monitoring, Repartition de Charge et Authentification
Talend ESB : Monitoring, Repartition de Charge et AuthentificationTalend ESB : Monitoring, Repartition de Charge et Authentification
Talend ESB : Monitoring, Repartition de Charge et Authentification
 
Système Information - ETL et EAI - Décisionnel et Opérationnel
Système Information - ETL et EAI - Décisionnel et OpérationnelSystème Information - ETL et EAI - Décisionnel et Opérationnel
Système Information - ETL et EAI - Décisionnel et Opérationnel
 
Cours d'algorithmique distribuée (2010-2011)
Cours d'algorithmique distribuée (2010-2011)Cours d'algorithmique distribuée (2010-2011)
Cours d'algorithmique distribuée (2010-2011)
 
La gestion des archives avec Alfresco
La gestion des archives avec AlfrescoLa gestion des archives avec Alfresco
La gestion des archives avec Alfresco
 
alphorm.com - Formation SQL Server 2012 (70-462)
alphorm.com - Formation SQL Server 2012 (70-462)alphorm.com - Formation SQL Server 2012 (70-462)
alphorm.com - Formation SQL Server 2012 (70-462)
 
Un exemple élémentaire d'application MVC en PHP
Un exemple élémentaire d'application MVC en PHPUn exemple élémentaire d'application MVC en PHP
Un exemple élémentaire d'application MVC en PHP
 
Les architectures client serveur
Les architectures client serveurLes architectures client serveur
Les architectures client serveur
 
Introduction to JCR and Apache Jackrabbi
Introduction to JCR and Apache JackrabbiIntroduction to JCR and Apache Jackrabbi
Introduction to JCR and Apache Jackrabbi
 
Cours uml
Cours umlCours uml
Cours uml
 
Architectures distribuées
Architectures distribuéesArchitectures distribuées
Architectures distribuées
 
BigData_TP5 : Neo4J
BigData_TP5 : Neo4JBigData_TP5 : Neo4J
BigData_TP5 : Neo4J
 
cours javascript.pptx
cours javascript.pptxcours javascript.pptx
cours javascript.pptx
 
Support de cours angular
Support de cours angularSupport de cours angular
Support de cours angular
 
Diaporamas-Bases de données.pdf
Diaporamas-Bases de données.pdfDiaporamas-Bases de données.pdf
Diaporamas-Bases de données.pdf
 
introduction à la modélisation objet
introduction à la modélisation objetintroduction à la modélisation objet
introduction à la modélisation objet
 
Database/ Bases de données
Database/ Bases de donnéesDatabase/ Bases de données
Database/ Bases de données
 
La plateforme JEE
La plateforme JEELa plateforme JEE
La plateforme JEE
 
Uml: Diagrammes de classes -- Concepts avances --- 27
Uml: Diagrammes de classes -- Concepts avances --- 27Uml: Diagrammes de classes -- Concepts avances --- 27
Uml: Diagrammes de classes -- Concepts avances --- 27
 

Similar to GaianDB

Seattle Scalability Meetup 6-26-13
Seattle Scalability Meetup 6-26-13Seattle Scalability Meetup 6-26-13
Seattle Scalability Meetup 6-26-13specialk29
 
Introduction to computer network 4th edition
Introduction to computer network   4th editionIntroduction to computer network   4th edition
Introduction to computer network 4th editionEnsign Handoko
 
Information Retrieval Dynamic Time Warping - Interspeech 2013 presentation
Information Retrieval Dynamic Time Warping - Interspeech 2013 presentationInformation Retrieval Dynamic Time Warping - Interspeech 2013 presentation
Information Retrieval Dynamic Time Warping - Interspeech 2013 presentationXavier Anguera
 
Object multifunctional indexing with an open API
Object multifunctional indexing with an open API Object multifunctional indexing with an open API
Object multifunctional indexing with an open API akvalex
 
Coding the Continuum
Coding the ContinuumCoding the Continuum
Coding the ContinuumIan Foster
 
NeuralProcessingofGeneralPurposeApproximatePrograms
NeuralProcessingofGeneralPurposeApproximateProgramsNeuralProcessingofGeneralPurposeApproximatePrograms
NeuralProcessingofGeneralPurposeApproximateProgramsMohid Nabil
 
Towards neuralprocessingofgeneralpurposeapproximateprograms
Towards neuralprocessingofgeneralpurposeapproximateprogramsTowards neuralprocessingofgeneralpurposeapproximateprograms
Towards neuralprocessingofgeneralpurposeapproximateprogramsParidha Saxena
 
Neurogrid : A Mixed-Analog-Digital Multichip System for Large-Scale Neural Si...
Neurogrid : A Mixed-Analog-Digital Multichip System for Large-Scale Neural Si...Neurogrid : A Mixed-Analog-Digital Multichip System for Large-Scale Neural Si...
Neurogrid : A Mixed-Analog-Digital Multichip System for Large-Scale Neural Si...IIT Bombay
 
Lecture 25
Lecture 25Lecture 25
Lecture 25Shani729
 
Enchancing the Data Collection in Tree based Wireless Sensor Networks
Enchancing the Data Collection in Tree based Wireless Sensor NetworksEnchancing the Data Collection in Tree based Wireless Sensor Networks
Enchancing the Data Collection in Tree based Wireless Sensor Networksijsrd.com
 
Security-Aware Scheduling for Real-Time Parallel Applications on Clusters
Security-Aware Scheduling for Real-Time Parallel Applications on ClustersSecurity-Aware Scheduling for Real-Time Parallel Applications on Clusters
Security-Aware Scheduling for Real-Time Parallel Applications on ClustersXiao Qin
 
Spectral-, source-, connectivity- and network analysis of EEG and MEG data
Spectral-, source-, connectivity- and network analysis of EEG and MEG dataSpectral-, source-, connectivity- and network analysis of EEG and MEG data
Spectral-, source-, connectivity- and network analysis of EEG and MEG dataRobert Oostenveld
 
Implementation on Data Security Approach in Dynamic Multi Hop Communication
 Implementation on Data Security Approach in Dynamic Multi Hop Communication Implementation on Data Security Approach in Dynamic Multi Hop Communication
Implementation on Data Security Approach in Dynamic Multi Hop CommunicationIJCSIS Research Publications
 
International Refereed Journal of Engineering and Science (IRJES)
International Refereed Journal of Engineering and Science (IRJES)International Refereed Journal of Engineering and Science (IRJES)
International Refereed Journal of Engineering and Science (IRJES)irjes
 

Similar to GaianDB (20)

Seattle Scalability Meetup 6-26-13
Seattle Scalability Meetup 6-26-13Seattle Scalability Meetup 6-26-13
Seattle Scalability Meetup 6-26-13
 
Introduction to computer network 4th edition
Introduction to computer network   4th editionIntroduction to computer network   4th edition
Introduction to computer network 4th edition
 
Information Retrieval Dynamic Time Warping - Interspeech 2013 presentation
Information Retrieval Dynamic Time Warping - Interspeech 2013 presentationInformation Retrieval Dynamic Time Warping - Interspeech 2013 presentation
Information Retrieval Dynamic Time Warping - Interspeech 2013 presentation
 
Paper_KST
Paper_KSTPaper_KST
Paper_KST
 
Object multifunctional indexing with an open API
Object multifunctional indexing with an open API Object multifunctional indexing with an open API
Object multifunctional indexing with an open API
 
NBITSearch. Features.
NBITSearch. Features.NBITSearch. Features.
NBITSearch. Features.
 
Ijnsa050209
Ijnsa050209Ijnsa050209
Ijnsa050209
 
Coding the Continuum
Coding the ContinuumCoding the Continuum
Coding the Continuum
 
NeuralProcessingofGeneralPurposeApproximatePrograms
NeuralProcessingofGeneralPurposeApproximateProgramsNeuralProcessingofGeneralPurposeApproximatePrograms
NeuralProcessingofGeneralPurposeApproximatePrograms
 
Towards neuralprocessingofgeneralpurposeapproximateprograms
Towards neuralprocessingofgeneralpurposeapproximateprogramsTowards neuralprocessingofgeneralpurposeapproximateprograms
Towards neuralprocessingofgeneralpurposeapproximateprograms
 
Neurogrid : A Mixed-Analog-Digital Multichip System for Large-Scale Neural Si...
Neurogrid : A Mixed-Analog-Digital Multichip System for Large-Scale Neural Si...Neurogrid : A Mixed-Analog-Digital Multichip System for Large-Scale Neural Si...
Neurogrid : A Mixed-Analog-Digital Multichip System for Large-Scale Neural Si...
 
User biglm
User biglmUser biglm
User biglm
 
Lecture 25
Lecture 25Lecture 25
Lecture 25
 
D031202018023
D031202018023D031202018023
D031202018023
 
G04701051058
G04701051058G04701051058
G04701051058
 
Enchancing the Data Collection in Tree based Wireless Sensor Networks
Enchancing the Data Collection in Tree based Wireless Sensor NetworksEnchancing the Data Collection in Tree based Wireless Sensor Networks
Enchancing the Data Collection in Tree based Wireless Sensor Networks
 
Security-Aware Scheduling for Real-Time Parallel Applications on Clusters
Security-Aware Scheduling for Real-Time Parallel Applications on ClustersSecurity-Aware Scheduling for Real-Time Parallel Applications on Clusters
Security-Aware Scheduling for Real-Time Parallel Applications on Clusters
 
Spectral-, source-, connectivity- and network analysis of EEG and MEG data
Spectral-, source-, connectivity- and network analysis of EEG and MEG dataSpectral-, source-, connectivity- and network analysis of EEG and MEG data
Spectral-, source-, connectivity- and network analysis of EEG and MEG data
 
Implementation on Data Security Approach in Dynamic Multi Hop Communication
 Implementation on Data Security Approach in Dynamic Multi Hop Communication Implementation on Data Security Approach in Dynamic Multi Hop Communication
Implementation on Data Security Approach in Dynamic Multi Hop Communication
 
International Refereed Journal of Engineering and Science (IRJES)
International Refereed Journal of Engineering and Science (IRJES)International Refereed Journal of Engineering and Science (IRJES)
International Refereed Journal of Engineering and Science (IRJES)
 

More from Dale Lane

Describing Kafka security in AsyncAPI
Describing Kafka security in AsyncAPIDescribing Kafka security in AsyncAPI
Describing Kafka security in AsyncAPIDale Lane
 
Our NASA Space Apps Challenge 2019 entry
Our NASA Space Apps Challenge 2019 entryOur NASA Space Apps Challenge 2019 entry
Our NASA Space Apps Challenge 2019 entryDale Lane
 
Useful Kafka tools
Useful Kafka toolsUseful Kafka tools
Useful Kafka toolsDale Lane
 
An intro to serverless and OpenWhisk for Kafka users
An intro to serverless and OpenWhisk for Kafka usersAn intro to serverless and OpenWhisk for Kafka users
An intro to serverless and OpenWhisk for Kafka usersDale Lane
 
How to increase the social impact you make
How to increase the social impact you makeHow to increase the social impact you make
How to increase the social impact you makeDale Lane
 
Introducing Machine Learning to Kids
Introducing Machine Learning to KidsIntroducing Machine Learning to Kids
Introducing Machine Learning to KidsDale Lane
 
Introducing machine learning to kids
Introducing machine learning to kidsIntroducing machine learning to kids
Introducing machine learning to kidsDale Lane
 
Small Spaces, Big Ideas - our Space Apps Challenge
Small Spaces, Big Ideas - our Space Apps ChallengeSmall Spaces, Big Ideas - our Space Apps Challenge
Small Spaces, Big Ideas - our Space Apps ChallengeDale Lane
 
The skills implications of Cognitive Computing
The skills implications of Cognitive ComputingThe skills implications of Cognitive Computing
The skills implications of Cognitive ComputingDale Lane
 
Conversational Internet - Creating a natural language interface for web pages
Conversational Internet - Creating a natural language interface for web pagesConversational Internet - Creating a natural language interface for web pages
Conversational Internet - Creating a natural language interface for web pagesDale Lane
 
Debugging Web Apps on Real Mobile Devices
Debugging Web Apps on Real Mobile DevicesDebugging Web Apps on Real Mobile Devices
Debugging Web Apps on Real Mobile DevicesDale Lane
 
Pushing, pulling or leaving the door open
Pushing, pulling or leaving the door openPushing, pulling or leaving the door open
Pushing, pulling or leaving the door openDale Lane
 
Push notifications
Push notificationsPush notifications
Push notificationsDale Lane
 
Fire Eagle Guest Pass
Fire Eagle Guest PassFire Eagle Guest Pass
Fire Eagle Guest PassDale Lane
 
Monitoring your electricity usage
Monitoring your electricity usageMonitoring your electricity usage
Monitoring your electricity usageDale Lane
 
An introduction to Windows Mobile development
An introduction to Windows Mobile developmentAn introduction to Windows Mobile development
An introduction to Windows Mobile developmentDale Lane
 
An Introduction to Windows PowerShell
An Introduction to Windows PowerShellAn Introduction to Windows PowerShell
An Introduction to Windows PowerShellDale Lane
 
Mowing the lawn
Mowing the lawnMowing the lawn
Mowing the lawnDale Lane
 

More from Dale Lane (20)

Describing Kafka security in AsyncAPI
Describing Kafka security in AsyncAPIDescribing Kafka security in AsyncAPI
Describing Kafka security in AsyncAPI
 
Our NASA Space Apps Challenge 2019 entry
Our NASA Space Apps Challenge 2019 entryOur NASA Space Apps Challenge 2019 entry
Our NASA Space Apps Challenge 2019 entry
 
Useful Kafka tools
Useful Kafka toolsUseful Kafka tools
Useful Kafka tools
 
An intro to serverless and OpenWhisk for Kafka users
An intro to serverless and OpenWhisk for Kafka usersAn intro to serverless and OpenWhisk for Kafka users
An intro to serverless and OpenWhisk for Kafka users
 
How to increase the social impact you make
How to increase the social impact you makeHow to increase the social impact you make
How to increase the social impact you make
 
Introducing Machine Learning to Kids
Introducing Machine Learning to KidsIntroducing Machine Learning to Kids
Introducing Machine Learning to Kids
 
Introducing machine learning to kids
Introducing machine learning to kidsIntroducing machine learning to kids
Introducing machine learning to kids
 
Small Spaces, Big Ideas - our Space Apps Challenge
Small Spaces, Big Ideas - our Space Apps ChallengeSmall Spaces, Big Ideas - our Space Apps Challenge
Small Spaces, Big Ideas - our Space Apps Challenge
 
Owls
OwlsOwls
Owls
 
The skills implications of Cognitive Computing
The skills implications of Cognitive ComputingThe skills implications of Cognitive Computing
The skills implications of Cognitive Computing
 
Conversational Internet - Creating a natural language interface for web pages
Conversational Internet - Creating a natural language interface for web pagesConversational Internet - Creating a natural language interface for web pages
Conversational Internet - Creating a natural language interface for web pages
 
Debugging Web Apps on Real Mobile Devices
Debugging Web Apps on Real Mobile DevicesDebugging Web Apps on Real Mobile Devices
Debugging Web Apps on Real Mobile Devices
 
Pushing, pulling or leaving the door open
Pushing, pulling or leaving the door openPushing, pulling or leaving the door open
Pushing, pulling or leaving the door open
 
Push notifications
Push notificationsPush notifications
Push notifications
 
Fire Eagle Guest Pass
Fire Eagle Guest PassFire Eagle Guest Pass
Fire Eagle Guest Pass
 
Monitoring your electricity usage
Monitoring your electricity usageMonitoring your electricity usage
Monitoring your electricity usage
 
CurrentCost
CurrentCostCurrentCost
CurrentCost
 
An introduction to Windows Mobile development
An introduction to Windows Mobile developmentAn introduction to Windows Mobile development
An introduction to Windows Mobile development
 
An Introduction to Windows PowerShell
An Introduction to Windows PowerShellAn Introduction to Windows PowerShell
An Introduction to Windows PowerShell
 
Mowing the lawn
Mowing the lawnMowing the lawn
Mowing the lawn
 

Recently uploaded

A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxAna-Maria Mihalceanu
 
Bitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactiveBitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactivestartupro
 
The work to make the piecework work: An ethnographic study of food delivery w...
The work to make the piecework work: An ethnographic study of food delivery w...The work to make the piecework work: An ethnographic study of food delivery w...
The work to make the piecework work: An ethnographic study of food delivery w...stockholm university
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsYoss Cohen
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...Karmanjay Verma
 
Why Agile? - A handbook behind Agile Evolution
Why Agile? - A handbook behind Agile EvolutionWhy Agile? - A handbook behind Agile Evolution
Why Agile? - A handbook behind Agile EvolutionDEEPRAJ PATHAK
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...amber724300
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessWSO2
 
Deliver Latency Free Customer Experience
Deliver Latency Free Customer ExperienceDeliver Latency Free Customer Experience
Deliver Latency Free Customer ExperienceOpsTree solutions
 
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023Joshua Flannery
 
Women in Automation 2024: Career session - explore career paths in automation
Women in Automation 2024: Career session - explore career paths in automationWomen in Automation 2024: Career session - explore career paths in automation
Women in Automation 2024: Career session - explore career paths in automationDianaGray10
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfAarwolf Industries LLC
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesBernd Ruecker
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Jeffrey Haguewood
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesManik S Magar
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFMichael Gough
 

Recently uploaded (20)

A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance Toolbox
 
Bitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactiveBitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactive
 
The work to make the piecework work: An ethnographic study of food delivery w...
The work to make the piecework work: An ethnographic study of food delivery w...The work to make the piecework work: An ethnographic study of food delivery w...
The work to make the piecework work: An ethnographic study of food delivery w...
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platforms
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
 
Why Agile? - A handbook behind Agile Evolution
Why Agile? - A handbook behind Agile EvolutionWhy Agile? - A handbook behind Agile Evolution
Why Agile? - A handbook behind Agile Evolution
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with Platformless
 
Deliver Latency Free Customer Experience
Deliver Latency Free Customer ExperienceDeliver Latency Free Customer Experience
Deliver Latency Free Customer Experience
 
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023
 
Women in Automation 2024: Career session - explore career paths in automation
Women in Automation 2024: Career session - explore career paths in automationWomen in Automation 2024: Career session - explore career paths in automation
Women in Automation 2024: Career session - explore career paths in automation
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdf
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architectures
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDF
 

GaianDB

  • 1. GaianDB A dynamic distributed federated database Dale Lane @dalelane
  • 2. A massively over-simplified view of data-warehousing...
  • 3. The “Internet of Things”
  • 8. A dynamic network Biologically-Inspired Self-Organisation Exploit natural selection in nature to build better networks Robust self-organizing network architectures Frameworks and algorithms for robust fault-tolerant information dissemination Robust communications with minimal complexity or human control
  • 9. Gaian database N0 N3 N11 N4 N5 N1 N2 N6 N7 N8 N10 N9 SQL Query N0 N3 N11 N4 N5 N1 N2 N6 N7 N8 N10 N9 SQL Query N0 N3 N11 N4 N5 N1 N2 N6 N7 N8 N10 N9 SQL Query N0 N3 N11 N4 N5 N1 N2 N6 N7 N8 N10 N9 SQL Queries Queries routed to all database nodes – a flood query, but retrieving only the data required to satisfy a query Exchanges query traffic in the network for data traffic – aiming to minimize total traffic Predicated on a concept of ‘store data locally - read data from anywhere’ paradigm
  • 10. Architecture GaianDB Derby Engine: Parsing, Compilation, Execution GaianPStmtNode VTI: Executes queries on physical leaf nodes + Propagates the original SQL (+ queryID & steps state info) to linked Gaian nodes Instantiates Invokes costing methods Pushes columns and ‘where’ clause in a structure MQ(tt) Stream Data Original SQL DB2 Oracle MS SQLServer Sybase MySQL Flat files In-memory tables Derby GaianDB GaianDB GaianDB propagate Text Index Derby tables N0 N3 N11 N4 N5 N1 N2 N6 N7 N8 N10 N9 SQL Query N0 N3 N11 N4 N5 N1 N2 N6 N7 N8 N10 N9 SQL Query Expanded Node Multithreaded, breadth-first query propagation Loop detection/handling – no duplicates
  • 11. Performance – with 1,250 nodes Query time for 1025 nodes, fetching up to 1025 rows from each y = 4.217x + 349.251 0 1000 2000 3000 4000 5000 6000 0 200 400 600 800 1000 1200 Row s fetched per node Time(milliseconds) Query Execute Time Total Query Time Linear (Total Query Time) Query Performance 0.0 53.9 107.8 161.7 215.6 269.5 323.4 377.3 431.2 485.1 539.0 0 200 400 600 800 1000 1200 Number of Nodes QueryTime(milliseconds) Average Query Time Predicted Max (Layers) Predicted Min (Layers)
  • 12. Performance questions The time to propagate a query to all of the nodes in the database, as a function of the number of database nodes (N); The time to fetch data from across the nodes of the database to a single node, as a function of the volume of data; The time to fetch data from across the database to multiple nodes concurrently querying, as a function of the number of nodes concurrently querying.
  • 13. Graph metrics The eccentricity ε(νi) of a graph vertex νi is the maximum graph distance between νi and any other vertex νj of G i.e. the "longest shortest path" between any two graph vertices (νi , νj) of the graph. The maximum eccentricity is the graph diameter Gd. The minimum graph eccentricity is the graph radius Gr. We define the size of G as the number of vertices N and the number of connections at each vertex as the vertex degree δi (1 < i ≤ N).
  • 14. Biologically inspired self-organisation 0 1 2 3 4 5 6 7 8 9 10 0 200 400 600 800 1000 Number of Nodes (N) GraphDimension(edges) Radius Diameter (1+e)ln(N) (1-e)ln(N) Network growth by preferential attachment Using a fitness function at each node Limit maximum vertex degree =10 Gd = nint [ (1+e) * ln(N) ] Gr = nint [ (1-e) * ln(N) ] e = 0.24
  • 15. Query propagation time The predicted maximum (Tmax) and minimum times (Tmin) to execute the flood query are: TL = link latency Tp = processor delay Tmax = (Gd + 1)(TL + Tp) Tmin = (Gr + 1)(TL + Tp) with the predicted execute query time from any node (Tν) being: Tν = (ε(ν) + 1)(TL + Tp) Hence substituting for ε(ν) Tν = nint[1 + B * ln(N) * (TL + Tp)]
  • 16. Measured query propagation IndividualQueryTimeScalability 0.0 53.9 107.8 161.7 215.6 269.5 323.4 377.3 431.2 485.1 539.0 592.9 0 200 400 600 800 1000 1200 Number of Nodes QueryTime(ms) AverageQueryTime PredictedMax(Diameter+1) PredictedMin(Radius+1) Queriednodeeccentricity+1 Individual Query Time Scalability 0 53.9 107.8 161.7 215.6 269.5 323.4 0 50 100 Number ofNodes QueryTime(ms) Individual Query Times Average Query Time Queried node eccentricity+1
  • 17. Measured data fetch Query time to fetch 1 million rows y = 4.217x + 349.251 y = 1.7383x + 678.141 0 1000 2000 3000 4000 5000 6000 0 200000 400000 600000 800000 1000000 1200000 Total Rows fetched Time(milliseconds) Total Query Time 1025 nodes Total Query Time 1 node Total Query Time 1 node indexed Linear (Total Query Time 1025 nodes) Linear (Total Query Time 1 node)
  • 25. Image credits Background: YouTube video “The Internet of Things”, IBM http://www.youtube.com/watch?v=sfEbMV295Kk Icons: DB and envelope icons, Tim Morgan http://flickr.com/photos/timothymorgan/sets/1615269 Microsoft Excel icon, Vincent Garnier (courtesy of IconArchive) http://iconarchive.com/show/softdimension-icons-by-benjigarner/Excel-icon.html Photo of car mechanics, Tomas http://flickr.com/photos/tma/2264878 All other images original from GaianDB work