• Like

Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

GaianDB

  • 2,629 views
Uploaded on

presentation I gave on GaianDB - a dynamic federated distributed database available on IBM alphaWorks …

presentation I gave on GaianDB - a dynamic federated distributed database available on IBM alphaWorks

The presentation wont make a lot of sense without speaker notes... which I've not written yet. Sorry about that.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,629
On Slideshare
0
From Embeds
0
Number of Embeds
4

Actions

Shares
Downloads
29
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. GaianDB A dynamic distributed federated database Dale Lane @dalelane
  • 2. A massively over-simplified view of data-warehousing...
  • 3. The “Internet of Things”
  • 4. GaianDB a dynamic distributed federated database
  • 5. Federated data
  • 6. Network of distributed databases
  • 7. A dynamic network
  • 8. A dynamic network Biologically-Inspired Self-Organisation Exploit natural selection in nature to build better networks Robust self-organizing network architectures Frameworks and algorithms for robust fault-tolerant information dissemination Robust communications with minimal complexity or human control
  • 9. Gaian database Queries routed to all database nodes – a N5 N7 N4 flood query, but retrieving only the data N6 required to satisfy a query N8 N3 N9 Exchanges query traffic in the network for N10 SQL Query Queries N2 data traffic – aiming to minimize total traffic N0 N1 N11 N5 N7 N4 N6 N8 N3 N9 N10 Predicated on a concept of ‘store SQL Query N2 N0 N1 data locally - read data from anywhere’ paradigm N11
  • 10. Architecture Multithreaded, breadth-first query propagation N5 N7 N4 N6 N8 Loop detection/handling – no duplicates N3 N9 N10 SQL Query N2 N0 N1 Expanded Node N11 GaianDB GaianDB Derby Engine: Parsing, Compilation, Execution GaianDB GaianDB Derby tables Pushes columns Instantiates Original SQLInvokes costingand ‘where’ clause propagate methods in a structure In-memory tables GaianPStmtNode VTI: Executes queries on physical leaf nodes + Propagates the original SQL (+ queryID & steps state info) to linked Gaian nodes MQ(tt) Stream Data MS Text Index Flat files Derby DB2 Oracle SQLServer Sybase MySQL
  • 11. Performance – with 1,250 nodes Query Performance 539.0 485.1 431.2 Query Time(milliseconds) 377.3 323.4 269.5 215.6 161.7 Average Query Time 107.8 Predicted Max (Layers) Predicted Min (Layers) 53.9 0.0 0 200 400 600 800 1000 1200 Number of Nodes Query time for 1025 nodes, fetching up to 1025 rows from each 6000 Query Execute Time Total Query Time 5000 Linear (Total Query Time) 4000 y = 4.217x + 349.251 Time (milliseconds) 3000 2000 1000 0 0 200 400 600 800 1000 1200 Row s fe tche d pe r node
  • 12. Performance questions The time to propagate a query to all of the nodes in the database, as a function of the number of database nodes (N); The time to fetch data from across the nodes of the database to a single node, as a function of the volume of data; The time to fetch data from across the database to multiple nodes concurrently querying, as a function of the number of nodes concurrently querying.
  • 13. Graph metrics The eccentricity ε(νi) of a graph vertex νi is the maximum graph distance between νi and any other vertex νj of G i.e. the "longest shortest path" between any two graph vertices (νi , νj) of the graph. The maximum eccentricity is the graph diameter Gd. The minimum graph eccentricity is the graph radius Gr. We define the size of G as the number of vertices N and the number of connections at each vertex as the vertex degree δi (1 < i ≤ N).
  • 14. Biologically inspired self-organisation Network growth by 10 9 preferential attachment 8 Using a fitness function at Graph Dimension (edges) 7 each node 6 5 Limit maximum vertex degree =10 4 Radius 3 Diameter 2 Gd = nint [ (1+e) * ln(N) ] (1+e)ln(N) 1 (1-e)ln(N) 0 Gr = nint [ (1-e) * ln(N) ] 0 200 400 600 800 1000 Number of Nodes (N) e = 0.24
  • 15. Query propagation time The predicted maximum (Tmax) and minimum times (Tmin) to execute the flood query are: TL = link latency Tp = processor delay Tmax = (Gd + 1)(TL + Tp) Tmin = (Gr + 1)(TL + Tp) with the predicted execute query time from any node (Tν) being: Tν = (ε(ν) + 1)(TL + Tp) Hence substituting for ε(ν) Tν = nint[1 + B * ln(N) * (TL + Tp)]
  • 16. Measured query propagation Individual Query Time Scalability Individual Query Time Scalability 323.4 592.9 539.0 269.5 485.1 431.2 215.6 377.3 323.4 Query Time (ms) Query Time (ms) 161.7 269.5 215.6 107.8 Individual Query Times Average Query Time Average Query Time 161.7 Predicted Max (Diameter+1) Queried node eccentricity+1 Predicted Min (Radius+1) 53.9 107.8 Queried node eccentricity+1 53.9 0 0.0 0 50 100 0 200 400 600 800 1000 1200 Number of Nodes Number of Nodes
  • 17. Measured data fetch Query time to fetch 1 million rows 6000 Total Query Time 1025 nodes Total Query Time 1 node 5000 Total Query Time 1 node indexed Linear (Total Query Time 1025 nodes) 4000 Linear (Total Query Time 1 node) Time (milliseconds) y = 4.217x + 349.251 3000 2000 y = 1.7383x + 678.141 1000 0 0 200000 400000 600000 800000 1000000 1200000 Total Rows fetched
  • 18. Example uses
  • 19. Smart Metering write centralised
  • 20. Smart Metering read centralised
  • 21. Smart Metering write distributed federated
  • 22. Smart Metering read distributed federated
  • 23. Other uses...
  • 24. http://www.alphaworks.ibm.com/tech/gaiandb
  • 25. Image credits Background: YouTube video “The Internet of Things”, IBM http://www.youtube.com/watch?v=sfEbMV295Kk Icons: DB and envelope icons, Tim Morgan http://flickr.com/photos/timothymorgan/sets/1615269 Microsoft Excel icon, Vincent Garnier (courtesy of IconArchive) http://iconarchive.com/show/softdimension-icons-by-benjigarner/Excel-icon.html Photo of car mechanics, Tomas http://flickr.com/photos/tma/2264878 All other images original from GaianDB work