NOSQL Now! Presentation, August 24, 2011: Graph Databases: Connecting the Dots in Big Data

Graph Databases : Connecting the Dots in Big Data Darren Wood Chief Architect, InfiniteGraph

Relationships are everywhere

Graph Databases
Not Really Graph Problems
Average age of my customers that purchased X
Which zip code buys the most of Y
Graph Problems
How is person A connected to person B
Can suspect Y be associated with location Z
Who are influencers within a social network ?
Copyright © InfiniteGraph

Graph Databases
Optimized around data relationships
Relationships as first class citizens
Super fast traversal between entities
Rich/flexible annotation of connections
Small focused API (typically not SQL)
Natively work with concepts of Vertex/Edge
SQL has no concept of “navigation”
Most attempts based in SQL are convoluted

Physical Storage Comparison Copyright © InfiniteGraph Meetings P1 Place Time P2 Alice Denver 5-27-10 Bob Calls From Time Duration To Bob 13:20 25 Carlos Bob 17:10 15 Charlie Payments From Date Amount To Carlos 5-12-10 100000 Charlie Met 5-27-10 Alice Called 13:20 Bob Payed 100000 Carlos Charlie Called 17:10 Rows/Columns/Tables Relationship/Graph Optimized

Simple API Copyright © InfiniteGraph Vertex alice = myGraph.addVertex(new Person(“Alice”)); Vertex bob = myGraph.addVertex(new Person(“Bob”)); Vertex carlos = myGraph.addVertex(new Person(“Carlos”)); Vertex charlie = myGraph.addVertex(new Person(“Charlie”)); alice.addEdge(new Meeting(“Denver”, “5-27-10”), bob); bob.addEdge(new Call(timestamp), carlos); carlos.addEdge(new Payment(100000.00), charlie); bob.addEdge(new Call(timestamp), charlie); Alice Carlos Charlie Bob Meets Calls Pays Calls

Query and Navigation
Queries – but not as you know them
More like a rules based search and discovery
Asynchronous Results
Copyright © InfiniteGraph Alice Carlos Charlie Bob Meets Calls Pays Calls “ Find all paths between Alice and Charlie” “ Find all paths between Alice and Charlie – within 2 degrees” “ Find all paths between Alice and Charlie – events in May 2010”

Navigation Example Copyright © InfiniteGraph // Create a qualifier that describes the target vertex Qualifier findCharliePredicate = new VertexPredicate(personType, "name == ’Charlie'" ); // Construct a navigator which starts with Alice and uses a result qualifier // to find all paths in the graph to Charlie Navigator charlieFinder = alice.navigate( Guide.SIMPLE_BREADTH_FIRST, // default guide Qualifier.ANY, // no path constraints findCharliePredicate , // find paths ending with Charlie myResultHandler); // fire results to supplied handler // Start the navigator charlieFinder.start();

Navigational Query Performance

Scaling Graphs – Getting Data In Copyright © InfiniteGraph IG Core/API Configuration Navigation Execution Management Extensions Session / TX Management Placement Standard Blocking Ingest/Placement (MDP Plugin) Objectivity/DB App-1 (Ingest V 1 ) App-2 (Ingest V 2 ) App-3 (Ingest V 3 ) V 1 V 2 V 3 App-1 (E 1 2 { V 1 V 2 }) App-2 (E 23 { V 2 V 3 }) App-3 E 12 E 23

Accelerated Ingest Copyright © InfiniteGraph IG Core/API Configuration Navigation Execution Management Extensions Session / TX Management Placement (Standard) Placement (Accelerated) V 1 V 2 V 3 E 12 E 23 Distributed Pipelines Staging Containers Pipeline Containers E(1->2) E(3->1) E(2->3) E(2->1) E(2->3) E(3->1) E(1->2) E(3->2) E(1->2) E(2->3) E(3->1) E(2->1) E(2->3) E(3->1) E(3->2) E(1->2)

Choose Your Own Consistency… Copyright © InfiniteGraph // Describe your requested model using policies PolicyChain myPolicies = new PolicyChain(new EdgePipeliningPolicy( true )); // Start a transaction with the policies you want Transaction tx = myGraph.beginTransaction( AccessMode.READ_WRITE, myPolicies); // This code doesn’t change, can be used with any policies alice.addEdge(new Meeting(“Denver”, “5-27-10”), bob); bob.addEdge(new Call(timestamp), carlos); tx.commit();

Indexing Framework
Focused on providing choice !
Manual Indexes for grouping data
Automatic Indexes for cross population
Query interface with qualification language
Pluggable query operators
External index support (Lucene)

Scaling Graphs – Distributed Navigation
Graph algorithms naturally branch
Requires orchestration of threads/agents
Copyright © InfiniteGraph Alice Carlos Charlie Bob Meets Calls Pays Dave Eve Chuck Calls Lives With Meets

Big Distributed Data (Traditional - Huge Generalization) Copyright © InfiniteGraph Distributed API Application(s) Partition 1 Partition 3 Partition 2 Partition ... n Processor Processor Processor Processor

Some customers and partners

NOSQL Now! Presentation, August 24, 2011: Graph Databases: Connecting the Dots in Big Data

by InfiniteGraph

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 46

Statistics