An overview of InfiniteGraph, the distributed graph database

An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph

For today’s discussion…
InfiniteGraph
Background / History
Architecture
What we are working on
Consistency Models
Challenges/Requirements of Distibution
Data model – Schema vs Schema-less
Copyright © InfiniteGraph

Building on Objectivity/DB
Derived from Objectivity/DB core
Distributed Object Database
Native core (broad platform support) with C++, Java, C# and Python Bindings
Thousands of deployments, many are 24x7x365
Markets : VLDB, Data Fusion / Metadata, Complex Object Models, Relationship Analytics

Relationship Analytics Experience
Relationship Analytics
Large part of Objy/DB Government business
Significant growth in the commercial space
Many OEM opportunities
Mostly based on graph theory
Hand coded algorithms

Needed something more…
Objectivity Provides
Powerful distributed object storage
Flexible class based persistence
Enterprise DB features
But….
Lots of custom coding
Complex deployment
No generic algorithm framework

Enter InfiniteGraph
Simple Graph focused API
Automated distribution and deployment
Mostly configuration driven
Java class based persistence
Property model support
Asynchronous navigation
Indexing framework

Some code… Copyright © InfiniteGraph Vertex alice = myGraph.addVertex(new Person(“Alice”)); Vertex bob = myGraph.addVertex(new Person(“Bob”)); Vertex carlos = myGraph.addVertex(new Person(“Carlos”)); Vertex charlie = myGraph.addVertex(new Person(“Charlie”)); alice.addEdge(new Meeting(“Denver”, “5-27-10”), bob); bob.addEdge(new Call(timestamp), carlos); carlos.addEdge(new Payment(100000.00), charlie); bob.addEdge(new Call(timestamp), charlie); Alice Carlos Charlie Bob Meets Calls Pays Calls

Basic Architecture Copyright © InfiniteGraph IG Core/API Configuration Navigation Execution Management Extensions Blueprints User Apps Objectivity/DB Distributed Database Session / TX Management Placement

Targeting Large Graphs
Graphs grow quickly
Billions of phone calls / day in US
Emails, social media events, IP Traffic
Financial transactions
Some analytics require navigation of large sections of the graph
Must distribute data and go parallel

Consistency Models
Trading off full consistency for performance
Relax locking, allow stale reads
Gain 100x edge ingest rates
Great for “social graph” applications
These tend to have relaxed consistency requirements
API allows choice per operation

Pipelining Copyright © InfiniteGraph IG Core/API Configuration Navigation Execution Management Extensions Session / TX Management Placement (MDP) Placement (Pipelining) V 1 V 2 V 3 E 12 E 23 Pipeline Manager Staging Containers Pipeline Containers E(1->2) E(3->1) E(2->3) E(2->1) E(2->3) E(3->1) E(1->2) E(3->2) E(1->2) E(2->3) E(3->1) E(2->1) E(2->3) E(3->1) E(3->2) E(1->2)

Distributing Navigation
Graph algorithms naturally branch
Breaking up the process is relatively simple
Orchestrating it is more challenging
Copyright © InfiniteGraph Alice Carlos Charlie Bob Meets Calls Pays Dave Eve Chuck Calls Lives With Meets

Making it scale…
A distributed data tier supports parallel IO
A distributed cache/processor tier for processing “navlets”
Ability to deal with remote data reads (fast)
Navlets hit processor/cache instances based on consistent hashing of their start vertex

Flexible Data Models
Schema isn’t for everyone
Great for relatively static models
Support for inheritance helps
So does auto-evolution
Looking to support document style elements
Support for ad-hoc attributes
Ability to mix both approaches

Other Projects
Blueprints / Gremlin support
Native Visualizer and Management Console
Built in Qualifiers
More Indexing Enhancements
Algorithms
Large scale samples

http://www.readwriteweb.com	2198
http://feeds2.feedburner.com	54
http://feedproxy.google.com	41
http://feeds.feedburner.com	30
http://news.jetlib.com	27
http://oschuba.cedar.buffalo.edu	10
http://www.hanrss.com	7
http://translate.googleusercontent.com	6
http://127.0.0.1:8795	4
http://semanticweb.collected.info	4
http://readwriteweb.com	3
http://www.molequedeideias.net	3
http://cloudcomputing.collected.info	3
http://techandstuff.collected.info	3
http://m.readwriteweb.com	3
http://www.seofacts.biz	3
http://digitalapprentices.collected.info	2
http://www.365online.nu	2
http://digitalscene.collected.info	2
http://translate.google.com	2
http://www.slideshare.net	2
http://localhost:8080	2
http://www.techgig.com	1
http://internetbusiness.collected.info	1
http://www.thelinuxcrowd.com	1
http://ideafarming.collected.info	1
http://trunk.ly	1
http://tweetree.com	1
http://www.thelinuxcrowd.co.za	1
http://zoomagbg.collected.info	1
http://blog.infinitegraph.com	1
http://futureweb.collected.info	1
http://lookupquick.com	1
http://www.readingshq.com	1
http://www2.warwick.ac.uk	1
http://websitestech.collected.info	1
http://www.gotubo.com	1
http://localhost:8087	1
http://tech.collected.info	1
http://marcryan.collected.info	1
http://filips.collected.info	1
http://techapplicant.collected.info	1
http://aitao.collected.info	1

An overview of InfiniteGraph, the distributed graph database

by InfiniteGraph on Apr 21, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

43 Embeds 2,432

Statistics

An overview of InfiniteGraph, the distributed graph database — Presentation Transcript