Introduction data structure for GraphDB

Introduction datastructure for GraphDB mixi.inc Shunya Kimura @kimuras

agenda• Motivation• Introduction• Structure for Node• Create Node• Appendix

Motivation

• Just interesting in GraphDB data structure• Why fast• Mysterious

Introduction

Neo4j• OSS implement of GraphDB• High availability• Scales to billions of nods and relationships• High speed querying through traversals• For more Information about Neo4j • http://www.slideshare.net/skimura/

Introduction for Graph Node Name: Kimura Job: Eng Relationship

Introduction for Graph Node Data Structure Create Node Relationship

Structure for Node

A node data structureNode Manager

A node data structureNode Manager Node Cache

A node data structureNode Manager Node Cache Relation Cache

A node data structureNode Manager Node Cache Relation Cache Cache Manager

A node data structureNode Manager Node Cache Relation Cache Cache Manager Property Index Manager

A node data structureNode Manager Node Cache Relation Cache Cache Manager Property Index Transaction Manager Manager

A node data structureNode Manager Node Cache Relation Cache Cache Manager Property Index Transaction Manager Persistence Manager Manager

A node data structure Node Manager Node Cache Relation Cache Cache Manager Property Index Transaction Manager Persistence Manager Manager id id id idrelationships relationships relationships relationships Node Node Node Node

Node Manager• NodeCache: Caching node id and node object• RelationCache: Caching rel id and Relation object• Cache Manager: Management cache threads• Property Index Manager: Management all nodes Properties• Persistence Manager: Management store

Relationship structure id relationships Node

Relationship structure idrelationships Node

Relationship structure Array structure for relationships idrelationships Node

Relationship structure relationships In Blocks Out Blocks Node

Relationship structureNode Node relationships In BlocksNode Node Out Blocks NodeNode NodeNode Node

Relationship structure In DirectionNode Node relationships In BlocksNode Node Out Blocks NodeNode NodeNode Node

Relationship structure In DirectionNode Node relationships In BlocksNode Node Out Blocks NodeNode Node Out DirectionNode Node

Relationship structure Node id relationships

Relationship structure Node id relationshipsrelationships

Relationship structure Node id relationshipsrelationships In blocks ID:1 ID:2 ID:3 ID:4 ID:5 ID:6 ID:7 ID:8 ID:9

Relationship structure Node id relationshipsrelationships In blocks ID:1 ID:2 ID:3 ID:4 ID:5 ID:6 ID:7 ID:8 ID:9 Out blocks ID:10 ID:11 ID:12 ID:13 ID:14 ID:15 ID:16 ID:17 ID:18

Relationship structure Node id Last In block relationshipsrelationships In blocks ID:1 ID:2 ID:3 ID:4 ID:5 ID:6 ID:7 ID:8 ID:9 Out blocks ID:10 ID:11 ID:12 ID:13 ID:14 ID:15 ID:16 ID:17 ID:18

Relationship structure Node id Last In block relationships Last Out blockrelationships In blocks ID:1 ID:2 ID:3 ID:4 ID:5 ID:6 ID:7 ID:8 ID:9 Out blocks ID:10 ID:11 ID:12 ID:13 ID:14 ID:15 ID:16 ID:17 ID:18

Create Nodes

Create Node• Understanding data structure with creating nodes • Create Node A • Create Node B • Create Relationship A to B

Create NodeGraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH );Node NodeA = graphDb.createNode();Node NodeB = graphDb.createNode();Relationship relationship =NodeA.createRelationshipTo( NodeB, RelTypes.KNOWS );

Create NodeGraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH );Node NodeA = graphDb.createNode();Node NodeB = graphDb.createNode();Relationship relationship =NodeA.createRelationshipTo( NodeB, RelTypes.KNOWS ); NodeA

Create NodeGraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH );Node NodeA = graphDb.createNode();Node NodeB = graphDb.createNode();Relationship relationship =NodeA.createRelationshipTo( NodeB, RelTypes.KNOWS ); NodeA NodeB

Introduction Create Node

Introduction Create Node NodeA

Introduction Create Node Node Manager Node Cache Persistence Manager Relation Cache NodeA

Introduction Create Node Create New ID id Node Manager Node Cache Persistence Manager Relation Cache NodeA

Introduction Create Node Create New ID id Create Node Node Manager id Node Cache relationships Persistence Manager NodeA Relation Cache NodeA

Introduction Create Node Create New ID id Create Node Node Manager id register Node cache Node Cache relationships Persistence Manager NodeA Relation Cache NodeA

Introduction Create Node Create New ID id Create Node Node Manager id register Node cache Node Cache relationships Persistence Manager register Persistence Manager NodeA Relation Cache NodeA

Cache• You can choice some cache type • Strong Reference Cache • Soft LRU Cache (default) • Weak LRU Cache • No Cache

What is LRU Cache• discards the least recently used items ﬁrst. http://en.wikipedia.org/wiki/Cache_algorithms

What is LRU Cache• discards the least recently used items ﬁrst. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE 1 VAL_1 2 VAL_2 3 VAL_3 4 VAL_4 5 VAL_5

What is LRU Cache• discards the least recently used items ﬁrst. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE 1 VAL_1 2 VAL_2 Priority 3 VAL_3 4 VAL_4 5 VAL_5

What is LRU Cache • discards the least recently used items ﬁrst. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUEGET KEY:5 1 VAL_1 2 VAL_2 Priority 3 VAL_3 4 VAL_4 5 VAL_5

What is LRU Cache • discards the least recently used items ﬁrst. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUEGET KEY:5 5 VAL_5 1 VAL_1 Priority 2 VAL_2 3 VAL_3 4 VAL_4

What is LRU Cache• discards the least recently used items ﬁrst. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE 5 VAL_5 1 VAL_1 Priority 2 VAL_2 3 VAL_3 4 VAL_4

What is LRU Cache • discards the least recently used items ﬁrst. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUEPUT KEY:6 5 VAL_5 1 VAL_1 Priority 2 VAL_2 3 VAL_3 4 VAL_4

What is LRU Cache • discards the least recently used items ﬁrst. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUEPUT KEY:6 6 VAL_6 5 VAL_5 Priority 1 VAL_1 2 VAL_2 3 VAL_3

Soft LRU Cache

Soft LRU Cache Soft LRU CacheCache (ConcurrentHashMap)Queue (SoftReferenceQueue)

Soft LRU Cache ConcurrentHashMap KEY VALUE 1 ref_1 Soft LRU Cache 2 ref_2 3 ref_3Cache (ConcurrentHashMap)Queue (SoftReferenceQueue)

Soft LRU Cache ConcurrentHashMap KEY VALUE 1 ref_1 Soft LRU Cache 2 ref_2 3 ref_3 Cache (ConcurrentHashMap) SoftRefrenceQueue Queue (SoftReferenceQueue) KEY VALUE(※SoftRefrence) 1 ref_1 2 ref_2 3 ref_3 ※Soft reference objects, which are cleared at the discretion of the garbage collector in response to memorydemand.http://docs.oracle.com/javase/7/docs/api/java/lang/ref/SoftReference.html

Comparing Cache type Cache Type Queue Type Reference Type Weak LRU Cache WeakReferenceQueue WeakReference Soft LRU Cache SoftRefrenceQueue SoftReference Strong Reference None None CacheRemaining Reference on memoryStrong > Soft > Weak

Introduction Create Node Create New ID id Create Node Node Manager id register Node cache Node Cache relationships Persistence Manager register Persistence Manager NodeA Relation Cache NodeA

Introduction Create Node Create New ID id Create Node Node Manager id register Node cache Node Cache relationships Persistence Manager register Persistence Manager NodeA Relation Cache NodeA NodeB

Introduction Create Node NodeA NodeB

Introduction Create Node id Node Manager relationships Node Cache NodeAPersistence Manager id Relation Cache relationships NodeB NodeA NodeB

Introduction Create Node id Node Manager relationships Node Cache NodeA Register OUT direction to NodeBPersistence Manager id Relation Cache relationships NodeB NodeA NodeB

Introduction Create Node id Node Manager relationships Node Cache NodeA Register OUT direction to NodeBPersistence Manager id Relation Cache relationships Register IN direction from NodeA NodeB NodeA NodeB

Introduction Create Node id Node Manager relationships Node Cache NodeAPersistence Manager id Relation Cache relationships NodeB NodeA NodeB

Introduction Create Node Create New Relation ID rel id id Node Manager relationships Node Cache NodeAPersistence Manager id Relation Cache relationships NodeB NodeA NodeB

Introduction Create Node Create New Relation ID create Relationship rel id id Node Manager relationships Node Cache NodeA RelationPersistence Manager id Relation Cache relationships NodeB NodeA NodeB

Appendix

Implemented Graph Algorithms• Centrality • Betweenness Centrality • Closeness Centrality• Path • A* • Dijkstart • Shortest Path

announcement

(NL ) (DSIRNLP)(Tokyo NLP)(Japan.R) ( ) (TokyoWebmining) (MongoDB Japan) http://www.ipsj.or.jp/10jigyo/taikai/74kai/event_2-6.html

Thanks!

http://a0.twimg.com	39
http://paper.li	25
http://us-w1.rockmelt.com	18
http://pdfslidestore.blogspot.com	5
https://si0.twimg.com	3
http://cordobamediterranea.blogspot.co.il	1
http://littlemisslizzyandfriends.blogspot.de	1
http://alem61mar.blogspot.co.uk	1
http://www.twylah.com	1
http://alem61mar.blogspot.com	1

Introduction data structure for GraphDB

by Shunya Kimura

Accessibility

Categories

Upload Details

Usage Rights

10 Embeds 95

Statistics

Introduction data structure for GraphDB Presentation Transcript