Introduction datastructure for GraphDB          mixi.inc       Shunya Kimura         @kimuras
agenda• Motivation• Introduction• Structure for Node• Create Node• Appendix
Motivation
• Just interesting in GraphDB data structure• Why fast• Mysterious
Introduction
Neo4j• OSS implement of GraphDB• High availability• Scales to billions of nods and relationships• High speed querying thro...
Introduction for Graph   Node  Name: Kimura    Job: Eng                 Relationship
Introduction for Graph   Node  Name: Kimura    Job: Eng                 Relationship
Introduction for Graph     Node Data Structure  Create Node                  Relationship
Structure for Node
A node data structureNode Manager
A node data structureNode Manager  Node Cache
A node data structureNode Manager  Node Cache   Relation Cache
A node data structureNode Manager  Node Cache   Relation Cache   Cache Manager
A node data structureNode Manager  Node Cache      Relation Cache   Cache Manager Property Index    Manager
A node data structureNode Manager  Node Cache        Relation Cache      Cache Manager Property Index                  Tra...
A node data structureNode Manager  Node Cache        Relation Cache        Cache Manager Property Index                  T...
A node data structure  Node Manager           Node Cache                 Relation Cache               Cache Manager       ...
Node Manager•   NodeCache: Caching node id and node object•   RelationCache: Caching rel id and Relation object•   Cache M...
Relationship structure      id relationships   Node
Relationship structure      idrelationships    Node
Relationship structure           Array structure for relationships      idrelationships    Node
Relationship structure        relationships           In Blocks          Out Blocks           Node
Relationship structureNode                    Node        relationships           In BlocksNode                    Node   ...
Relationship structure       In DirectionNode                             Node                 relationships              ...
Relationship structure       In DirectionNode                                Node                 relationships           ...
Relationship structure                 Node      id relationships
Relationship structure                    Node         id    relationshipsrelationships
Relationship structure                            Node          id     relationshipsrelationships In blocks  ID:1    ID:2 ...
Relationship structure                            Node          id     relationshipsrelationships In blocks  ID:1    ID:2 ...
Relationship structure                                Node          id                Last In block     relationshipsrelat...
Relationship structure                                 Node          id                Last In block     relationships    ...
Create Nodes
Create Node• Understanding data structure with creating  nodes • Create Node A • Create Node B • Create Relationship A to B
Create NodeGraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH );Node NodeA = graphDb.createNode();Node Node...
Create NodeGraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH );Node NodeA = graphDb.createNode();Node Node...
Create NodeGraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH );Node NodeA = graphDb.createNode();Node Node...
Create NodeGraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH );Node NodeA = graphDb.createNode();Node Node...
Introduction Create Node
Introduction Create Node     NodeA
Introduction Create Node     Node Manager      Node Cache   Persistence Manager     Relation Cache             NodeA
Introduction Create Node               Create New ID   id     Node Manager      Node Cache   Persistence Manager     Relat...
Introduction Create Node               Create New ID   id                                    Create Node     Node Manager ...
Introduction Create Node               Create New ID      id                                               Create Node    ...
Introduction Create Node               Create New ID          id                                                 Create No...
Cache• You can choice some cache type • Strong Reference Cache • Soft LRU Cache (default) • Weak LRU Cache • No Cache
What is LRU Cache• discards the least recently used items first.                   http://en.wikipedia.org/wiki/Cache_algor...
What is LRU Cache• discards the least recently used items first.                   http://en.wikipedia.org/wiki/Cache_algor...
What is LRU Cache• discards the least recently used items first.                   http://en.wikipedia.org/wiki/Cache_algor...
What is LRU Cache • discards the least recently used items first.                    http://en.wikipedia.org/wiki/Cache_alg...
What is LRU Cache • discards the least recently used items first.                    http://en.wikipedia.org/wiki/Cache_alg...
What is LRU Cache• discards the least recently used items first.                   http://en.wikipedia.org/wiki/Cache_algor...
What is LRU Cache • discards the least recently used items first.                    http://en.wikipedia.org/wiki/Cache_alg...
What is LRU Cache • discards the least recently used items first.                    http://en.wikipedia.org/wiki/Cache_alg...
Soft LRU Cache
Soft LRU Cache Soft LRU CacheCache (ConcurrentHashMap)Queue (SoftReferenceQueue)
Soft LRU Cache                             ConcurrentHashMap                                  KEY            VALUE        ...
Soft LRU Cache                                                               ConcurrentHashMap                            ...
Soft LRU Cache                                                               ConcurrentHashMap                            ...
Comparing Cache type Cache Type          Queue Type          Reference Type Weak LRU Cache     WeakReferenceQueue     Weak...
Introduction Create Node               Create New ID          id                                                 Create No...
Introduction Create Node               Create New ID          id                                                 Create No...
Introduction Create Node     NodeA     NodeB
Introduction Create Node     NodeA     NodeB
Introduction Create Node                                           id  Node Manager                        relationships  ...
Introduction Create Node                                                         id  Node Manager                         ...
Introduction Create Node                                                                       id  Node Manager           ...
Introduction Create Node                                           id  Node Manager                        relationships  ...
Introduction Create Node                   Create New Relation ID                                      rel id             ...
Introduction Create Node                   Create New Relation ID                                                  create ...
Introduction Create Node                   Create New Relation ID                                                  create ...
Appendix
Implemented Graph        Algorithms• Centrality • Betweenness Centrality • Closeness Centrality• Path • A* • Dijkstart • S...
announcement
(NL   )  (DSIRNLP)(Tokyo NLP)(Japan.R)  (                    )  (TokyoWebmining)  (MongoDB Japan)                      htt...
Thanks!
Upcoming SlideShare
Loading in...5
×

Introduction data structure for GraphDB

5,930

Published on

Introduce graphDB structure with Neo4j.

Published in: Technology, Business
4 Comments
27 Likes
Statistics
Notes
No Downloads
Views
Total Views
5,930
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
0
Comments
4
Likes
27
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Introduction data structure for GraphDB

    1. 1. Introduction datastructure for GraphDB mixi.inc Shunya Kimura @kimuras
    2. 2. agenda• Motivation• Introduction• Structure for Node• Create Node• Appendix
    3. 3. Motivation
    4. 4. • Just interesting in GraphDB data structure• Why fast• Mysterious
    5. 5. Introduction
    6. 6. Neo4j• OSS implement of GraphDB• High availability• Scales to billions of nods and relationships• High speed querying through traversals• For more Information about Neo4j • http://www.slideshare.net/skimura/
    7. 7. Introduction for Graph Node Name: Kimura Job: Eng Relationship
    8. 8. Introduction for Graph Node Name: Kimura Job: Eng Relationship
    9. 9. Introduction for Graph Node Data Structure Create Node Relationship
    10. 10. Structure for Node
    11. 11. A node data structureNode Manager
    12. 12. A node data structureNode Manager Node Cache
    13. 13. A node data structureNode Manager Node Cache Relation Cache
    14. 14. A node data structureNode Manager Node Cache Relation Cache Cache Manager
    15. 15. A node data structureNode Manager Node Cache Relation Cache Cache Manager Property Index Manager
    16. 16. A node data structureNode Manager Node Cache Relation Cache Cache Manager Property Index Transaction Manager Manager
    17. 17. A node data structureNode Manager Node Cache Relation Cache Cache Manager Property Index Transaction Manager Persistence Manager Manager
    18. 18. A node data structure Node Manager Node Cache Relation Cache Cache Manager Property Index Transaction Manager Persistence Manager Manager id id id idrelationships relationships relationships relationships Node Node Node Node
    19. 19. Node Manager• NodeCache: Caching node id and node object• RelationCache: Caching rel id and Relation object• Cache Manager: Management cache threads• Property Index Manager: Management all nodes Properties• Persistence Manager: Management store
    20. 20. Relationship structure id relationships Node
    21. 21. Relationship structure idrelationships Node
    22. 22. Relationship structure Array structure for relationships idrelationships Node
    23. 23. Relationship structure relationships In Blocks Out Blocks Node
    24. 24. Relationship structureNode Node relationships In BlocksNode Node Out Blocks NodeNode NodeNode Node
    25. 25. Relationship structure In DirectionNode Node relationships In BlocksNode Node Out Blocks NodeNode NodeNode Node
    26. 26. Relationship structure In DirectionNode Node relationships In BlocksNode Node Out Blocks NodeNode Node Out DirectionNode Node
    27. 27. Relationship structure Node id relationships
    28. 28. Relationship structure Node id relationshipsrelationships
    29. 29. Relationship structure Node id relationshipsrelationships In blocks ID:1 ID:2 ID:3 ID:4 ID:5 ID:6 ID:7 ID:8 ID:9
    30. 30. Relationship structure Node id relationshipsrelationships In blocks ID:1 ID:2 ID:3 ID:4 ID:5 ID:6 ID:7 ID:8 ID:9 Out blocks ID:10 ID:11 ID:12 ID:13 ID:14 ID:15 ID:16 ID:17 ID:18
    31. 31. Relationship structure Node id Last In block relationshipsrelationships In blocks ID:1 ID:2 ID:3 ID:4 ID:5 ID:6 ID:7 ID:8 ID:9 Out blocks ID:10 ID:11 ID:12 ID:13 ID:14 ID:15 ID:16 ID:17 ID:18
    32. 32. Relationship structure Node id Last In block relationships Last Out blockrelationships In blocks ID:1 ID:2 ID:3 ID:4 ID:5 ID:6 ID:7 ID:8 ID:9 Out blocks ID:10 ID:11 ID:12 ID:13 ID:14 ID:15 ID:16 ID:17 ID:18
    33. 33. Create Nodes
    34. 34. Create Node• Understanding data structure with creating nodes • Create Node A • Create Node B • Create Relationship A to B
    35. 35. Create NodeGraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH );Node NodeA = graphDb.createNode();Node NodeB = graphDb.createNode();Relationship relationship =NodeA.createRelationshipTo( NodeB, RelTypes.KNOWS );
    36. 36. Create NodeGraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH );Node NodeA = graphDb.createNode();Node NodeB = graphDb.createNode();Relationship relationship =NodeA.createRelationshipTo( NodeB, RelTypes.KNOWS ); NodeA
    37. 37. Create NodeGraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH );Node NodeA = graphDb.createNode();Node NodeB = graphDb.createNode();Relationship relationship =NodeA.createRelationshipTo( NodeB, RelTypes.KNOWS ); NodeA NodeB
    38. 38. Create NodeGraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH );Node NodeA = graphDb.createNode();Node NodeB = graphDb.createNode();Relationship relationship =NodeA.createRelationshipTo( NodeB, RelTypes.KNOWS ); NodeA NodeB
    39. 39. Introduction Create Node
    40. 40. Introduction Create Node NodeA
    41. 41. Introduction Create Node Node Manager Node Cache Persistence Manager Relation Cache NodeA
    42. 42. Introduction Create Node Create New ID id Node Manager Node Cache Persistence Manager Relation Cache NodeA
    43. 43. Introduction Create Node Create New ID id Create Node Node Manager id Node Cache relationships Persistence Manager NodeA Relation Cache NodeA
    44. 44. Introduction Create Node Create New ID id Create Node Node Manager id register Node cache Node Cache relationships Persistence Manager NodeA Relation Cache NodeA
    45. 45. Introduction Create Node Create New ID id Create Node Node Manager id register Node cache Node Cache relationships Persistence Manager register Persistence Manager NodeA Relation Cache NodeA
    46. 46. Cache• You can choice some cache type • Strong Reference Cache • Soft LRU Cache (default) • Weak LRU Cache • No Cache
    47. 47. What is LRU Cache• discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms
    48. 48. What is LRU Cache• discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE 1 VAL_1 2 VAL_2 3 VAL_3 4 VAL_4 5 VAL_5
    49. 49. What is LRU Cache• discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE 1 VAL_1 2 VAL_2 Priority 3 VAL_3 4 VAL_4 5 VAL_5
    50. 50. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUEGET KEY:5 1 VAL_1 2 VAL_2 Priority 3 VAL_3 4 VAL_4 5 VAL_5
    51. 51. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUEGET KEY:5 5 VAL_5 1 VAL_1 Priority 2 VAL_2 3 VAL_3 4 VAL_4
    52. 52. What is LRU Cache• discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE 5 VAL_5 1 VAL_1 Priority 2 VAL_2 3 VAL_3 4 VAL_4
    53. 53. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUEPUT KEY:6 5 VAL_5 1 VAL_1 Priority 2 VAL_2 3 VAL_3 4 VAL_4
    54. 54. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUEPUT KEY:6 6 VAL_6 5 VAL_5 Priority 1 VAL_1 2 VAL_2 3 VAL_3
    55. 55. Soft LRU Cache
    56. 56. Soft LRU Cache Soft LRU CacheCache (ConcurrentHashMap)Queue (SoftReferenceQueue)
    57. 57. Soft LRU Cache ConcurrentHashMap KEY VALUE 1 ref_1 Soft LRU Cache 2 ref_2 3 ref_3Cache (ConcurrentHashMap)Queue (SoftReferenceQueue)
    58. 58. Soft LRU Cache ConcurrentHashMap KEY VALUE 1 ref_1 Soft LRU Cache 2 ref_2 3 ref_3 Cache (ConcurrentHashMap) SoftRefrenceQueue Queue (SoftReferenceQueue) KEY VALUE(※SoftRefrence) 1 ref_1 2 ref_2 3 ref_3 ※Soft reference objects, which are cleared at the discretion of the garbage collector in response to memorydemand.http://docs.oracle.com/javase/7/docs/api/java/lang/ref/SoftReference.html
    59. 59. Soft LRU Cache ConcurrentHashMap KEY VALUE 1 ref_1 Soft LRU Cache 2 ref_2 3 ref_3 Cache (ConcurrentHashMap) SoftRefrenceQueue Queue (SoftReferenceQueue) KEY VALUE(※SoftRefrence) 1 ref_1 2 ref_2 3 ref_3 ※Soft reference objects, which are cleared at the discretion of the garbage collector in response to memorydemand.http://docs.oracle.com/javase/7/docs/api/java/lang/ref/SoftReference.html
    60. 60. Comparing Cache type Cache Type Queue Type Reference Type Weak LRU Cache WeakReferenceQueue WeakReference Soft LRU Cache SoftRefrenceQueue SoftReference Strong Reference None None CacheRemaining Reference on memoryStrong > Soft > Weak
    61. 61. Introduction Create Node Create New ID id Create Node Node Manager id register Node cache Node Cache relationships Persistence Manager register Persistence Manager NodeA Relation Cache NodeA
    62. 62. Introduction Create Node Create New ID id Create Node Node Manager id register Node cache Node Cache relationships Persistence Manager register Persistence Manager NodeA Relation Cache NodeA NodeB
    63. 63. Introduction Create Node NodeA NodeB
    64. 64. Introduction Create Node NodeA NodeB
    65. 65. Introduction Create Node id Node Manager relationships Node Cache NodeAPersistence Manager id Relation Cache relationships NodeB NodeA NodeB
    66. 66. Introduction Create Node id Node Manager relationships Node Cache NodeA Register OUT direction to NodeBPersistence Manager id Relation Cache relationships NodeB NodeA NodeB
    67. 67. Introduction Create Node id Node Manager relationships Node Cache NodeA Register OUT direction to NodeBPersistence Manager id Relation Cache relationships Register IN direction from NodeA NodeB NodeA NodeB
    68. 68. Introduction Create Node id Node Manager relationships Node Cache NodeAPersistence Manager id Relation Cache relationships NodeB NodeA NodeB
    69. 69. Introduction Create Node Create New Relation ID rel id id Node Manager relationships Node Cache NodeAPersistence Manager id Relation Cache relationships NodeB NodeA NodeB
    70. 70. Introduction Create Node Create New Relation ID create Relationship rel id id Node Manager relationships Node Cache NodeA RelationPersistence Manager id Relation Cache relationships NodeB NodeA NodeB
    71. 71. Introduction Create Node Create New Relation ID create Relationship rel id id Node Manager relationships Node Cache NodeA RelationPersistence Manager id Relation Cache relationships NodeB NodeA NodeB
    72. 72. Appendix
    73. 73. Implemented Graph Algorithms• Centrality • Betweenness Centrality • Closeness Centrality• Path • A* • Dijkstart • Shortest Path
    74. 74. announcement
    75. 75. (NL ) (DSIRNLP)(Tokyo NLP)(Japan.R) ( ) (TokyoWebmining) (MongoDB Japan) http://www.ipsj.or.jp/10jigyo/taikai/74kai/event_2-6.html
    76. 76. Thanks!

    ×