Successfully reported this slideshow.

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Introduction data structure for GraphDB

  1. 1. Introduction data structure for GraphDB mixi.inc Shunya Kimura @kimuras
  2. 2. agenda • Motivation • Introduction • Structure for Node • Create Node • Appendix
  3. 3. Motivation
  4. 4. • Just interesting in GraphDB data structure • Why fast • Mysterious
  5. 5. Introduction
  6. 6. Neo4j • OSS implement of GraphDB • High availability • Scales to billions of nods and relationships • High speed querying through traversals • For more Information about Neo4j • http://www.slideshare.net/skimura/
  7. 7. Introduction for Graph Node Name: Kimura Job: Eng Relationship
  8. 8. Introduction for Graph Node Name: Kimura Job: Eng Relationship
  9. 9. Introduction for Graph Node Data Structure Create Node Relationship
  10. 10. Structure for Node
  11. 11. A node data structure Node Manager
  12. 12. A node data structure Node Manager Node Cache
  13. 13. A node data structure Node Manager Node Cache Relation Cache
  14. 14. A node data structure Node Manager Node Cache Relation Cache Cache Manager
  15. 15. A node data structure Node Manager Node Cache Relation Cache Cache Manager Property Index Manager
  16. 16. A node data structure Node Manager Node Cache Relation Cache Cache Manager Property Index Transaction Manager Manager
  17. 17. A node data structure Node Manager Node Cache Relation Cache Cache Manager Property Index Transaction Manager Persistence Manager Manager
  18. 18. A node data structure Node Manager Node Cache Relation Cache Cache Manager Property Index Transaction Manager Persistence Manager Manager id id id id relationships relationships relationships relationships Node Node Node Node
  19. 19. Node Manager • NodeCache: Caching node id and node object • RelationCache: Caching rel id and Relation object • Cache Manager: Management cache threads • Property Index Manager: Management all nodes Properties • Persistence Manager: Management store
  20. 20. Relationship structure id relationships Node
  21. 21. Relationship structure id relationships Node
  22. 22. Relationship structure Array structure for relationships id relationships Node
  23. 23. Relationship structure relationships In Blocks Out Blocks Node
  24. 24. Relationship structure Node Node relationships In Blocks Node Node Out Blocks Node Node Node Node Node
  25. 25. Relationship structure In Direction Node Node relationships In Blocks Node Node Out Blocks Node Node Node Node Node
  26. 26. Relationship structure In Direction Node Node relationships In Blocks Node Node Out Blocks Node Node Node Out Direction Node Node
  27. 27. Relationship structure Node id relationships
  28. 28. Relationship structure Node id relationships relationships
  29. 29. Relationship structure Node id relationships relationships In blocks ID:1 ID:2 ID:3 ID:4 ID:5 ID:6 ID:7 ID:8 ID:9
  30. 30. Relationship structure Node id relationships relationships In blocks ID:1 ID:2 ID:3 ID:4 ID:5 ID:6 ID:7 ID:8 ID:9 Out blocks ID:10 ID:11 ID:12 ID:13 ID:14 ID:15 ID:16 ID:17 ID:18
  31. 31. Relationship structure Node id Last In block relationships relationships In blocks ID:1 ID:2 ID:3 ID:4 ID:5 ID:6 ID:7 ID:8 ID:9 Out blocks ID:10 ID:11 ID:12 ID:13 ID:14 ID:15 ID:16 ID:17 ID:18
  32. 32. Relationship structure Node id Last In block relationships Last Out block relationships In blocks ID:1 ID:2 ID:3 ID:4 ID:5 ID:6 ID:7 ID:8 ID:9 Out blocks ID:10 ID:11 ID:12 ID:13 ID:14 ID:15 ID:16 ID:17 ID:18
  33. 33. Create Nodes
  34. 34. Create Node • Understanding data structure with creating nodes • Create Node A • Create Node B • Create Relationship A to B
  35. 35. Create Node GraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH ); Node NodeA = graphDb.createNode(); Node NodeB = graphDb.createNode(); Relationship relationship = NodeA.createRelationshipTo( NodeB, RelTypes.KNOWS );
  36. 36. Create Node GraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH ); Node NodeA = graphDb.createNode(); Node NodeB = graphDb.createNode(); Relationship relationship = NodeA.createRelationshipTo( NodeB, RelTypes.KNOWS ); NodeA
  37. 37. Create Node GraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH ); Node NodeA = graphDb.createNode(); Node NodeB = graphDb.createNode(); Relationship relationship = NodeA.createRelationshipTo( NodeB, RelTypes.KNOWS ); NodeA NodeB
  38. 38. Create Node GraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH ); Node NodeA = graphDb.createNode(); Node NodeB = graphDb.createNode(); Relationship relationship = NodeA.createRelationshipTo( NodeB, RelTypes.KNOWS ); NodeA NodeB
  39. 39. Introduction Create Node
  40. 40. Introduction Create Node NodeA
  41. 41. Introduction Create Node Node Manager Node Cache Persistence Manager Relation Cache NodeA
  42. 42. Introduction Create Node Create New ID id Node Manager Node Cache Persistence Manager Relation Cache NodeA
  43. 43. Introduction Create Node Create New ID id Create Node Node Manager id Node Cache relationships Persistence Manager NodeA Relation Cache NodeA
  44. 44. Introduction Create Node Create New ID id Create Node Node Manager id register Node cache Node Cache relationships Persistence Manager NodeA Relation Cache NodeA
  45. 45. Introduction Create Node Create New ID id Create Node Node Manager id register Node cache Node Cache relationships Persistence Manager register Persistence Manager NodeA Relation Cache NodeA
  46. 46. Cache • You can choice some cache type • Strong Reference Cache • Soft LRU Cache (default) • Weak LRU Cache • No Cache
  47. 47. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms
  48. 48. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE 1 VAL_1 2 VAL_2 3 VAL_3 4 VAL_4 5 VAL_5
  49. 49. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE 1 VAL_1 2 VAL_2 Priority 3 VAL_3 4 VAL_4 5 VAL_5
  50. 50. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE GET KEY:5 1 VAL_1 2 VAL_2 Priority 3 VAL_3 4 VAL_4 5 VAL_5
  51. 51. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE GET KEY:5 5 VAL_5 1 VAL_1 Priority 2 VAL_2 3 VAL_3 4 VAL_4
  52. 52. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE 5 VAL_5 1 VAL_1 Priority 2 VAL_2 3 VAL_3 4 VAL_4
  53. 53. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE PUT KEY:6 5 VAL_5 1 VAL_1 Priority 2 VAL_2 3 VAL_3 4 VAL_4
  54. 54. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE PUT KEY:6 6 VAL_6 5 VAL_5 Priority 1 VAL_1 2 VAL_2 3 VAL_3
  55. 55. Soft LRU Cache
  56. 56. Soft LRU Cache Soft LRU Cache Cache (ConcurrentHashMap) Queue (SoftReferenceQueue)
  57. 57. Soft LRU Cache ConcurrentHashMap KEY VALUE 1 ref_1 Soft LRU Cache 2 ref_2 3 ref_3 Cache (ConcurrentHashMap) Queue (SoftReferenceQueue)
  58. 58. Soft LRU Cache ConcurrentHashMap KEY VALUE 1 ref_1 Soft LRU Cache 2 ref_2 3 ref_3 Cache (ConcurrentHashMap) SoftRefrenceQueue Queue (SoftReferenceQueue) KEY VALUE(※SoftRefrence) 1 ref_1 2 ref_2 3 ref_3 ※Soft reference objects, which are cleared at the discretion of the garbage collector in response to memory demand. http://docs.oracle.com/javase/7/docs/api/java/lang/ref/SoftReference.html
  59. 59. Soft LRU Cache ConcurrentHashMap KEY VALUE 1 ref_1 Soft LRU Cache 2 ref_2 3 ref_3 Cache (ConcurrentHashMap) SoftRefrenceQueue Queue (SoftReferenceQueue) KEY VALUE(※SoftRefrence) 1 ref_1 2 ref_2 3 ref_3 ※Soft reference objects, which are cleared at the discretion of the garbage collector in response to memory demand. http://docs.oracle.com/javase/7/docs/api/java/lang/ref/SoftReference.html
  60. 60. Comparing Cache type Cache Type Queue Type Reference Type Weak LRU Cache WeakReferenceQueue WeakReference Soft LRU Cache SoftRefrenceQueue SoftReference Strong Reference None None Cache Remaining Reference on memory Strong > Soft > Weak
  61. 61. Introduction Create Node Create New ID id Create Node Node Manager id register Node cache Node Cache relationships Persistence Manager register Persistence Manager NodeA Relation Cache NodeA
  62. 62. Introduction Create Node Create New ID id Create Node Node Manager id register Node cache Node Cache relationships Persistence Manager register Persistence Manager NodeA Relation Cache NodeA NodeB
  63. 63. Introduction Create Node NodeA NodeB
  64. 64. Introduction Create Node NodeA NodeB
  65. 65. Introduction Create Node id Node Manager relationships Node Cache NodeA Persistence Manager id Relation Cache relationships NodeB NodeA NodeB
  66. 66. Introduction Create Node id Node Manager relationships Node Cache NodeA Register OUT direction to NodeB Persistence Manager id Relation Cache relationships NodeB NodeA NodeB
  67. 67. Introduction Create Node id Node Manager relationships Node Cache NodeA Register OUT direction to NodeB Persistence Manager id Relation Cache relationships Register IN direction from NodeA NodeB NodeA NodeB
  68. 68. Introduction Create Node id Node Manager relationships Node Cache NodeA Persistence Manager id Relation Cache relationships NodeB NodeA NodeB
  69. 69. Introduction Create Node Create New Relation ID rel id id Node Manager relationships Node Cache NodeA Persistence Manager id Relation Cache relationships NodeB NodeA NodeB
  70. 70. Introduction Create Node Create New Relation ID create Relationship rel id id Node Manager relationships Node Cache NodeA Relation Persistence Manager id Relation Cache relationships NodeB NodeA NodeB
  71. 71. Introduction Create Node Create New Relation ID create Relationship rel id id Node Manager relationships Node Cache NodeA Relation Persistence Manager id Relation Cache relationships NodeB NodeA NodeB
  72. 72. Appendix
  73. 73. Implemented Graph Algorithms • Centrality • Betweenness Centrality • Closeness Centrality • Path • A* • Dijkstart • Shortest Path
  74. 74. announcement
  75. 75. (NL ) (DSIRNLP) (Tokyo NLP) (Japan.R) ( ) (TokyoWebmining) (MongoDB Japan) http://www.ipsj.or.jp/10jigyo/taikai/74kai/event_2-6.html
  76. 76. Thanks!

Editor's Notes

  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • ×